运行paddle.fluid.install_check.run_check()时报NCCL错误
Created by: ywlorna
- 版本、环境信息： 1）PaddlePaddle版本：1.7.1.post107 2）GPU：两块p40，cuda9.0，cudnn7 3）系统环境：Centos7，python2.7.5
- 安装方式： python -m pip install paddlepaddle-gpu
- 运行import paddle.fluid没问题； 运行paddle.fluid.install_check.run_check()报错信息如下： Running Verify Paddle Program ... W0402 12:55:56.624688 30772 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.1, Runtime API Version: 10.0 W0402 12:55:56.627355 30772 device_context.cc:245] device: 0, cuDNN Version: 7.0. W0402 12:55:56.627372 30772 device_context.cc:271] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.6, but CUDNN version in your machine is 7.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version. Your Paddle works well on SINGLE GPU or CPU. I0402 12:55:58.561069 30772 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 2 cards are used, so 2 programs are executed in parallel. /usr/lib64/python2.7/site-packages/paddle/fluid/executor.py:782: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") 2020-04-02 12:56:01,231-WARNING: Your Paddle has some problem with multiple GPU. This may be caused by:
- There is only 1 or 0 GPU visible on your Device;
- No.1 or No.2 GPU or both of them are occupied now
- Wrong installation of NVIDIA-NCCL2, please follow instruction on https://github.com/NVIDIA/nccl-tests to test your NCCL, or reinstall it following https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html
Original Error is:
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::platform::NCCLContextMap::NCCLContextMap(std::vector<paddle::platform::Place, std::allocatorpaddle::platform::Place > const&, ncclUniqueId*, unsigned long, unsigned long) 3 paddle::framework::ParallelExecutor::ParallelExecutor(std::vector<paddle::platform::Place, std::allocatorpaddle::platform::Place > const&, std::vector<std::string, std::allocatorstd::string > const&, std::string const&, paddle::framework::Scope*, std::vector<paddle::framework::Scope*, std::allocatorpaddle::framework::Scope* > const&, paddle::framework::details::ExecutionStrategy const&, paddle::framework::details::BuildStrategy const&, paddle::framework::ir::Graph*)
Error Message Summary:
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
- New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
- Recommended issue content: all error stack information [unhandled system error] at (/paddle/paddle/fluid/platform/nccl_helper.h:113)
**- NCCL版本：**libnccl-2.6.4-1+cuda9.0.x86_64 在https://github.com/NVIDIA/nccl-tests上测试了nccl，是没问题的，如下：
nThread 1 nGpus 2 minBytes 8 maxBytes 268435456 step: 2(factor) warmup iters: 5 iters: 20 validation: 1
Using devices
Rank 0 Pid 48339 on 5ac526b8-faa6-45fd-b33c-0fda50680877 device 0 [0x06] Tesla P40
Rank 1 Pid 48339 on 5ac526b8-faa6-45fd-b33c-0fda50680877 device 1 [0x0e] Tesla P40
out-of-place in-place
size count type redop time algbw busbw error time algbw busbw error
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
8 2 float sum 129.2 0.00 0.00 0e+00 128.3 0.00 0.00 0e+00
16 4 float sum 128.8 0.00 0.00 0e+00 128.1 0.00 0.00 0e+00
32 8 float sum 128.2 0.00 0.00 0e+00 127.9 0.00 0.00 0e+00
64 16 float sum 128.5 0.00 0.00 0e+00 15.78 0.00 0.00 0e+00
128 32 float sum 19.44 0.01 0.01 0e+00 49.95 0.00 0.00 0e+00...