Skip to content

P
Paddle

PaddlePaddle / Paddle
大约 2 年 前同步成功

通知 2325
Star 20933
Fork 5424
P
Paddle
开放中
Opened by saxon_zh@saxon_zhGuest

单机多卡训练报错

Created by: xeyond

  • 版本、环境信息：    1）PaddlePaddle版本：1.7    2）GPU：P40，Cuda9    4）系统环境：CentOS，Python 2.7.14
  • 训练信息    1）单机，多卡
  • 问题描述：使用单机多卡模式训练模型，报以下错误。place由CUDA改成CPU模式就可以正常运行 日志如下： I0429 13:05:16.509969 124336 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 8 cards are used, so 8 programs are executed in parallel. /home/wangxiyang01/.pyenv/versions/2.7.14/lib/python2.7/site-packages/paddle/fluid/executor.py:789: UserWarning: The following exception is not an EOF exception. "The following exception is not an EOF exception.") Traceback (most recent call last): File "train2.py", line 98, in metrics = exe.run(compiled_prog, feed=data, fetch_list=[avg_cost, y_pred, y]) File "/home/wangxiyang01/.pyenv/versions/2.7.14/lib/python2.7/site-packages/paddle/fluid/executor.py", line 790, in run six.reraise(*sys.exc_info()) File "/home/wangxiyang01/.pyenv/versions/2.7.14/lib/python2.7/site-packages/paddle/fluid/executor.py", line 785, in run use_program_cache=use_program_cache) File "/home/wangxiyang01/.pyenv/versions/2.7.14/lib/python2.7/site-packages/paddle/fluid/executor.py", line 840, in _run_impl program._compile(scope, self.place) File "/home/wangxiyang01/.pyenv/versions/2.7.14/lib/python2.7/site-packages/paddle/fluid/compiler.py", line 434, in _compile places=self._places) File "/home/wangxiyang01/.pyenv/versions/2.7.14/lib/python2.7/site-packages/paddle/fluid/compiler.py", line 387, in _compile_data_parallel self._exec_strategy, self._build_strategy, self._graph) paddle.fluid.core_avx.EnforceNotMet:

-------------------------------------------- C++ Call Stacks (More useful to developers): -------------------------------------------- 0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::platform::NCCLGroupGuard::~NCCLGroupGuard() 3 paddle::framework::ParallelExecutor::BCastParamsToDevices(std::vector<std::string, std::allocatorstd::string > const&, int) const 4 paddle::framework::ParallelExecutor::ParallelExecutor(std::vector<paddle::platform::Place, std::allocatorpaddle::platform::Place > const&, std::vector<std::string, std::allocatorstd::string > const&, std::string const&, paddle::framework::Scope*, std::vector<paddle::framework::Scope*, std::allocatorpaddle::framework::Scope* > const&, paddle::framework::details::ExecutionStrategy const&, paddle::framework::details::BuildStrategy const&, paddle::framework::ir::Graph*)

---------------------- Error Message Summary: ---------------------- Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.