多卡训练时出现问题
Created by: zhengzhe97
Enforce failed. Expected member_->places_.size() == lod_tensors.size(), but received member_->places_.size():2 != lod_tensors.size():1. The number of samples of current batch is less than the count of devices, currently, it is not allowed. (2 vs 1) at [/paddle/paddle/fluid/framework/parallel_executor.cc:314] PaddlePaddle Call Stacks: 0 0x7fd3f0002e96p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486 1 0x7fd3f00f804ep paddle::framework::ParallelExecutor::FeedAndSplitTensorIntoLocalScopes(std::unordered_map<std::string, paddle::framework::LoDTensor, std::hashstd::string, std::equal_tostd::string, std::allocator<std::pair<std::string const, paddle::framework::LoDTensor> > > const&) + 1886 2 0x7fd3f005aa61p 3 0x7fd3f002a81ep 4 0x551c73p PyCFunction_Call + 323 5 0x5c9b44p PyEval_EvalFrameEx + 30692 6 0x5cbcd1p 7 0x5ca777p PyEval_EvalFrameEx + 33815 8 0x5cbcd1p 9 0x61ab52p 10 0x551be2p PyCFunction_Call + 178 11 0x5ca66ep PyEval_EvalFrameEx + 33550 12 0x5cbcd1p 13 0x5ca777p PyEval_EvalFrameEx + 33815 14 0x5cbcd1p 15 0x5ca777p PyEval_EvalFrameEx + 33815 16 0x5cbcd1p 17 0x5ca777p PyEval_EvalFrameEx + 33815 18 0x5cbcd1p 19 0x5ccc95p 20 0x46278ap PyObject_Call + 90 21 0x5c4f69p PyEval_EvalFrameEx + 11273 22 0x5cbcd1p 23 0x5ca777p PyEval_EvalFrameEx + 33815 24 0x5cbcd1p 25 0x5ca777p PyEval_EvalFrameEx + 33815 26 0x5ca8a2p PyEval_EvalFrameEx + 34114 27 0x5ca8a2p PyEval_EvalFrameEx + 34114 28 0x5cbcd1p 29 0x5ccc95p 30 0x46278ap PyObject_Call + 90 31 0x5c4f69p PyEval_EvalFrameEx + 11273 32 0x5cbcd1p 33 0x5ccc95p 34 0x46278ap PyObject_Call + 90 35 0x5c4f69p PyEval_EvalFrameEx + 11273 36 0x5cbcd1p 37 0x5ca777p PyEval_EvalFrameEx + 33815 38 0x5ca8a2p PyEval_EvalFrameEx + 34114 39 0x5cbcd1p 40 0x5ccc95p 41 0x46278ap PyObject_Call + 90 42 0x5c4f69p PyEval_EvalFrameEx + 11273 43 0x5cbcd1p 44 0x5ca777p PyEval_EvalFrameEx + 33815 45 0x5cbcd1p 46 0x5ccb16p 47 0x46278ap PyObject_Call + 90 48 0x5c4f69p PyEval_EvalFrameEx + 11273 49 0x5ca8a2p PyEval_EvalFrameEx + 34114 50 0x5ca8a2p PyEval_EvalFrameEx + 34114 51 0x5ca8a2p PyEval_EvalFrameEx + 34114 52 0x5ca8a2p PyEval_EvalFrameEx + 34114 53 0x5ca8a2p PyEval_EvalFrameEx + 34114 54 0x5cbcd1p 55 0x5ca777p PyEval_EvalFrameEx + 33815 56 0x5cbcd1p 57 0x61ab52p 58 0x551be2p PyCFunction_Call + 178 59 0x5ca66ep PyEval_EvalFrameEx + 33550 60 0x5cbcd1p 61 0x5ca777p PyEval_EvalFrameEx + 33815 62 0x5cbcd1p 63 0x5ccb16p 64 0x46278ap PyObject_Call + 90 65 0x598e4bp 66 0x619eb5p Py_Main + 2309 67 0x41fc2dp main + 349 68 0x7fd460db6f45p __libc_start_main + 245 69 0x41fcdep