- Recommended issue content: all error stack information: unspecified launch failure at (/paddle/paddle/fluid/framework/details/op_handle_base.cc:39)
Created by: 60999
python cls_MobileNetV2_train.py
2020-05-18 23:03:03 [INFO] Starting to read file list from dataset...
2020-05-18 23:03:03 [INFO] 10 samples in file /home/aistudio/work/angle/cls/train_list.txt
2020-05-18 23:03:03 [INFO] Starting to read file list from dataset...
2020-05-18 23:03:03 [INFO] 10 samples in file /home/aistudio/work/angle/cls/train_list.txt
2020-05-18 23:03:07 [INFO] Decompressing /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained.tar...
W0518 23:03:08.569744 27905 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 10.1, Runtime API Version: 9.0
W0518 23:03:08.573760 27905 device_context.cc:245] device: 0, cuDNN Version: 7.3.
2020-05-18 23:03:10 [INFO] Load pretrain weights from /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained.
2020-05-18 23:03:10 [WARNING] [SKIP] Shape of pretrained weight /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained/fc10_weights doesn't match.(Pretrained: (1280, 1000), Actual: (1280, 10))
2020-05-18 23:03:10 [WARNING] [SKIP] Shape of pretrained weight /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained/fc10_offset doesn't match.(Pretrained: (1000,), Actual: (10,))
2020-05-18 23:03:10 [INFO] There are 265 varaibles in /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained are loaded.
I0518 23:03:13.865628 27905 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel.
I0518 23:03:13.901172 27905 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1
I0518 23:03:13.944672 27905 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I0518 23:03:13.963923 27905 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0
Error: /paddle/paddle/fluid/operators/cross_entropy_op.h:173 Assertion label >= 0 && label < feature_size_ failed. Variable value (label) of OP(fluid.layers.cross_entropy) expected >= 0 and < 10, but got 10. Please check label value.
terminate called after throwing an instance of 'paddle::platform::EnforceNotMet'
what():
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::framework::details::OpHandleBase::~OpHandleBase() 3 paddle::framework::details::FetchOpHandle::~FetchOpHandle() 4 paddle::framework::ir::Node::~Node() 5 paddle::framework::ir::Node::~Node() 6 paddle::framework::details::ClearFetchOp(paddle::framework::ir::Graph*, std::vector<paddle::framework::details::OpHandleBase*, std::allocatorpaddle::framework::details::OpHandleBase* >) 7 paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal(std::vector<paddle::framework::details::OpHandleBase, std::allocatorpaddle::framework::details::OpHandleBase* >*) 8 paddle::framework::details::FastThreadedSSAGraphExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&) 9 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&) 10 paddle::framework::ParallelExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&)
Error Message Summary:
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
- New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
- Recommended issue content: all error stack information: unspecified launch failure at (/paddle/paddle/fluid/framework/details/op_handle_base.cc:39)
W0518 23:03:15.879923 27905 init.cc:209] Warning: PaddlePaddle catches a failure signal, it may not work properly W0518 23:03:15.879958 27905 init.cc:211] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0518 23:03:15.879962 27905 init.cc:214] The detail failure signal is:
W0518 23:03:15.879966 27905 init.cc:217] *** Aborted at 1589814195 (unix time) try "date -d @1589814195" if you are using GNU date ***
W0518 23:03:15.881515 27905 init.cc:217] PC: @ 0x0 (unknown)
W0518 23:03:15.881626 27905 init.cc:217] *** SIGABRT (@0x3e800006d01) received by PID 27905 (TID 0x7fa0728e4700) from PID 27905; stack trace: ***
W0518 23:03:15.882773 27905 init.cc:217] @ 0x7fa0724cb390 (unknown)
W0518 23:03:15.883764 27905 init.cc:217] @ 0x7fa072125428 gsignal
W0518 23:03:15.884759 27905 init.cc:217] @ 0x7fa07212702a abort
W0518 23:03:15.885454 27905 init.cc:217] @ 0x7fa032e3e84a __gnu_cxx::__verbose_terminate_handler()
W0518 23:03:15.886009 27905 init.cc:217] @ 0x7fa032e3cf47 __cxxabiv1::__terminate()
W0518 23:03:15.886548 27905 init.cc:217] @ 0x7fa032e3c3a5 __cxa_call_terminate
W0518 23:03:15.887130 27905 init.cc:217] @ 0x7fa032e3cbd8 __gxx_personality_v0
W0518 23:03:15.887647 27905 init.cc:217] @ 0x7fa032d89aab _Unwind_RaiseException_Phase2
W0518 23:03:15.888190 27905 init.cc:217] @ 0x7fa032d89f49 _Unwind_Resume
W0518 23:03:15.890183 27905 init.cc:217] @ 0x7f9fd5ee990c paddle::framework::details::OpHandleBase::~OpHandleBase()
W0518 23:03:15.891499 27905 init.cc:217] @ 0x7f9fd5ea3d81 paddle::framework::details::FetchOpHandle::~FetchOpHandle()
W0518 23:03:15.893075 27905 init.cc:217] @ 0x7f9fd38e2d09 paddle::framework::ir::Node::~Node()
W0518 23:03:15.895388 27905 init.cc:217] @ 0x7f9fd38e2eb1 paddle::framework::ir::Node::~Node()
W0518 23:03:15.896546 27905 init.cc:217] @ 0x7f9fd5ea66c6 paddle::framework::details::ClearFetchOp()
W0518 23:03:15.897429 27905 init.cc:217] @ 0x7f9fd5ea26ba paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal()
W0518 23:03:15.900287 27905 init.cc:217] @ 0x7f9fd5ea129a paddle::framework::details::FastThreadedSSAGraphExecutor::Run()
W0518 23:03:15.901213 27905 init.cc:217] @ 0x7f9fd5deb7f0 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run()
W0518 23:03:15.903004 27905 init.cc:217] @ 0x7f9fd39e3088 paddle::framework::ParallelExecutor::Run()
W0518 23:03:15.903160 27905 init.cc:217] @ 0x7f9fd35fc308 ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9framework16ParallelExecutorERKSt6vectorISsSaISsEEE199_S9_INS6_9LoDTensorESaISF_EEIS8_SD_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESY
W0518 23:03:15.903861 27905 init.cc:217] @ 0x7f9fd3650ed1 pybind11::cpp_function::dispatcher()
W0518 23:03:15.904192 27905 init.cc:217] @ 0x55ee5b32c744 _PyMethodDef_RawFastCallKeywords
W0518 23:03:15.904430 27905 init.cc:217] @ 0x55ee5b32c861 _PyCFunction_FastCallKeywords
W0518 23:03:15.904656 27905 init.cc:217] @ 0x55ee5b3986e8 _PyEval_EvalFrameDefault
W0518 23:03:15.904867 27905 init.cc:217] @ 0x55ee5b2dc539 _PyEval_EvalCodeWithName
W0518 23:03:15.905076 27905 init.cc:217] @ 0x55ee5b32bf57 _PyFunction_FastCallKeywords
W0518 23:03:15.905300 27905 init.cc:217] @ 0x55ee5b3948cc _PyEval_EvalFrameDefault
W0518 23:03:15.905509 27905 init.cc:217] @ 0x55ee5b2dc539 _PyEval_EvalCodeWithName
W0518 23:03:15.905704 27905 init.cc:217] @ 0x55ee5b32bf57 _PyFunction_FastCallKeywords
W0518 23:03:15.905926 27905 init.cc:217] @ 0x55ee5b3948cc _PyEval_EvalFrameDefault
W0518 23:03:15.906143 27905 init.cc:217] @ 0x55ee5b2dc539 _PyEval_EvalCodeWithName
W0518 23:03:15.906337 27905 init.cc:217] @ 0x55ee5b32bf57 _PyFunction_FastCallKeywords
W0518 23:03:15.906560 27905 init.cc:217] @ 0x55ee5b3948cc _PyEval_EvalFrameDefault
Aborted (core dumped)
aistudio@jupyter-255651-455384:~/work/angle$ python cls_MobileNetV2_train.py
2020-05-18 23:04:49 [INFO] Starting to read file list from dataset...
2020-05-18 23:04:49 [INFO] 10 samples in file /home/aistudio/work/angle/cls/train_list.txt
2020-05-18 23:04:49 [INFO] Starting to read file list from dataset...
2020-05-18 23:04:49 [INFO] 10 samples in file /home/aistudio/work/angle/cls/train_list.txt
2020-05-18 23:04:53 [INFO] Decompressing /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained.tar...
W0518 23:04:54.474054 28277 device_context.cc:237] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 10.1, Runtime API Version: 9.0
W0518 23:04:54.478101 28277 device_context.cc:245] device: 0, cuDNN Version: 7.3.
2020-05-18 23:04:56 [INFO] Load pretrain weights from /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained.
2020-05-18 23:04:56 [WARNING] [SKIP] Shape of pretrained weight /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained/fc10_weights doesn't match.(Pretrained: (1280, 1000), Actual: (1280, 10))
2020-05-18 23:04:56 [WARNING] [SKIP] Shape of pretrained weight /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained/fc10_offset doesn't match.(Pretrained: (1000,), Actual: (10,))
2020-05-18 23:04:56 [INFO] There are 265 varaibles in /home/aistudio/work/angle/output/cls_MobileNetV2/pretrain/MobileNetV2_pretrained are loaded.
I0518 23:04:59.827353 28277 parallel_executor.cc:440] The Program will be executed on CUDA using ParallelExecutor, 1 cards are used, so 1 programs are executed in parallel.
I0518 23:04:59.861418 28277 build_strategy.cc:365] SeqOnlyAllReduceOps:0, num_trainers:1
I0518 23:04:59.903942 28277 parallel_executor.cc:307] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I0518 23:04:59.926091 28277 parallel_executor.cc:375] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0
Error: /paddle/paddle/fluid/operators/cross_entropy_op.h:173 Assertion label >= 0 && label < feature_size_ failed. Variable value (label) of OP(fluid.layers.cross_entropy) expected >= 0 and < 10, but got 10. Please check label value.
terminate called after throwing an instance of 'paddle::platform::EnforceNotMet'
what():
C++ Call Stacks (More useful to developers):
0 std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int) 1 paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) 2 paddle::framework::details::OpHandleBase::~OpHandleBase() 3 paddle::framework::details::FetchOpHandle::~FetchOpHandle() 4 paddle::framework::ir::Node::~Node() 5 paddle::framework::ir::Node::~Node() 6 paddle::framework::details::ClearFetchOp(paddle::framework::ir::Graph*, std::vector<paddle::framework::details::OpHandleBase*, std::allocatorpaddle::framework::details::OpHandleBase* >) 7 paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal(std::vector<paddle::framework::details::OpHandleBase, std::allocatorpaddle::framework::details::OpHandleBase* >*) 8 paddle::framework::details::FastThreadedSSAGraphExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&) 9 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&) 10 paddle::framework::ParallelExecutor::Run(std::vector<std::string, std::allocatorstd::string > const&)
Error Message Summary:
Error: An error occurred here. There is no accurate error hint for this error yet. We are continuously in the process of increasing hint for this kind of error check. It would be helpful if you could inform us of how this conversion went by opening a github issue. And we will resolve it with high priority.
- New issue link: https://github.com/PaddlePaddle/Paddle/issues/new
- Recommended issue content: all error stack information: unspecified launch failure at (/paddle/paddle/fluid/framework/details/op_handle_base.cc:39)
W0518 23:05:01.833124 28277 init.cc:209] Warning: PaddlePaddle catches a failure signal, it may not work properly W0518 23:05:01.833159 28277 init.cc:211] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle W0518 23:05:01.833163 28277 init.cc:214] The detail failure signal is:
W0518 23:05:01.833173 28277 init.cc:217] *** Aborted at 1589814301 (unix time) try "date -d @1589814301" if you are using GNU date *** W0518 23:05:01.834684 28277 init.cc:217] PC: @ 0x0 (unknown) W0518 23:05:01.834777 28277 init.cc:217] *** SIGABRT (@0x3e800006e75) received by PID 28277 (TID 0x7ff751aba700) from PID 28277; stack trace: *** W0518 23:05:01.835809 28277 init.cc:217] @ 0x7ff7516a1390 (unknown) W0518 23:05:01.836807 28277 init.cc:217] @ 0x7ff7512fb428 gsignal W0518 23:05:01.837767 28277 init.cc:217] @ 0x7ff7512fd02a abort W0518 23:05:01.838440 28277 init.cc:217] @ 0x7ff71201484a __gnu_cxx::__verbose_terminate_handler() W0518 23:05:01.839000 28277 init.cc:217] @ 0x7ff712012f47 __cxxabiv1::__terminate() W0518 23:05:01.839529 28277 init.cc:217] @ 0x7ff7120123a5 __cxa_call_terminate W0518 23:05:01.840111 28277 init.cc:217] @ 0x7ff712012bd8 __gxx_personality_v0 W0518 23:05:01.840646 28277 init.cc:217] @ 0x7ff711f5faab _Unwind_RaiseException_Phase2 W0518 23:05:01.841154 28277 init.cc:217] @ 0x7ff711f5ff49 _Unwind_Resume W0518 23:05:01.843084 28277 init.cc:217] @ 0x7ff6b50bf90c paddle::framework::details::OpHandleBase::~OpHandleBase() W0518 23:05:01.844424 28277 init.cc:217] @ 0x7ff6b5079d81 paddle::framework::details::FetchOpHandle::~FetchOpHandle() W0518 23:05:01.845989 28277 init.cc:217] @ 0x7ff6b2ab8d09 paddle::framework::ir::Node::~Node() W0518 23:05:01.848273 28277 init.cc:217] @ 0x7ff6b2ab8eb1 paddle::framework::ir::Node::~Node() W0518 23:05:01.849385 28277 init.cc:217] @ 0x7ff6b507c6c6 paddle::framework::details::ClearFetchOp() W0518 23:05:01.850240 28277 init.cc:217] @ 0x7ff6b50786ba paddle::framework::details::FastThreadedSSAGraphExecutor::ExecutionFinal() W0518 23:05:01.853020 28277 init.cc:217] @ 0x7ff6b507729a paddle::framework::details::FastThreadedSSAGraphExecutor::Run() W0518 23:05:01.853901 28277 init.cc:217] @ 0x7ff6b4fc17f0 paddle::framework::details::ScopeBufferedSSAGraphExecutor::Run() W0518 23:05:01.855635 28277 init.cc:217] @ 0x7ff6b2bb9088 paddle::framework::ParallelExecutor::Run() W0518 23:05:01.855758 28277 init.cc:217] @ 0x7ff6b27d2308 ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9framework16ParallelExecutorERKSt6vectorISsSaISsEEE199_S9_INS6_9LoDTensorESaISF_EEIS8_SD_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESY W0518 23:05:01.856428 28277 init.cc:217] @ 0x7ff6b2826ed1 pybind11::cpp_function::dispatcher() W0518 23:05:01.856714 28277 init.cc:217] @ 0x564f32662744 _PyMethodDef_RawFastCallKeywords W0518 23:05:01.856937 28277 init.cc:217] @ 0x564f32662861 _PyCFunction_FastCallKeywords W0518 23:05:01.857156 28277 init.cc:217] @ 0x564f326ce6e8 _PyEval_EvalFrameDefault W0518 23:05:01.857358 28277 init.cc:217] @ 0x564f32612539 _PyEval_EvalCodeWithName W0518 23:05:01.857549 28277 init.cc:217] @ 0x564f32661f57 _PyFunction_FastCallKeywords W0518 23:05:01.857766 28277 init.cc:217] @ 0x564f326ca8cc _PyEval_EvalFrameDefault W0518 23:05:01.857964 28277 init.cc:217] @ 0x564f32612539 _PyEval_EvalCodeWithName W0518 23:05:01.858151 28277 init.cc:217] @ 0x564f32661f57 _PyFunction_FastCallKeywords W0518 23:05:01.858374 28277 init.cc:217] @ 0x564f326ca8cc _PyEval_EvalFrameDefault W0518 23:05:01.858579 28277 init.cc:217] @ 0x564f32612539 _PyEval_EvalCodeWithName W0518 23:05:01.858772 28277 init.cc:217] @ 0x564f32661f57 _PyFunction_FastCallKeywords W0518 23:05:01.858994 28277 init.cc:217] @ 0x564f326ca8cc _PyEval_EvalFrameDefault Aborted (core dumped)
aistudio labels.txt