Created by: kolinwei

[23:05:14]W: [Step 3/3] [37m--- detected 63 subgraphs [0m [23:05:22]W: [Step 3/3] I0508 23:05:22.874912 22781 build_strategy.cc:305] SeqOnlyAllReduceOps:0, num_trainers:1 [23:05:27]W: [Step 3/3] F0508 23:05:27.544423 22801 device_context.cc:331] cudaStreamSynchronize unspecified launch failure errno:4 [23:05:27]W: [Step 3/3] *** Check failure stack trace: *** [23:05:27]W: [Step 3/3] @ 0x7fa1e086f57d google::LogMessage::Fail() [23:05:27]W: [Step 3/3] @ 0x7fa1e087302c google::LogMessage::SendToLog() [23:05:27]W: [Step 3/3] @ 0x7fa1e086f0a3 google::LogMessage::Flush() [23:05:27]W: [Step 3/3] @ 0x7fa1e087453e google::LogMessageFatal::~LogMessageFatal() [23:05:27]W: [Step 3/3] @ 0x7fa1e24b6e0d _ZNSt17_Function_handlerIFvvEZNK6paddle8platform17CUDADeviceContext4WaitEvEUlvE_E9_M_invokeERKSt9_Any_data [23:05:27]W: [Step 3/3] @ 0x7fa1e24c4c64 paddle::platform::TemporaryAllocator::Release() [23:05:27]W: [Step 3/3] @ 0x7fa1e24b9d81 paddle::platform::CUDADeviceContext::Wait() [23:05:27]W: [Step 3/3] @ 0x7fa1e2443a76 paddle::framework::OperatorWithKernel::RunImpl() [23:05:27]W: [Step 3/3] @ 0x7fa1e2443cf1 paddle::framework::OperatorWithKernel::RunImpl() [23:05:27]W: [Step 3/3] @ 0x7fa1e244131c paddle::framework::OperatorBase::Run() [23:05:27]W: [Step 3/3] @ 0x7fa1e225028a paddle::framework::details::ComputationOpHandle::RunImpl() [23:05:27]W: [Step 3/3] @ 0x7fa1e2242980 paddle::framework::details::OpHandleBase::Run() [23:05:27]W: [Step 3/3] @ 0x7fa1e2223bb2 paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp() [23:05:27]W: [Step 3/3] @ 0x7fa1e2223eef _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data [23:05:27]W: [Step 3/3] @ 0x7fa1e14d72a3 std::_Function_handler<>::_M_invoke() [23:05:27]W: [Step 3/3] @ 0x7fa1e0807af7 std::__future_base::_State_base::_M_do_set() [23:05:27]W: [Step 3/3] @ 0x7fa210875e03 __pthread_once_internal [23:05:27]W: [Step 3/3] @ 0x7fa1e2220402 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv [23:05:27]W: [Step 3/3] @ 0x7fa1e0809074 _ZZN10ThreadPoolC1EmENKUlvE_clEv [23:05:27]W: [Step 3/3] @ 0x7fa204dce470 (unknown) [23:05:27]W: [Step 3/3] @ 0x7fa210870aa1 start_thread [23:05:27]W: [Step 3/3] @ 0x7fa20ff32bcd clone [23:05:27]W: [Step 3/3] @ (nil) (unknown) [23:05:27] : [Step 3/3] ['----------- Configuration Arguments -----------'] [23:05:27] : [Step 3/3] ['base_lr: 0.001'] [23:05:27] : [Step 3/3] ['batch_size: 2'] [23:05:27] : [Step 3/3] ['dataset_path: /root/.cache/paddle/dataset/cityscape/'] [23:05:27] : [Step 3/3] ['enable_ce: True'] [23:05:27] : [Step 3/3] ['init_weights_path: None'] [23:05:27] : [Step 3/3] ['load_logit_layer: True'] [23:05:27] : [Step 3/3] ['memory_optimize: True'] [23:05:27] : [Step 3/3] ['norm_type: bn'] [23:05:27] : [Step 3/3] ['num_classes: 19'] [23:05:27] : [Step 3/3] ['parallel: True'] [23:05:27] : [Step 3/3] ['profile: False'] [23:05:27] : [Step 3/3] ['save_weights_path: output1'] [23:05:27] : [Step 3/3] ['total_step: 50'] [23:05:27] : [Step 3/3] ['train_crop_size: 769'] [23:05:27] : [Step 3/3] ['use_gpu: True'] [23:05:27] : [Step 3/3] ['use_py_reader: True'] [23:05:27] : [Step 3/3] ['------------------------------------------------'] [23:05:27] : [Step 3/3] ['total number 2975'] [23:05:27] : [Step 3/3] ["You can install 'prefetch_generator' for acceleration of data reading."] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).'] [23:05:27] : [Step 3/3] ['Exception: /workspace/paddle/fluid/operators/cross_entropy_op.h:158 Assertion label >= 0 && label < feature_size_ failed (The label is out of the range. 255).']