elementwise_mul raises an exception thrust::system::system_error
Created by: Akeepers
-
版本、环境信息: 1)PaddlePaddle版本:PaddlePaddle 1.8.1.post107 2)系统环境:单机单卡训练,Python 3.5.2
-
使用的ERNIE2.0代码做特定任务的fine-tune实验,如果在网络(静态图)中加入以下代码:
其中np_event_emb是一个shape为size=(args.num_trigger_labels, ernie_config['hidden_size'])的numpy array
event_emb = fluid.embedding(
input=event_type_ids,
size=(args.num_trigger_labels, ernie_config['hidden_size']),
param_attr=fluid.ParamAttr(
name="emb_weight",
initializer=fluid.initializer.NumpyArrayInitializer(
np_event_emb),
trainable=False),
dtype='float32') # (batch_size, hidden_dim)
错误信息如下:
[INFO] 2020-06-22 15:10:09,054 [ init.py: 88]: Load pretraining parameters from /mnt/du/yangpan03/pretrain_models/ernie_large/params.
W0622 15:10:11.955546 23037 operator.cc:187] elementwise_mul raises an exception thrust::system::system_error, parallel_for failed: unspecified launch failure
F0622 15:10:11.955682 23037 exception_holder.h:37] std::exception caught, parallel_for failed: unspecified launch failure
*** Check failure stack trace: ***
@ 0x7fba257107dd google::LogMessage::Fail()
@ 0x7fba2571428c google::LogMessage::SendToLog()
@ 0x7fba25710303 google::LogMessage::Flush()
@ 0x7fba2571579e google::LogMessageFatal::~LogMessageFatal()
@ 0x7fba288bf828 paddle::framework::details::ExceptionHolder::Catch()
@ 0x7fba2895a91e paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync()
@ 0x7fba289582bf paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp()
@ 0x7fba28958584 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data
@ 0x7fba2576dca3 std::_Function_handler<>::_M_invoke()
@ 0x7fba2556b237 std::__future_base::_State_base::_M_do_set()
@ 0x7fbb06659a99 __pthread_once_slow
@ 0x7fba28954752 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv
@ 0x7fba2556d694 _ZZN10ThreadPoolC1EmENKUlvE_clEv
@ 0x7fba60892c80 (unknown)
@ 0x7fbb066526ba start_thread
@ 0x7fbb0638841d clone
@ (nil) (unknown)
./train_ee_multi.sh: line 54: 22448 Aborted