paddlecloud训练运行的时候报错,但是本地能够跑通。
Created by: tangxuvis
报错信息如下: I1211 17:55:12.311203 622 operator.cc:172] CUDAPlace(3) Op(fill_constant), inputs:{ShapeTensor[], ShapeTensorList[]}, outputs:{Out[fill_constant_5.tmp_0:float1]}. I1211 17:55:12.311429 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:library_type[PLAIN] I1211 17:55:12.311679 622 operator.cc:172] CUDAPlace(0) Op(scale), inputs:{X[conv2d_31.w_0:float256, 256, 1, 1]}, outputs:{Out[_generated_var_89:float256, 256, 1, 1]}. I1211 17:55:12.311764 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] W1211 17:55:12.362390 628 init.cc:205] *** Aborted at 1576058112 (unix time) try "date -d @1576058112" if you are using GNU date *** W1211 17:55:12.364377 628 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:12.364996 628 init.cc:205] *** SIGSEGV (@0x9) received by PID 628 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:12.366531 628 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:12.368046 628 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:12.369441 628 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:12.370069 628 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:12.370676 628 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:12.371279 628 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:12.371882 628 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:12.372488 628 init.cc:205] @ 0x7f207cd28786 (unknown) W1211 17:55:12.373083 628 init.cc:205] @ 0x7f207cd4f3ba (unknown) W1211 17:55:12.373685 628 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:12.374284 628 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:12.375697 628 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:12.377102 628 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.378505 628 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.379904 628 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.381297 628 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.382711 628 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:12.384109 628 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:12.385510 628 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.386914 628 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:12.388299 628 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.389699 628 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:12.391093 628 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:12.392491 628 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:12.394089 628 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.395531 628 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:12.396924 628 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.398335 628 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:12.399725 628 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.401141 628 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:12.402559 628 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:12.403947 628 init.cc:205] @ 0x7f2100121d33 PyObject_Call I1211 17:55:12.443738 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[conv2d_18.w_0:float64, 48, 3, 3]}, outputs:{Out[_generated_var_70:float64, 48, 3, 3]}. I1211 17:55:12.443996 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(2)]:library_type[PLAIN] I1211 17:55:12.444236 622 operator.cc:172] CUDAPlace(2) Op(scale), inputs:{X[batch_norm_26.b_0:float512]}, outputs:{Out[_generated_var_42:float512]}. I1211 17:55:12.444309 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] W1211 17:55:12.484035 650 init.cc:205] *** Aborted at 1576058112 (unix time) try "date -d @1576058112" if you are using GNU date *** W1211 17:55:12.485893 650 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:12.486470 650 init.cc:205] *** SIGSEGV (@0x9) received by PID 650 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:12.487954 650 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:12.489465 650 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:12.490861 650 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:12.491494 650 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:12.492100 650 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:12.492702 650 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:12.493301 650 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:12.493901 650 init.cc:205] @ 0x7f207cd28786 (unknown) W1211 17:55:12.494501 650 init.cc:205] @ 0x7f207cd4f3ba (unknown) W1211 17:55:12.495100 650 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:12.495697 650 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:12.497109 650 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:12.498520 650 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.499915 650 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.501315 650 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.502728 650 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.504143 650 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:12.505547 650 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:12.506945 650 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.508356 650 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:12.509749 650 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.511144 650 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:12.512547 650 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:12.513947 650 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:12.515350 650 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.516772 650 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:12.518162 650 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.519579 650 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:12.520968 650 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.522382 650 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:12.523798 650 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:12.525187 650 init.cc:205] @ 0x7f2100121d33 PyObject_Call I1211 17:55:12.582463 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[conv2d_10.w_0:float64, 64, 3, 3]}, outputs:{Out[_generated_var_60:float64, 64, 3, 3]}. I1211 17:55:12.582695 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:library_type[PLAIN] W1211 17:55:12.719278 672 init.cc:205] *** Aborted at 1576058112 (unix time) try "date -d @1576058112" if you are using GNU date *** I1211 17:55:12.720058 622 operator.cc:172] CUDAPlace(0) Op(scale), inputs:{X[batch_norm_7.w_0:float64]}, outputs:{Out[_generated_var_53:float64]}. I1211 17:55:12.720321 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(2)]:library_type[PLAIN] W1211 17:55:12.721171 672 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:12.721756 672 init.cc:205] *** SIGSEGV (@0x9) received by PID 672 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:12.723291 672 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:12.725076 672 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:12.726680 672 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:12.727449 672 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:12.728113 672 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:12.728726 672 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:12.729429 672 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:12.730069 672 init.cc:205] @ 0x7f207cd28786 (unknown) W1211 17:55:12.730753 672 init.cc:205] @ 0x7f207cd4f3ba (unknown) W1211 17:55:12.731379 672 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:12.731995 672 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:12.733599 672 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:12.735080 672 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.736599 672 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.737996 672 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.739392 672 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.740803 672 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:12.742198 672 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:12.743600 672 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.745007 672 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:12.746392 672 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.747792 672 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:12.749187 672 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:12.750586 672 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:12.751983 672 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.753391 672 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:12.754781 672 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.756186 672 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:12.757575 672 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:12.758985 672 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:12.760396 672 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:12.761791 672 init.cc:205] @ 0x7f2100121d33 PyObject_Call I1211 17:55:12.853328 622 operator.cc:172] CUDAPlace(2) Op(scale), inputs:{X[batch_norm_18.w_0:float64]}, outputs:{Out[_generated_var_25:float64]}. I1211 17:55:12.853631 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] W1211 17:55:12.972546 696 init.cc:205] *** Aborted at 1576058112 (unix time) try "date -d @1576058112" if you are using GNU date *** W1211 17:55:12.974452 696 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:12.975075 696 init.cc:205] *** SIGSEGV (@0x9) received by PID 696 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:12.976712 696 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:12.978221 696 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:12.979625 696 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:12.980258 696 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:12.980868 696 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:12.981479 696 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:12.982089 696 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:12.982697 696 init.cc:205] @ 0x7f207cd42e0e (unknown) W1211 17:55:12.983304 696 init.cc:205] @ 0x7f207cd4f952 (unknown) W1211 17:55:12.983908 696 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:12.984519 696 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:12.985926 696 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:12.987339 696 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.988754 696 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:12.990162 696 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.991583 696 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx I1211 17:55:12.991716 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[conv2d_34.w_0:float12, 256, 3, 3]}, outputs:{Out[_generated_var_94:float12, 256, 3, 3]}. I1211 17:55:12.992015 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(1)]:library_type[PLAIN] I1211 17:55:12.992238 622 operator.cc:172] CUDAPlace(1) Op(scale), inputs:{X[conv2d_6.w_0:float48, 128, 1, 1]}, outputs:{Out[_generated_var_109:float48, 128, 1, 1]}. I1211 17:55:12.992311 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(1)]:library_type[PLAIN] W1211 17:55:12.993003 696 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:12.994412 696 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:12.995960 696 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:12.997572 696 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:12.999138 696 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.000613 696 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:13.002121 696 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.003646 696 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.005187 696 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.006615 696 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:13.008008 696 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.009418 696 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:13.010816 696 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.012233 696 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:13.013658 696 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:13.015048 696 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.022902 694 init.cc:205] *** Aborted at 1576058113 (unix time) try "date -d @1576058113" if you are using GNU date *** W1211 17:55:13.024642 694 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:13.025174 694 init.cc:205] *** SIGSEGV (@0x9) received by PID 694 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:13.026650 694 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:13.028143 694 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:13.029549 694 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:13.030181 694 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:13.030792 694 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:13.031395 694 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:13.031999 694 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:13.032610 694 init.cc:205] @ 0x7f207cd28786 (unknown) W1211 17:55:13.033216 694 init.cc:205] @ 0x7f207cd4f3ba (unknown) W1211 17:55:13.033826 694 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:13.034427 694 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:13.035840 694 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:13.037241 694 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.038650 694 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.040050 694 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.041453 694 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.042866 694 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:13.044262 694 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:13.045668 694 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.047080 694 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:13.048499 694 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.049911 694 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:13.051317 694 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.052724 694 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.054131 694 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.055550 694 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:13.056944 694 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.058351 694 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:13.059746 694 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.061162 694 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:13.062578 694 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:13.063968 694 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.110980 735 init.cc:205] *** Aborted at 1576058113 (unix time) try "date -d @1576058113" if you are using GNU date *** W1211 17:55:13.112923 735 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:13.113534 735 init.cc:205] *** SIGSEGV (@0x9) received by PID 735 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:13.115048 735 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:13.116569 735 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:13.117972 735 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:13.118619 735 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:13.119230 735 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:13.119845 735 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:13.120455 735 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:13.121073 735 init.cc:205] @ 0x7f207cd28786 (unknown) W1211 17:55:13.121703 735 init.cc:205] @ 0x7f207cd4f3ba (unknown) W1211 17:55:13.122306 735 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:13.122920 735 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:13.124356 735 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:13.125798 735 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx I1211 17:55:13.125938 622 operator.cc:172] CUDAPlace(1) Op(scale), inputs:{X[conv2d_40.w_0:float4, 512, 3, 3]}, outputs:{Out[_generated_var_107:float4, 512, 3, 3]}. I1211 17:55:13.126168 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] W1211 17:55:13.127216 735 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.128633 735 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.130105 735 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.131709 735 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:13.133247 735 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:13.134697 735 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.136133 735 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:13.137666 735 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.139086 735 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:13.140482 735 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.141882 735 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.143277 735 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.144693 735 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:13.146116 735 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.147578 735 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:13.149005 735 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.150425 735 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:13.151850 735 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:13.153240 735 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.237246 760 init.cc:205] *** Aborted at 1576058113 (unix time) try "date -d @1576058113" if you are using GNU date *** W1211 17:55:13.239156 760 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:13.239750 760 init.cc:205] *** SIGSEGV (@0x9) received by PID 760 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:13.241266 760 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:13.242796 760 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:13.244195 760 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:13.244833 760 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:13.245434 760 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:13.246037 760 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:13.246639 760 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:13.247236 760 init.cc:205] @ 0x7f207cd42e0e (unknown) W1211 17:55:13.247836 760 init.cc:205] @ 0x7f207cd4f952 (unknown) W1211 17:55:13.248435 760 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:13.249037 760 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:13.250524 760 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:13.251935 760 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.253342 760 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.254750 760 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.256146 760 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.257593 760 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:13.258999 760 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:13.260442 760 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx I1211 17:55:13.260857 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[batch_norm_1.b_0:float48]}, outputs:{Out[_generated_var_6:float48]}. I1211 17:55:13.261041 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] I1211 17:55:13.261111 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[conv2d_29.b_0:float16]}, outputs:{Out[_generated_var_84:float16]}. I1211 17:55:13.261214 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:library_type[PLAIN] W1211 17:55:13.261864 760 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:13.263255 760 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.264741 760 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:13.266295 760 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.267863 760 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.269294 760 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.270781 760 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:13.272264 760 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.273727 760 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:13.275120 760 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.276561 760 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:13.277989 760 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:13.279374 760 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.378332 782 init.cc:205] *** Aborted at 1576058113 (unix time) try "date -d @1576058113" if you are using GNU date *** W1211 17:55:13.380215 782 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:13.380817 782 init.cc:205] *** SIGSEGV (@0x9) received by PID 782 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:13.382308 782 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:13.383821 782 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:13.385221 782 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:13.385864 782 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:13.386479 782 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:13.387085 782 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:13.387694 782 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:13.388296 782 init.cc:205] @ 0x7f207cd42e0e (unknown) W1211 17:55:13.388907 782 init.cc:205] @ 0x7f207cd4f952 (unknown) W1211 17:55:13.389521 782 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:13.390131 782 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:13.391556 782 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:13.392962 782 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.394374 782 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx I1211 17:55:13.395077 622 operator.cc:172] CUDAPlace(0) Op(scale), inputs:{X[conv2d_6.w_0:float48, 128, 1, 1]}, outputs:{Out[_generated_var_109:float48, 128, 1, 1]}. I1211 17:55:13.395263 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(2)]:library_type[PLAIN] I1211 17:55:13.395380 622 operator.cc:172] CUDAPlace(2) Op(scale), inputs:{X[conv2d_4.w_0:float64, 128, 1, 1]}, outputs:{Out[_generated_var_105:float64, 128, 1, 1]}. I1211 17:55:13.395421 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(2)]:library_type[PLAIN] I1211 17:55:13.395471 622 operator.cc:172] CUDAPlace(2) Op(scale), inputs:{X[batch_norm_5.w_0:float64]}, outputs:{Out[_generated_var_49:float64]}. I1211 17:55:13.395524 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] I1211 17:55:13.395608 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[batch_norm_18.b_0:float64]}, outputs:{Out[_generated_var_24:float64]}. I1211 17:55:13.395643 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] I1211 17:55:13.395680 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[batch_norm_10.w_0:float64]}, outputs:{Out[_generated_var_9:float64]}. I1211 17:55:13.395716 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(2)]:library_type[PLAIN] I1211 17:55:13.395752 622 operator.cc:172] CUDAPlace(2) Op(scale), inputs:{X[batch_norm_24.b_0:float64]}, outputs:{Out[_generated_var_38:float64]}. W1211 17:55:13.395790 782 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx I1211 17:55:13.395799 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(1)]:library_type[PLAIN] I1211 17:55:13.395913 622 operator.cc:172] CUDAPlace(1) Op(fill_constant), inputs:{ShapeTensor[], ShapeTensorList[]}, outputs:{Out[fill_constant_1.tmp_0:float1]}. I1211 17:55:13.396031 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:library_type[PLAIN] I1211 17:55:13.396078 622 operator.cc:172] CUDAPlace(0) Op(scale), inputs:{X[conv2d_33.w_0:float12, 256, 3, 3]}, outputs:{Out[_generated_var_92:float12, 256, 3, 3]}. I1211 17:55:13.396116 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:library_type[PLAIN] I1211 17:55:13.396153 622 operator.cc:172] CUDAPlace(0) Op(scale), inputs:{X[conv2d_21.b_0:float256]}, outputs:{Out[_generated_var_75:float256]}. I1211 17:55:13.396193 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(2)]:library_type[PLAIN] I1211 17:55:13.396227 622 operator.cc:172] CUDAPlace(2) Op(scale), inputs:{X[conv2d_38.b_0:float8]}, outputs:{Out[_generated_var_101:float8]}. I1211 17:55:13.396266 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:library_type[PLAIN] I1211 17:55:13.396302 622 operator.cc:172] CUDAPlace(0) Op(scale), inputs:{X[batch_norm_18.w_0:float64]}, outputs:{Out[_generated_var_25:float64]}. I1211 17:55:13.396343 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] I1211 17:55:13.396378 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[batch_norm_11.w_0:float64]}, outputs:{Out[_generated_var_11:float64]}. I1211 17:55:13.396467 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(1)]:library_type[PLAIN] I1211 17:55:13.396525 622 operator.cc:172] CUDAPlace(1) Op(scale), inputs:{X[conv2d_9.w_0:float64, 48, 3, 3]}, outputs:{Out[_generated_var_112:float64, 48, 3, 3]}. I1211 17:55:13.396566 622 operator.cc:992] expected_kernel_key:data_type[float]:data_layout[ANY_LAYOUT]:place[CUDAPlace(3)]:library_type[PLAIN] I1211 17:55:13.396605 622 operator.cc:172] CUDAPlace(3) Op(scale), inputs:{X[batch_norm_26.b_0:float512]}, outputs:{Out[_generated_var_42:float512]}. I1211 17:55:13.396651 622 operator.cc:992] expected_kernel_key:data_type[int64_t]:data_layout[ANY_LAYOUT]:place[CPUPlace]:library_type[PLAIN] I1211 17:55:13.396692 622 operator.cc:172] CUDAPlace(0) Op(increment), inputs:{X[@LR_DECAY_COUNTER@:int64_t1]}, outputs:{Out[@LR_DECAY_COUNTER@:int64_t1]}. I1211 17:55:13.396716 622 operator.cc:992] expected_kernel_key:data_type[int64_t]:data_layout[ANY_LAYOUT]:place[CPUPlace]:library_type[PLAIN] I1211 17:55:13.396775 622 operator.cc:172] CUDAPlace(0) Op(cast), inputs:{X[@LR_DECAY_COUNTER@:int64_t1]}, outputs:{Out[cast_0.tmp_0:float1]}. I1211 17:55:13.396930 622 buffered_reader.cc:50] BufferedReader W1211 17:55:13.397194 782 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx I1211 17:55:13.398283 622 operator.cc:172] CUDAPlace(0) Op(create_double_buffer_reader), inputs:{UnderlyingReader[create_py_reader_0:-1]}, outputs:{Out[double_buffer_0:-1]}. I1211 17:55:13.398336 622 read_op.cc:76] read op in W1211 17:55:13.398720 782 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:13.400353 782 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:13.401932 782 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.403434 782 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:13.404969 782 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.406554 782 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:13.408030 782 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.409423 782 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.410823 782 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.412236 782 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:13.413623 782 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.415031 782 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:13.416517 782 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.417955 782 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:13.419382 782 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:13.420821 782 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.451467 804 init.cc:205] *** Aborted at 1576058113 (unix time) try "date -d @1576058113" if you are using GNU date *** W1211 17:55:13.453516 804 init.cc:205] PC: @ 0x0 (unknown) W1211 17:55:13.454082 804 init.cc:205] *** SIGSEGV (@0x9) received by PID 804 (TID 0x7f209b099700) from PID 9; stack trace: *** W1211 17:55:13.455580 804 init.cc:205] @ 0x7f20ffec0c20 (unknown) W1211 17:55:13.457093 804 init.cc:205] @ 0x7f20ff473c9c __GI___libc_free W1211 17:55:13.458501 804 init.cc:205] @ 0x7f20ffeb8ab9 __pthread_create_2_1 W1211 17:55:13.459131 804 init.cc:205] @ 0x7f207c9b41dd (unknown) W1211 17:55:13.459743 804 init.cc:205] @ 0x7f207c9b532f (unknown) W1211 17:55:13.460345 804 init.cc:205] @ 0x7f207c9b5c30 (unknown) W1211 17:55:13.460950 804 init.cc:205] @ 0x7f207c9b1bf3 (unknown) W1211 17:55:13.461555 804 init.cc:205] @ 0x7f207cd42e0e (unknown) W1211 17:55:13.462154 804 init.cc:205] @ 0x7f207cd4f952 (unknown) W1211 17:55:13.462755 804 init.cc:205] @ 0x7f207cd536cf (unknown) W1211 17:55:13.463356 804 init.cc:205] @ 0x7f207c6b0d95 (unknown) W1211 17:55:13.464768 804 init.cc:205] @ 0x7f21001d9ce8 PyEval_EvalFrameEx W1211 17:55:13.466183 804 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.467594 804 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.468997 804 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.470397 804 init.cc:205] @ 0x7f21001d9d70 PyEval_EvalFrameEx W1211 17:55:13.471813 804 init.cc:205] @ 0x7f21001480b3 (unknown) W1211 17:55:13.473213 804 init.cc:205] @ 0x7f21001d58b2 PyEval_EvalFrameEx W1211 17:55:13.474619 804 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.476028 804 init.cc:205] @ 0x7f2100153905 (unknown) W1211 17:55:13.477413 804 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.478818 804 init.cc:205] @ 0x7f21001d70a2 PyEval_EvalFrameEx W1211 17:55:13.480216 804 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.481612 804 init.cc:205] @ 0x7f21001d9e9e PyEval_EvalFrameEx W1211 17:55:13.483016 804 init.cc:205] @ 0x7f21001dc37d PyEval_EvalCodeEx W1211 17:55:13.484422 804 init.cc:205] @ 0x7f2100153830 (unknown) W1211 17:55:13.485822 804 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.487231 804 init.cc:205] @ 0x7f210013074d (unknown) W1211 17:55:13.488626 804 init.cc:205] @ 0x7f2100121d33 PyObject_Call W1211 17:55:13.490036 804 init.cc:205] @ 0x7f210018d36f (unknown) W1211 17:55:13.491452 804 init.cc:205] @ 0x7f210018beff (unknown) W1211 17:55:13.492841 804 init.cc:205] @ 0x7f2100121d33 PyObject_Call