Created by: ccmeteorljh

object_detection 模型跑了14个pass出现下面错误： paddle_version: develop 9月5号编译版本 cloud地址： http://paddlecloud.baidu-int.com:8088/paddle/jobRunInfo?jobId=job-e6c5b98d4926999f&flag=jobs&groupName=k8s_gpu_demo&groupId=c0a1f165-6279-5320-b9e7-e0218c7a87f5&currentPage=1&currentKey=1

Epoc 14, batch 140, loss 2.786850214, time 13.3026299477
Batch 0, map [0.76196724]
Batch 10, map [0.7178482]
Batch 20, map [0.72573954]
Batch 30, map [0.72100013]
Epoc 14, test map [0.71945566]
save models to model/best_model
Best test map 0.719455659389
*** Error in `python': malloc(): memory corruption (fast): 0x00007f74a0a0e830 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f76a2fb47e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x82651)[0x7f76a2fbf651]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f76a2fc1184]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7f769936ae78]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNSt6vectorISsSaISsEEC2ERKS1_+0xbf)[0x7f764498514f]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNSt8_Rb_treeISsSt4pairIKSsSt6vectorISsSaISsEEESt10_Select1stIS5_ESt4lessISsESaIS5_EE14_M_create_nodeIJRKS5_EEEPSt13_Rb_tree_nodeIS5_EDpOT_+0x58)[0x7f764498cd98]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNSt8_Rb_treeISsSt4pairIKSsSt6vectorISsSaISsEEESt10_Select1stIS5_ESt4lessISsESaIS5_EE7_M_copyEPKSt13_Rb_tree_nodeIS5_EPSD_+0x23)[0x7f764498ce13]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework12OperatorBaseC2ERKSsRKSt3mapISsSt6vectorISsSaISsEESt4lessISsESaISt4pairIS2_S7_EEESF_RKSt13unordered_mapISsN5boost7variantINSH_5blankEifSsS5_IiSaIiEES5_IfSaIfEES7_bS5_IbSaIbEEPNS0_9BlockDescElS5_ISR_SaISR_EENSH_6detail7variant5void_ESW_SW_SW_SW_SW_SW_SW_EESt4hashISsESt8equal_toISsESaISA_IS2_SX_EEE+0x6d)[0x7f7645af1a4d]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNSt17_Function_handlerIFPN6paddle9framework12OperatorBaseERKSsRKSt3mapISsSt6vectorISsSaISsEESt4lessISsESaISt4pairIS4_S9_EEESH_RKSt13unordered_mapISsN5boost7variantINSJ_5blankEifSsS7_IiSaIiEES7_IfSaIfEES9_bS7_IbSaIbEEPNS1_9BlockDescElS7_IST_SaIST_EENSJ_6detail7variant5void_ESY_SY_SY_SY_SY_SY_SY_EESt4hashISsESt8equal_toISsESaISC_IS4_SZ_EEEEZNKS1_7details12OpInfoFillerINS0_9operators8AssignOpELNS1A_14OpInfoFillTypeE0EEclEPKcPNS1_6OpInfoEEUlS5_SH_SH_S18_E_E9_M_invokeERKSt9_Any_dataS5_SH_SH_S18_+0x3e)[0x7f7644f8083e]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework10OpRegistry8CreateOpERKSsRKSt3mapISsSt6vectorISsSaISsEESt4lessISsESaISt4pairIS2_S7_EEESF_St13unordered_mapISsN5boost7variantINSH_5blankEifSsS5_IiSaIiEES5_IfSaIfEES7_bS5_IbSaIbEEPNS0_9BlockDescElS5_ISR_SaISR_EENSH_6detail7variant5void_ESW_SW_SW_SW_SW_SW_SW_EESt4hashISsESt8equal_toISsESaISA_IS2_SX_EEE+0x1c9)[0x7f7645ae7d69]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework10OpRegistry8CreateOpERKNS0_6OpDescE+0x66)[0x7f7645ae7ee6]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework8Executor7PrepareERKNS0_11ProgramDescEi+0xdc)[0x7f7644a4dd1c]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework8Executor3RunERKNS0_11ProgramDescEPNS0_5ScopeEibb+0x61)[0x7f7644a4e1e1]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNK6paddle9operators18ConditionalBlockOp7RunImplERKNS_9framework5ScopeERKN5boost7variantINS_8platform9CUDAPlaceENS8_8CPUPlaceENS8_15CUDAPinnedPlaceENS6_6detail7variant5void_ESE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_SE_EE+0x300)[0x7f7645452f80]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework12OperatorBase3RunERKNS0_5ScopeERKN5boost7variantINS_8platform9CUDAPlaceENS7_8CPUPlaceENS7_15CUDAPinnedPlaceENS5_6detail7variant5void_ESD_SD_SD_SD_SD_SD_SD_SD_SD_SD_SD_SD_SD_SD_SD_SD_EE+0xfc)[0x7f7645aeceac]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(+0x1c20597)[0x7f764592d597]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(+0x1c4b690)[0x7f7645958690]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework7details12OpHandleBase17RunAndRecordEventERKSt8functionIFvvEE+0x325)[0x7f7645957ef5]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework7details19ComputationOpHandle7RunImplEv+0x5f)[0x7f764592d06f]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZN6paddle9framework7details12OpHandleBase3RunEb+0x75)[0x7f7645958f95]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(+0x1bfedba)[0x7f764590bdba]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIvEES3_EvEEE9_M_invokeERKSt9_Any_data+0x23)[0x7f76457899f3]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZNSt13__future_base11_State_base9_M_do_setERSt8functionIFSt10unique_ptrINS_12_Result_baseENS3_8_DeleterEEvEERb+0x27)[0x7f7644af5b47]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xea99)[0x7f76a3315a99]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(+0x1bfda12)[0x7f764590aa12]
/usr/local/lib/python2.7/site-packages/paddle/fluid/core.so(_ZZN10ThreadPoolC1EmENKUlvE_clEv+0x194)[0x7f7644af7684]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f7699395c80]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f76a330e6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f76a304441d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 103:0b 145757879                         /usr/local/bin/python2.7
00600000-00601000 r--p 00000000 103:0b 145757879                         /usr/local/bin/python2.7
00601000-00602000 rw-p 00001000 103:0b 145757879                         /usr/local/bin/python2.7
023b5000-12ac64000 rw-p 00000000 00:00 0                                 [heap]
200000000-200200000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0 
200400000-200404000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0 
200600000-200a00000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
200a00000-201800000 ---p 00000000 00:00 0 
201800000-201804000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
201804000-201a00000 ---p 00000000 00:00 0 
201a00000-201e00000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
201e00000-202c00000 ---p 00000000 00:00 0 
202c00000-202c04000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
202c04000-202e00000 ---p 00000000 00:00 0 
202e00000-203200000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
203200000-204000000 ---p 00000000 00:00 0 
204000000-204004000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
204004000-204200000 ---p 00000000 00:00 0 
204200000-204600000 rw-s 00000000 00:28 3944625322                       /dev/nvidiactl
204600000-205400000 ---p 00000000 00:00 0