Created by: gongweibao

[12:07:11]	[Step 1/1]  95/125 Test  #91: test_parallel_executor ..........................***Failed   21.92 sec
[12:07:11]	[Step 1/1] test_batchnorm_fc (test_parallel_executor.TestMNIST) ... Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/local/osd0/docker/overlay2/l/TMOSD4B44HN3GEC53NFUC5BN43:/var/local/osd0/docker/overlay2/l/U4WDH3Y2LWKMQDW7Y24YJW3SIB:/var/local/osd0/docker/overlay2/l/MMIJLSJ5AVQAR5LHS7QNANN2M3:/var/local/osd0/docker/overlay2/l/4FVA5X5PNK6K2OZ3NOBQOSQTLI:/var/local/osd0/docker/overlay2/l/5632XT4QWFIZ5D36AGH3U77423:/var/local/osd0/docker/overlay2/l/7R6XX4RRQ7A3M7G67HLJDVBIVC:/var/local/osd0/docker/overlay2/l/GN7QGVCODG2EB6TT55EJUYJOYS:/var/local/osd0/docker/overlay2/l/AKKHPLEYNBT'
[12:07:11]	[Step 1/1] Unexpected end of /proc/mounts line `XXSLD7BGOLKFXPV:/var/local/osd0/docker/overlay2/l/TFDMWNY6RZXNJNYYI6F7DKSHKY:/var/local/osd0/docker/overlay2/l/ACIFLN5D7CMTWFRDCVD3CVQUNP:/var/local/osd0/docker/overlay2/l/E4H35JWT6QPOHIBFX2MKVNCGLK:/var/local/osd0/docker/overlay2/l/RI6HYLPYTDKKSOJB2P3ILIRSTU:/var/local/osd0/docker/overlay2/l/JWA2ZHGCVUBKOGAVTFA4BVMF6A:/var/local/osd0/docker/overlay2/l/GIYKGNNAOF3ZCOSQNQCARLLCWT:/var/local/osd0/docker/overlay2/l/XWCPUMBKEPJDN2UV5DYFBK6FU3:/var/local/osd0/docker/overlay2/l/7RRVJENU5M24YVXWTS2VOCMNN3:/var/lo'
[12:07:11]	[Step 1/1] Unexpected end of /proc/mounts line `cal/osd0/docker/overlay2/l/T6JQVLYJ7GW5R2JQ26OZFTNTSY:/var/local/osd0/docker/overlay2/l/PJFRVPSM32SF57VNBVKZIGFEK4:/var/local/osd0/docker/overlay2/l/E6CFFCJL7L2CBK6NEXFYGXKWLH:/var/local/osd0/docker/overlay2/l/P7EZDHBUIQZORKHHQXZ3C7NMB5:/var/local/osd0/docker/overlay2/l/3UG5WY6A4WXFYDM7PEZC2ESGME:/var/local/osd0/docker/overlay2/l/ED5SIL2GZI3FSUKBKK45R7IHLN:/var/local/osd0/docker/overlay2/l/BVKDBS4LDZOMFYF42ODTWBZ7F7:/var/local/osd0/docker/overlay2/l/RUICJ424PWFBEGQ5BLPGM7GATB:/var/local/osd0/docker/overlay'
[12:07:11]	[Step 1/1] Unexpected end of /proc/mounts line `2/l/2GYE5K5NOY547AYBEJGGDZJAZP:/var/local/osd0/docker/overlay2/l/6ME7C3Q4WVSU3AFMVUVDR2CE4H:/var/local/osd0/docker/overlay2/l/FLNKN3DJXIR6ZIQ7EON2NPJ5KU:/var/local/osd0/docker/overlay2/l/ZIHSLFIWESAVOLK4NABPEBXFI5:/var/local/osd0/docker/overlay2/l/ZOE3VRRVOPWTCBIXS6GNCWLRFE:/var/local/osd0/docker/overlay2/l/WTS5NNZEYVLCVNE2HY2YTS6WTE,upperdir=/var/local/osd0/docker/overlay2/ec1d0cee4813cdee09643b02a44f851ceda2b9515ef674cb188e95598833917a/diff,workdir=/var/local/osd0/docker/overlay2/ec1d0cee4813cdee09643b02'
[12:07:11]	[Step 1/1] [4.7432337 4.5613756] [0.93494874 1.0563544 ]
[12:07:11]	[Step 1/1] ERROR
[12:07:11]	[Step 1/1] test_simple_fc (test_parallel_executor.TestMNIST) ... ERROR
[12:07:11]	[Step 1/1] test_resnet (test_parallel_executor.TestResnet) ... ERROR
[12:07:11]	[Step 1/1] test_main (test_parallel_executor.TestTransformer) ... skipped 'transformer is buggy in multi gpu'
[12:07:11]	[Step 1/1] 
[12:07:11]	[Step 1/1] ======================================================================
[12:07:11]	[Step 1/1] ERROR: test_batchnorm_fc (test_parallel_executor.TestMNIST)
[12:07:11]	[Step 1/1] ----------------------------------------------------------------------
[12:07:11]	[Step 1/1] Traceback (most recent call last):
[12:07:11]	[Step 1/1]   File "test_parallel_executor.py", line 266, in test_batchnorm_fc
[12:07:11]	[Step 1/1]     "label": label})
[12:07:11]	[Step 1/1]   File "test_parallel_executor.py", line 218, in check_network_convergence
[12:07:11]	[Step 1/1]     exe.run([], feed_dict=feed_dict)
[12:07:11]	[Step 1/1]   File "/paddle/build/python/paddle/fluid/parallel_executor.py", line 91, in run
[12:07:11]	[Step 1/1]     self.executor.run(fetch_list, fetch_var_name, feed_tensor_dict)
[12:07:11]	[Step 1/1] EnforceNotMet: an illegal memory access was encountered at [/paddle/paddle/fluid/framework/details/op_handle_base.cc:80]
[12:07:11]	[Step 1/1] PaddlePaddle Call Stacks: 
[12:07:11]	[Step 1/1] 0       0x7f310e0b2c9cp paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 572
[12:07:11]	[Step 1/1] 1       0x7f310ec9f11fp paddle::framework::details::OpHandleBase::Wait(paddle::platform::DeviceContext*) + 511
[12:07:11]	[Step 1/1] 2       0x7f310ec9088dp paddle::framework::details::ComputationOpHandle::RunImpl() + 205
[12:07:11]	[Step 1/1] 3       0x7f310ec9f679p paddle::framework::details::OpHandleBase::Run(bool) + 393
[12:07:11]	[Step 1/1] 4       0x7f310ec96a4ap
[12:07:11]	[Step 1/1] 5       0x7f310eb1791ep std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) + 46
[12:07:11]	[Step 1/1] 6       0x7f315484ca99p
[12:07:11]	[Step 1/1] 7       0x7f310ec94b0dp
[12:07:11]	[Step 1/1] 8       0x7f310ec9a954p std::thread::_Impl<std::_Bind_simple<ThreadPool::ThreadPool(unsigned long)::{lambda()#1} ()> >::_M_run() + 340
[12:07:11]	[Step 1/1] 9       0x7f314f0f0c80p
[12:07:11]	[Step 1/1] 10      0x7f31548456bap
[12:07:11]	[Step 1/1] 11      0x7f315457b41dp clone + 109
[12:07:11]	[Step 1/1] 
[12:07:11]	[Step 1/1] 
[12:07:11]	[Step 1/1] ======================================================================
[12:07:11]	[Step 1/1] ERROR: test_simple_fc (test_parallel_executor.TestMNIST)
[12:07:11]	[Step 1/1] ----------------------------------------------------------------------
[12:07:11]	[Step 1/1] Traceback (most recent call last):
[12:07:11]	[Step 1/1]   File "test_parallel_executor.py", line 251, in test_simple_fc
[12:07:11]	[Step 1/1]     self.check_network_convergence(simple_fc_net)
[12:07:11]	[Step 1/1]   File "test_parallel_executor.py", line 210, in check_network_convergence
[12:07:11]	[Step 1/1]     exe = fluid.ParallelExecutor(loss_name=loss.name, use_cuda=True)
[12:07:11]	[Step 1/1]   File "/paddle/build/python/paddle/fluid/parallel_executor.py", line 69, in __init__
[12:07:11]	[Step 1/1]     allow_op_delay)
[12:07:11]	[Step 1/1] RuntimeError: function_attributes(): after cudaFuncGetAttributes: an illegal memory access was encountered
[12:07:11]	[Step 1/1] 
[12:07:11]	[Step 1/1] ======================================================================
[12:07:11]	[Step 1/1] ERROR: test_resnet (test_parallel_executor.TestResnet)
[12:07:11]	[Step 1/1] ----------------------------------------------------------------------
[12:07:11]	[Step 1/1] Traceback (most recent call last):
[12:07:11]	[Step 1/1]   File "test_parallel_executor.py", line 295, in test_resnet
[12:07:11]	[Step 1/1]     batch_size=batch_size)
[12:07:11]	[Step 1/1]   File "test_parallel_executor.py", line 210, in check_network_convergence
[12:07:11]	[Step 1/1]     exe = fluid.ParallelExecutor(loss_name=loss.name, use_cuda=True)
[12:07:11]	[Step 1/1]   File "/paddle/build/python/paddle/fluid/parallel_executor.py", line 69, in __init__
[12:07:11]	[Step 1/1]     allow_op_delay)
[12:07:11]	[Step 1/1] RuntimeError: function_attributes(): after cudaFuncGetAttributes: an illegal memory access was encountered
[12:07:11]	[Step 1/1] 
[12:07:11]	[Step 1/1] ----------------------------------------------------------------------
[12:07:11]	[Step 1/1] Ran 4 tests in 11.272s