Control GPU memory usage in unit tests
Created by: wangkuiyi
@emailweixu found that tests can fail for memory allocation if running them in parallel ctest -I 121,123 -j
:
I found that tests can fail for memory allocation if running them in parallel: ctest -I 121,123 -j
121: Traceback (most recent call last):
121: File "/home/wei/code/baidu/idl/dl/robot/external/paddle/python/paddle/v2/framework/tests/op_test_util.py", line 37, in test_all
121: var.set(arr, place)
121: RuntimeError: Insufficient GPU memory to allocation. at [/home/wei/code/baidu/idl/dl/robot/external/paddle/paddle/framework/tensor.h:131]
121: Call Stacks:
121: /home/wei/code/baidu/idl/dl/robot/build/.env/local/lib/python2.7/site-packages/paddle/v2/framework/core.so(_ZN6paddle8platform13EnforceNotMetC1ENSt15__exception_ptr13exception_ptrEPKci+0x1d2) [0x7fde36115fc2]
121: /home/wei/code/baidu/idl/dl/robot/build/.env/local/lib/python2.7/site-packages/paddle/v2/framework/core.so(_ZN6paddle9framework6Tensor15PlaceholderImplIfNS_8platform8GPUPlaceEEC1ES4_m+0x130) [0x7fde36136824]
121: /home/wei/code/baidu/idl/dl/robot/build/.env/local/lib/python2.7/site-packages/paddle/v2/framework/core.so(_ZN6paddle9framework6Tensor12mutable_dataIfEEPT_N5boost7variantINS_8platform8GPUPlaceEJNS7_8CPUPlaceEEEE+0x1b1) [0x7fde3612bbf9]
....
1/3 Test #123: test_softmax_op .................. Passed 1.31 sec
2/3 Test #122: test_sigmoid_op ..................***Failed 1.40 sec
3/3 Test #121: test_add_two_op ..................***Failed 1.42 sec