自己写operator,Check_Grad出现问题,希望帮忙解决一下
Created by: zzhzz
给自己的operator做unittest,得到错误信息如下:
: E. 63707 357: ====================================================================== 63708 357: ERROR: test_check_grad (test_tree_conv_op.TestTreeConvOp) 63709 357: ---------------------------------------------------------------------- 63710 357: Traceback (most recent call last): 63711 357: File "/mnt/Paddle/build/python/paddle/fluid/tests/unittests/test_tree_conv_op.py", line 79, in test_check_grad 63712 357: self.check_grad(['NodesVector', 'Filter'], 'Out', max_relative_error=0.5) 63713 357: File "/mnt/Paddle/build/python/paddle/fluid/tests/unittests/op_test.py", line 413, in check_grad 63714 357: user_defined_grads) 63715 357: File "/mnt/Paddle/build/python/paddle/fluid/tests/unittests/op_test.py", line 449, in check_grad_with_place 63716 357: output_names, no_grad_set) 63717 357: File "/mnt/Paddle/build/python/paddle/fluid/tests/unittests/op_test.py", line 527, in _get_gradient 63718 357: executor.run(prog, feed_dict, fetch_list, return_numpy=False))) 63719 357: File "/mnt/Paddle/build/python/paddle/fluid/executor.py", line 470, in run 63720 357: self.executor.run(program.desc, scope, 0, True, True) 63721 357: ValueError: Invalid index
打印Paddle的Log,在报错附近的Log如下:
63681 357: I1027 04:09:31.935364 62382 operator.cc:143] CUDAPlace(0) Op(feed), inputs:{X[feed:-1]}, outputs:{Out[EdgeSet-1]}. 63682 357: I1027 04:09:31.935484 62382 feed_op.cc:50] Feed Var feed's 2 column to var EdgeSet 63683 357: I1027 04:09:31.935643 62382 operator.cc:155] CUDAPlace(0) Op(feed), inputs:{X[feed:-1]}, outputs:{Out[EdgeSet1, 16, 2]}. 63684 357: I1027 04:09:31.935760 62382 operator.cc:143] CUDAPlace(0) Op(feed), inputs:{X[feed:-1]}, outputs:{Out[NodesVector-1]}. 63685 357: I1027 04:09:31.935853 62382 feed_op.cc:50] Feed Var feed's 1 column to var NodesVector 63686 357: I1027 04:09:31.935986 62382 operator.cc:155] CUDAPlace(0) Op(feed), inputs:{X[feed:-1]}, outputs:{Out[NodesVector1, 17, 3 0]}. 63687 357: I1027 04:09:31.936120 62382 operator.cc:143] CUDAPlace(0) Op(feed), inputs:{X[feed:-1]}, outputs:{Out[Filter-1]}. 63688 357: I1027 04:09:31.936219 62382 feed_op.cc:50] Feed Var feed's 0 column to var Filter 63689 357: I1027 04:09:31.936372 62382 operator.cc:155] CUDAPlace(0) Op(feed), inputs:{X[feed:-1]}, outputs:{Out[Filter[30, 3, 10, 1] ({})]}. 63690 357: I1027 04:09:31.936532 62382 operator.cc:143] CUDAPlace(0) Op(tree_conv), inputs:{EdgeSet[EdgeSet:int1, 16, 2], Filter[Filter :double30, 3, 10, 1], NodesVector[NodesVector:double1, 17, 30]}, outputs:{Out[Out-1]}. 63691 357: I1027 04:09:31.936687 62382 operator.cc:718] expected_kernel_key:data_type[double]:data_layout[ANY_LAYOUT]:place[CUDAPlace(0)]:lib rary_type[PLAIN] 63692 357: I1027 04:09:31.936822 62382 tensor_util.cu:25] TensorCopy 30, 3, 10, 1 from CUDAPlace(0) to CUDAPlace(0) 63693 357: I1027 04:09:31.937206 62382 tree_conv_op.h:58] Tree Conv: FORWARD 63694 357: I1027 04:09:31.937328 62382 tensor_util.cu:25] TensorCopy 16, 2 from CUDAPlace(0) to CPUPlace 63695 357: I1027 04:09:31.938135 62382 tree_conv_op.h:66] Patch dims 17, 90 63696 357: I1027 04:09:31.938192 62382 tree_conv_op.h:67] W dims 90, 10 63697 357: I1027 04:09:31.938225 62382 tree_conv_op.h:68] Out dims 17, 10 63698 357: I1027 04:09:31.938405 62382 tree_conv_op.h:71] Tree Conv: FORWARD DONE 63699 357: I1027 04:09:31.938585 62382 operator.cc:155] CUDAPlace(0) Op(tree_conv), inputs:{EdgeSet[EdgeSet:int1, 16, 2], Filter[Filter :double30, 3, 10, 1], NodesVector[NodesVector:double1, 17, 30]}, outputs:{Out[Out1, 17, 10, 1]}. 63700 357: I1027 04:09:31.938710 62382 operator.cc:143] CUDAPlace(0) Op(fetch), inputs:{X[Out:double1, 17, 10, 1]}, outputs:{Out[fetch -1]}. 63701 357: I1027 04:09:31.938815 62382 tensor_util.cu:107] TensorCopySync 1, 17, 10, 1 from CUDAPlace(0) to CPUPlace
63702 357: I1027 04:09:31.938997 62382 fetch_op.cc:60] Fetch variable Out to fetch 63703 357: I1027 04:09:31.939146 62382 operator.cc:155] CUDAPlace(0) Op(fetch), inputs:{X[Out:double1, 17, 10, 1]}, outputs:{Out[fetch -1]}. 63704 357: I1027 04:09:31.939473 62382 feed_fetch_method.cc:51] Fetch fetch with index 0 shape= 1, 17, 10, 1 63705 357: test_tree_conv_op failed
请问这是什么原因造成的?