parallel_nn related crash
Created by: byzhang
If I turn on --parallel_nn, the translation demo (./train.sh) will crash:
#0 __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:33
#1 0x0000000000740f90 in paddle::BaseMatrixT<float>::assign(paddle::BaseMatrixT<float>&) ()
#2 0x00000000005b4fb5 in paddle::GruStepLayer::forward (this=0x16875e80, passType=<optimized out>) at /home/byzhang/src/github/Paddle/paddle/gserver/layers/GruStepLayer.cpp:98
#3 0x000000000062639e in paddle::NeuralNetwork::forward (this=0x16866ac0, inArgs=..., outArgs=0x7fff86369b10, passType=paddle::enumeration_wrapper::PASS_TRAIN) at /home/byzhang/src/github/Paddle/paddle/gserver/gradientmachines/NeuralNetwork.cpp:242
#4 0x00000000006164af in paddle::RecurrentGradientMachine::forward (this=<optimized out>, inArgs=..., outArgs=<optimized out>, passType=paddle::enumeration_wrapper::PASS_TRAIN) at /home/byzhang/src/github/Paddle/paddle/gserver/gradientmachines/RecurrentGradientMachine.cpp:546
#5 0x000000000057257a in paddle::RecurrentLayerGroup::forward (this=<optimized out>, passType=<optimized out>) at /home/byzhang/src/github/Paddle/paddle/gserver/layers/RecurrentLayerGroup.cpp:42
#6 0x0000000000621854 in paddle::ParallelThread::computeThread (this=0x168c59b0) at /home/byzhang/src/github/Paddle/paddle/gserver/gradientmachines/ParallelNeuralNetwork.cpp:174
#7 0x00007ffff6486c30 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007ffff78c2184 in start_thread (arg=0x7fff8636a700) at pthread_create.c:312
#9 0x00007ffff5c0e37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
The commit is https://github.com/byzhang/Paddle/commit/de9d6c6f39ed8068ffb30e5dde7e3eda6ed889bd
I tried to fix the aforementioned crash, but it crashed somewhere else related with unassigned realLayer_ of ScatterAgentLayer. So I need your help on debugging the crash.