When configured use "--loadsave_parameters_in_pserver" trainning fails to init parameters
Created by: typhoonzero
Word embedding using v1 API:
Log is:
I0814 05:35:20.026533 69 Util.cpp:166] commandline: /usr/local/lib/python2.7/dist-packages/usr/local/opt/paddle/bin/paddle_trainer --loadsave_parameters_in_pserver=true --job=train --use_old_updater=1 --port=7164 --nics=eth0 --ports_num=1 --ports_num_for_sparse=1 --num_passes=1 --trainer_count=1 --saving_period=1 --log_period=20 --local=0 --rdma_tcp=tcp --config=emb_train.py --use_gpu=0 --trainer_id=0 --save_dir=./output --pservers=10.1.90.7,10.1.53.8 --num_gradient_servers=4
[INFO 2017-08-14 05:35:20,519 networks.py:1491] The input order is [w1, w2, w3, w4, label]
[INFO 2017-08-14 05:35:20,519 networks.py:1497] The output order is [__cost_0__]
I0814 05:35:20.521780 69 Trainer.cpp:162] trainer mode: SgdSparseCpuTraining
I0814 05:35:20.583519 69 PyDataProvider2.cpp:243] loading dataprovider dataprovider_emb::process
I0814 05:35:20.584769 69 GradientMachine.cpp:85] Initing parameters..
*** Aborted at 1502688920 (unix time) try "date -d @1502688920" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x30) received by PID 69 (TID 0x7fea192fe740) from PID 48; stack trace: ***
@ 0x7fea18cdb390 (unknown)
@ 0xc95a3a paddle::VectorT<>::getSize()
@ 0xd7057e paddle::Parameter::isFullSize()
@ 0xd700d2 paddle::GradientMachine::randParameters()
@ 0xe85f4f paddle::Trainer::init()
@ 0xbda83e main
@ 0x7fea178ed830 __libc_start_main
@ 0xbda4b9 _start
@ 0x0 (unknown)
/usr/local/bin/paddle: line 113: 69 Segmentation fault (core dumped) ${DEBUGGER} /usr/local/lib/python2.7/dist-packages/usr/local/opt/paddle/bin/paddle_trainer ${@:2}
Seems that "PARAMETER_VALUE" is not properly initialized in this case.