Created by: hijkzzz

Thu Mar 1 11:05:13 2018[1,44]:Shell Script Stack Trace Thu Mar 1 11:05:13 2018[1,44]: @: [./log.sh: 61] check_return Thu Mar 1 11:05:13 2018[1,44]: @: [./start_server.sh: 34] main Thu Mar 1 11:05:13 2018[1,44]: Thu Mar 1 11:05:13 2018[1,44]:+ exit 1 Thu Mar 1 11:05:13 2018[1,38]:WARNING: Logging before InitGoogleLogging() is written to STDERR Thu Mar 1 11:05:13 2018[1,38]:F0301 11:05:13.781890 50448 hl_dso_loader.cc:97] Check failed: nullptr != dso_handle Failed to find cuda library: libcudart.so Thu Mar 1 11:05:13 2018[1,38]:Please specify its path correctly using one of the following ideas: Thu Mar 1 11:05:13 2018[1,38]:Idea 1. set cuda and cudnn lib path at runtime. http://www.paddlepaddle.org/doc/ui/cmd_argument/argument_outline.html Thu Mar 1 11:05:13 2018[1,38]:For instance, issue command: paddle train --use_gpu=1 --cuda_dir=/usr/local/cudnn/lib --cudnn_dir=/usr/local/cudnn/lib ... Thu Mar 1 11:05:13 2018[1,38]:Idea 2. set environment variable LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS. Thu Mar 1 11:05:13 2018[1,38]:For instance, issue command: export LD_LIBRARY_PATH=... Thu Mar 1 11:05:13 2018[1,38]:Note: After Mac OS 10.11, using the DYLD_LIBRARY_PATH is impossible unless System Integrity Protection (SIP) is disabled. However, @Idea 1always work well. Thu Mar 1 11:05:13 2018[1,47]:WARNING: Logging before InitGoogleLogging() is written to STDERR Thu Mar 1 11:05:13 2018[1,47]:F0301 11:05:13.787817 41376 hl_dso_loader.cc:97] Check failed: nullptr != dso_handle Failed to find cuda library: libcudart.so Thu Mar 1 11:05:13 2018[1,47]:Please specify its path correctly using one of the following ideas: Thu Mar 1 11:05:13 2018[1,47]:Idea 1. set cuda and cudnn lib path at runtime. http://www.paddlepaddle.org/doc/ui/cmd_argument/argument_outline.html Thu Mar 1 11:05:13 2018[1,47]:For instance, issue command: paddle train --use_gpu=1 --cuda_dir=/usr/local/cudnn/lib --cudnn_dir=/usr/local/cudnn/lib ... Thu Mar 1 11:05:13 2018[1,47]:Idea 2. set environment variable LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS. Thu Mar 1 11:05:13 2018[1,47]:For instance, issue command: export LD_LIBRARY_PATH=... Thu Mar 1 11:05:13 2018[1,47]:Note: After Mac OS 10.11, using the DYLD_LIBRARY_PATH is impossible unless System Integrity Protection (SIP) is disabled. However, @Idea 1always work well. Thu Mar 1 11:05:13 2018[1,47]: Check failure stack trace: *** Thu Mar 1 11:05:13 2018[1,47]:./start_server.sh: line 33: 41376 Aborted GLOG_logtostderr=0 GLOG_log_dir="./log" ./paddle_pserver2 --num_gradient_servers=${OMPI_COMM_WORLD_SIZE} --nics=${nics} ${server_arg} --rdma_tcp=${rdma_tcp} --comment=$comment Thu Mar 1 11:05:13 2018[1,47]:+ check_return 'paddle_pserver2 failed' Thu Mar 1 11:05:13 2018[1,47]:+ '[' 134 -ne 0 ']' Thu Mar 1 11:05:13 2018[1,47]:+ echo '[./start_server.sh : 34] [main]' Thu Mar 1 11:05:13 2018[1,47]:[./start_server.sh : 34] [main] Thu Mar 1 11:05:13 2018[1,47]:+ echo '[FATAL]: paddle_pserver2 failed' Thu Mar 1 11:05:13 2018[1,47]:[FATAL]: paddle_pserver2 failed Thu Mar 1 11:05:13 2018[1,47]:+ get_stack Thu Mar 1 11:05:13 2018[1,47]:+ set +x Thu Mar 1 11:05:13 2018[1,47]: