更新本地提交package中的paddle二进制后集群报错
Created by: JayEworld
本地提交集群训练任务
之前是从wget http://paddle-mpi-package.gz.bcebos.com/cluster_train_cpu_nordma.tar.gz下载的提交程序包,但是发现不支持layer.factorization_machine,考虑是版本太低问题,于是按照“ 更新本地提交package中的paddle二进制”的步骤更新了程序包里的paddle。
更新时使用的whl是:paddlepaddle-latest-cp27-cp27m-linux_x86_64.whl
集群是http://nmg-hpc-hlan-mon.dmop.baidu.com:8919/ 提交的任务是app-user-20180822115211-3409 报错信息如下:
Wed Aug 22 12:10:41 2018[1,28]<stderr>:RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
Wed Aug 22 12:10:41 2018[1,28]<stderr>:Traceback (most recent call last):
Wed Aug 22 12:10:41 2018[1,28]<stderr>: File "conf/trainer_config.conf", line 184, in <module>
Wed Aug 22 12:10:41 2018[1,28]<stderr>: pservers=os.getenv("PADDLE_PSERVERS", "127.0.0.1"))
Wed Aug 22 12:10:41 2018[1,28]<stderr>: File "/home/disk1/normandy/maybach/app-user-20180822115211-3409/workspace/python27-gcc482/lib/python2.7/site-packages/paddle/v2/__init__.py", line 128, in init
Wed Aug 22 12:10:41 2018[1,28]<stderr>: import py_paddle.swig_paddle as api
Wed Aug 22 12:10:41 2018[1,28]<stderr>: File "/home/disk1/normandy/maybach/app-user-20180822115211-3409/workspace/python27-gcc482/lib/python2.7/site-packages/py_paddle/__init__.py", line 15, in <module>
Wed Aug 22 12:10:41 2018[1,28]<stderr>: from util import DataProviderWrapperConverter
Wed Aug 22 12:10:41 2018[1,28]<stderr>: File "/home/disk1/normandy/maybach/app-user-20180822115211-3409/workspace/python27-gcc482/lib/python2.7/site-packages/py_paddle/util.py", line 18, in <module>
Wed Aug 22 12:10:41 2018[1,28]<stderr>: import swig_paddle
Wed Aug 22 12:10:41 2018[1,28]<stderr>: File "/home/disk1/normandy/maybach/app-user-20180822115211-3409/workspace/python27-gcc482/lib/python2.7/site-packages/py_paddle/swig_paddle.py", line 28, in <module>
Wed Aug 22 12:10:41 2018[1,28]<stderr>: _swig_paddle = swig_import_helper()
Wed Aug 22 12:10:41 2018[1,28]<stderr>: File "/home/disk1/normandy/maybach/app-user-20180822115211-3409/workspace/python27-gcc482/lib/python2.7/site-packages/py_paddle/swig_paddle.py", line 24, in swig_import_helper
Wed Aug 22 12:10:41 2018[1,28]<stderr>: _mod = imp.load_module('_swig_paddle', fp, pathname, description)
Wed Aug 22 12:10:41 2018[1,28]<stderr>:ImportError: numpy.core.multiarray failed to import