import paddle出现ImportError:dlopen:cannot load any more object with static TLS
Created by: OliverLPH
- 标题:import paddle.fluid 时报错dlopen:cannot load any more object with static TLS
- 版本、环境信息: 1)PaddlePaddle版本:develop:a6e935f4 2)CPU:MKL/openblas 3)GPU:CUDA9/CUDA10 4)系统环境:paddlepaddle/paddle_manylinux_devel:cuda10.0_cudnn7,centos6.1,gcc482
- 安装方式信息: 1)pip安装/docker安装 2)本地编译: 3)docker编译:paddlepaddle/paddle_manylinux_devel:cuda10.0_cudnn7 特殊环境请注明:需升级镜像中的glibc版本
- 复现信息:
由于镜像中的glibc版本仅为2.13,而TensorRT的lib依赖的glibc需高于2.13,所以先升级镜像中的glibc至2.18,升级命令如下。
wget http://ftp.gnu.org/gnu/glibc/glibc-2.18.tar.gz
tar -xvf glibc-2.18.tar.gz
cd glibc-2.18
unset LD_LIBRARY_PATH # must unset LD_LIBRARY_PATH before compile glibc
mkdir build && cd build && ../configure --prefix=/usr && make -j4 && make install
export LD_LIBRARY_PATH=/opt/rh/devtoolset-2/root/usr/lib64:/opt/rh/devtoolset-2/root/usr/lib:/usr/local/lib64:/usr/local/lib:
然后安装ci上编译的develop的cuda9.0_cudnn7_mkl whl包
打开Python 后import paddle.fluid
其中mkl的包会报错。
Python 2.7.11 (default, Nov 15 2019, 12:44:37)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import paddle
>>> import paddle.fluid as fluid
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/_internal/cpython-2.7.11-ucs4/lib/python2.7/site-packages/paddle/fluid/__init__.py", line 35, in <module>
from . import framework
File "/opt/_internal/cpython-2.7.11-ucs4/lib/python2.7/site-packages/paddle/fluid/framework.py", line 35, in <module>
from . import core
File "/opt/_internal/cpython-2.7.11-ucs4/lib/python2.7/site-packages/paddle/fluid/core.py", line 189, in <module>
raise e
ImportError: dlopen: cannot load any more object with static TLS
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
看了这个跟TLS有关的issueCannot load any more object with static TLS while executing crnn_ctc on multi-devices后验证了关闭 mkl的whl包,从ci上下载了 cuda9.0_cudnn7_openblas whl包 而openblas的包不报错,且可以正常使用
Python 2.7.11 (default, Nov 15 2019, 12:44:37)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import paddle.fluid as fluid
>>> import paddle
>>>
- 问题描述:
目前验证过 paddle1.6.1 paddle1.6.2开启mkl的whl包,如果升级了镜像glibc并不会造成该问题。该问题仅在develop上可以复现。