提交 786faabf 编写于 作者: W wangkuiyi 提交者: GitHub

Merge pull request #1233 from wangkuiyi/dockerfile-dev

Paddle standard development environment as a Docker image
......@@ -16,70 +16,71 @@ Developers can work on PaddlePaddle using Docker. This allows
developers to work on different platforms -- Linux, Mac OS X, and
Windows -- in a consistent way.
The general development workflow with Docker and CMake is as follows:
1. Get the source code of Paddle:
1. Build the Development Environment as a Docker Image
.. code-block:: bash
git clone https://github.com/PaddlePaddle/Paddle.git
git clone --recursive https://github.com/PaddlePaddle/Paddle
cd Paddle
docker build -t paddle:dev -f paddle/scripts/docker/Dockerfile .
2. Build a development Docker image :code:`paddle:dev` from the source
code. This image contains all the development tools and
dependencies of PaddlePaddle.
Note that by default :code:`docker build` wouldn't import source
tree into the image and build it. If we want to do that, we need
to set a build arg:
.. code-block:: bash
cd paddle
docker build -t paddle:dev -f paddle/scripts/docker/Dockerfile .
docker build -t paddle:dev -f paddle/scripts/docker/Dockerfile --build-arg BUILD_AND_INSTALL=ON .
Sometimes docker build might suffer from a slow network connection to the official Ubuntu apt-source servers. In such case, we can specify an apt-source mirror server that is geologically nearer to us. In the following example, we specified an apt-source server that responds fast in China.You can specify the UBUNTU MIRROR with :code:`--build-arg UBUNTU_MIRROR` like the example below.
2. Run the Development Environment
Once we got the image :code:`paddle:dev`, we can use it to develop
Paddle by mounting the local source code tree into a container that
runs the image:
.. code-block:: bash
docker build \
--build-arg UBUNTU_MIRROR="http://mirrors.163.com" \
-t paddle:dev \
-f paddle/scripts/docker/Dockerfile .
docker run -d -p 2202:22 -v $PWD:/paddle paddle:dev
This runs a container of the development environment Docker image
with the local source tree mounted to :code:`/paddle` of the
container.
Note that the default entry-point of :code:`paddle:dev` is
:code:`sshd`, and above :code:`docker run` commands actually starts
an SSHD server listening on port 2202. This allows us to log into
this container with:
.. code-block:: bash
ssh root@localhost -p 2202
3. Run the image as a container and mounting local source code
directory into the container. This allows us to change the code on
the host and build it within the container.
Usually, I run above commands on my Mac. I can also run them on a
GPU server :code:`xxx.yyy.zzz.www` and ssh from my Mac to it:
.. code-block:: bash
docker run \
-d \
--name paddle \
-p 2022:22 \
-v $PWD:/paddle \
paddle:dev
my-mac$ ssh root@xxx.yyy.zzz.www -p 2202
where :code:`-d` makes the container running in background,
:code:`--name paddle` allows us to run a nginx container to serve
documents in this container, :code:`-p 2022:22` allows us to SSH
into this container, :code:`-v $PWD:/paddle` shares the source code
on the host with the container.
3. Build and Install Using the Development Environment
4. SSH into the container:
Once I am in the container, I can use
:code:`paddle/scripts/docker/build.sh` to build, install, and test
Paddle:
.. code-block:: bash
ssh root@localhost -p 2022
/paddle/paddle/scripts/docker/build.sh
5. We can edit the source code in the container or on this host. Then
we can build using cmake
This builds everything about Paddle in :code:`/paddle/build`. And
we can run unit tests there:
.. code-block:: bash
cd /paddle # where paddle source code has been mounted into the container
mkdir -p build
cd build
cmake -DWITH_TESTING=ON ..
make -j `nproc`
CTEST_OUTPUT_ON_FAILURE=1 ctest
cd /paddle/build
ctest
CPU-only and GPU Images
......
......@@ -13,8 +13,8 @@ RUN apt-get update && \
apt-get install -y automake clang-3.8 llvm-3.8 libclang-3.8-dev && \
apt-get clean -y
RUN pip install --upgrade pip
RUN pip install 'protobuf==3.1.0.post1' && \
RUN pip install --upgrade pip && \
pip install -U protobuf && \
pip install -U wheel pillow BeautifulSoup && \
pip install -U docopt PyYAML sphinx && \
pip install -U sphinx_rtd_theme recommonmark jupyter
......@@ -23,10 +23,12 @@ RUN curl -sSL https://cmake.org/files/v3.4/cmake-3.4.1.tar.gz | tar -xz && \
cd cmake-3.4.1 && ./bootstrap && make -j4 && make install && \
cd .. && rm -rf cmake-3.4.1
ARG BUILD_AND_INSTALL
ARG WITH_AVX
ARG WITH_DOC
ARG WITH_STYLE_CHECK
ENV BUILD_AND_INSTALL=${BUILD_AND_INSTALL:-OFF}
ENV WITH_GPU=OFF
ENV WITH_AVX=${WITH_AVX:-ON}
ENV WITH_DOC=${WITH_DOC:-ON}
......@@ -37,10 +39,6 @@ COPY . /paddle/
RUN /paddle/paddle/scripts/docker/build.sh
VOLUME ["/usr/share/nginx/html/data", "/usr/share/nginx/html/paddle"]
RUN echo 'export LD_LIBRARY_PATH=/usr/lib64:${LD_LIBRARY_PATH}' >> /etc/profile
RUN pip install /usr/local/opt/paddle/share/wheels/*.whl
RUN paddle version # print version after build
# Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service
RUN mkdir /var/run/sshd
RUN echo 'root:root' | chpasswd
......
......@@ -13,8 +13,8 @@ RUN apt-get update && \
apt-get install -y automake clang-3.8 llvm-3.8 libclang-3.8-dev && \
apt-get clean -y
RUN pip install --upgrade pip
RUN pip install 'protobuf==3.1.0.post1' && \
RUN pip install --upgrade pip && \
pip install -U protobuf && \
pip install -U wheel pillow BeautifulSoup && \
pip install -U docopt PyYAML sphinx && \
pip install -U sphinx_rtd_theme recommonmark jupyter
......@@ -23,10 +23,12 @@ RUN curl -sSL https://cmake.org/files/v3.4/cmake-3.4.1.tar.gz | tar -xz && \
cd cmake-3.4.1 && ./bootstrap && make -j4 && make install && \
cd .. && rm -rf cmake-3.4.1
ARG BUILD_AND_INSTALL
ARG WITH_AVX
ARG WITH_DOC
ARG WITH_STYLE_CHECK
ENV BUILD_AND_INSTALL=${BUILD_AND_INSTALL:-OFF}
ENV WITH_GPU=ON
ENV WITH_AVX=${WITH_AVX:-ON}
ENV WITH_DOC=${WITH_DOC:-ON}
......@@ -37,10 +39,6 @@ COPY . /paddle/
RUN /paddle/paddle/scripts/docker/build.sh
VOLUME ["/usr/share/nginx/html/data", "/usr/share/nginx/html/paddle"]
RUN echo 'export LD_LIBRARY_PATH=/usr/lib64:${LD_LIBRARY_PATH}' >> /etc/profile
RUN pip install /usr/local/opt/paddle/share/wheels/*.whl
RUN paddle version # print version after build
# Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service
RUN mkdir /var/run/sshd
RUN echo 'root:root' | chpasswd
......
......@@ -8,42 +8,50 @@ function abort(){
trap 'abort' 0
set -e
if [ ${WITH_GPU} == 'ON' ]; then
# If Dockerfile.* sets BUILD_AND_INSTALL to 'ON', it would have copied
# source tree to /paddle, and this scripts should build it into
# /paddle/build.
if [[ ${BUILD_AND_INSTALL:-ON} == 'ON' ]]; then
if [[ ${WITH_GPU:-OFF} == 'ON' ]]; then
ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so /usr/lib/libcudnn.so
fi
fi
mkdir -p /paddle/build # -p means no error if exists
cd /paddle/build
cmake .. \
mkdir -p /paddle/build # -p means no error if exists
cd /paddle/build
cmake .. \
-DWITH_DOC=ON \
-DWITH_GPU=${WITH_GPU} \
-DWITH_AVX=${WITH_AVX} \
-DWITH_GPU=${WITH_GPU:-OFF} \
-DWITH_AVX=${WITH_AVX:-OFF} \
-DWITH_SWIG_PY=ON \
-DCUDNN_ROOT=/usr/ \
-DWITH_STYLE_CHECK=OFF \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON
make -j `nproc`
make install
make -j `nproc`
make install
# Install woboq_codebrowser.
git clone https://github.com/woboq/woboq_codebrowser /woboq
cd /woboq
cmake -DLLVM_CONFIG_EXECUTABLE=/usr/bin/llvm-config-3.8 \
# Install woboq_codebrowser.
git clone https://github.com/woboq/woboq_codebrowser /woboq
cd /woboq
cmake -DLLVM_CONFIG_EXECUTABLE=/usr/bin/llvm-config-3.8 \
-DCMAKE_BUILD_TYPE=Release \
.
make
make
export WOBOQ_OUT=/usr/share/nginx/html/paddle
export BUILD_DIR=/paddle/build
mkdir -p $WOBOQ_OUT
cp -rv /woboq/data $WOBOQ_OUT/../data
/woboq/generator/codebrowser_generator \
export WOBOQ_OUT=/usr/share/nginx/html/paddle
export BUILD_DIR=/paddle/build
mkdir -p $WOBOQ_OUT
cp -rv /woboq/data $WOBOQ_OUT/../data
/woboq/generator/codebrowser_generator \
-b /paddle/build \
-a \
-o $WOBOQ_OUT \
-p paddle:/paddle
/woboq/indexgenerator/codebrowser_indexgenerator $WOBOQ_OUT
cd /woboq
make clean
rm -rf /paddle/build
/woboq/indexgenerator/codebrowser_indexgenerator $WOBOQ_OUT
cd /woboq
make clean
pip install /usr/local/opt/paddle/share/wheels/*.whl
paddle version
fi
trap : 0
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册