Merge pull request #1233 from wangkuiyi/dockerfile-dev

Paddle standard development environment as a Docker image

doc/getstarted/build_and_install/docker_install_en.rst

@@ -16,70 +16,71 @@ Developers can work on PaddlePaddle using Docker.  This allows
 developers to work on different platforms -- Linux, Mac OS X, and
 Windows -- in a consistent way.
 
-The general development workflow with Docker and CMake is as follows:
-
-1. Get the source code of Paddle:
+1. Build the Development Environment as a Docker Image
 
    .. code-block:: bash
 
-      git clone https://github.com/PaddlePaddle/Paddle.git
+      git clone --recursive https://github.com/PaddlePaddle/Paddle
+      cd Paddle
+      docker build -t paddle:dev -f paddle/scripts/docker/Dockerfile .
 
 
-2. Build a development Docker image :code:`paddle:dev` from the source
-   code.  This image contains all the development tools and
-   dependencies of PaddlePaddle.
+   Note that by default :code:`docker build` wouldn't import source
+   tree into the image and build it.  If we want to do that, we need
+   to set a build arg:
 
    .. code-block:: bash
 
-      cd paddle
-      docker build -t paddle:dev -f paddle/scripts/docker/Dockerfile .
+      docker build -t paddle:dev -f paddle/scripts/docker/Dockerfile --build-arg BUILD_AND_INSTALL=ON .
+
+
+2. Run the Development Environment
 
-   Sometimes docker build might suffer from a slow network connection to the official Ubuntu apt-source servers. In such case, we can specify an apt-source mirror server that is geologically nearer to us. In the following example, we specified an apt-source server that responds fast in China.You can specify the UBUNTU MIRROR with :code:`--build-arg UBUNTU_MIRROR` like the example below.
+   Once we got the image :code:`paddle:dev`, we can use it to develop
+   Paddle by mounting the local source code tree into a container that
+   runs the image:
 
    .. code-block:: bash
 
-      docker build \
-       --build-arg UBUNTU_MIRROR="http://mirrors.163.com" \
-       -t paddle:dev \
-       -f paddle/scripts/docker/Dockerfile .
+      docker run -d -p 2202:22 -v $PWD:/paddle paddle:dev
+
+   This runs a container of the development environment Docker image
+   with the local source tree mounted to :code:`/paddle` of the
+   container.
+
+   Note that the default entry-point of :code:`paddle:dev` is
+   :code:`sshd`, and above :code:`docker run` commands actually starts
+   an SSHD server listening on port 2202.  This allows us to log into
+   this container with:
+
+   .. code-block:: bash
 
+      ssh root@localhost -p 2202
 
-3. Run the image as a container and mounting local source code
-   directory into the container.  This allows us to change the code on
-   the host and build it within the container.
+   Usually, I run above commands on my Mac.  I can also run them on a
+   GPU server :code:`xxx.yyy.zzz.www` and ssh from my Mac to it:
 
    .. code-block:: bash
 
-      docker run       \
-       -d              \
-       --name paddle   \
-       -p 2022:22      \
-       -v $PWD:/paddle \
-       paddle:dev
+      my-mac$ ssh root@xxx.yyy.zzz.www -p 2202
 
-   where :code:`-d` makes the container running in background,
-   :code:`--name paddle` allows us to run a nginx container to serve
-   documents in this container, :code:`-p 2022:22` allows us to SSH
-   into this container, :code:`-v $PWD:/paddle` shares the source code
-   on the host with the container.
+3. Build and Install Using the Development Environment
 
-4. SSH into the container:
+   Once I am in the container, I can use
+   :code:`paddle/scripts/docker/build.sh` to build, install, and test
+   Paddle:
 
    .. code-block:: bash
 
-      ssh root@localhost -p 2022
+      /paddle/paddle/scripts/docker/build.sh
 
-5. We can edit the source code in the container or on this host.  Then
-   we can build using cmake
+   This builds everything about Paddle in :code:`/paddle/build`.  And
+   we can run unit tests there:
 
    .. code-block:: bash
 
-      cd /paddle # where paddle source code has been mounted into the container
-      mkdir -p build
-      cd build
-      cmake -DWITH_TESTING=ON ..
-      make -j `nproc`
-      CTEST_OUTPUT_ON_FAILURE=1 ctest
+      cd /paddle/build
+      ctest
 
 
 CPU-only and GPU Images

paddle/scripts/docker/Dockerfile

@@ -13,8 +13,8 @@ RUN apt-get update && \
     apt-get install -y automake clang-3.8 llvm-3.8 libclang-3.8-dev && \
     apt-get clean -y
 
-RUN pip install --upgrade pip
-RUN pip install 'protobuf==3.1.0.post1' && \
+RUN pip install --upgrade pip && \
+    pip install -U protobuf && \
     pip install -U wheel pillow BeautifulSoup && \
     pip install -U docopt PyYAML sphinx && \
     pip install -U sphinx_rtd_theme recommonmark jupyter
@@ -23,10 +23,12 @@ RUN curl -sSL https://cmake.org/files/v3.4/cmake-3.4.1.tar.gz | tar -xz && \
     cd cmake-3.4.1 && ./bootstrap && make -j4 && make install && \
     cd .. && rm -rf cmake-3.4.1
 
+ARG BUILD_AND_INSTALL
 ARG WITH_AVX
 ARG WITH_DOC
 ARG WITH_STYLE_CHECK
 
+ENV BUILD_AND_INSTALL=${BUILD_AND_INSTALL:-OFF}
 ENV WITH_GPU=OFF
 ENV WITH_AVX=${WITH_AVX:-ON}
 ENV WITH_DOC=${WITH_DOC:-ON}
@@ -37,10 +39,6 @@ COPY . /paddle/
 RUN /paddle/paddle/scripts/docker/build.sh
 VOLUME ["/usr/share/nginx/html/data", "/usr/share/nginx/html/paddle"]
 
-RUN echo 'export LD_LIBRARY_PATH=/usr/lib64:${LD_LIBRARY_PATH}' >> /etc/profile
-RUN pip install /usr/local/opt/paddle/share/wheels/*.whl
-RUN paddle version  # print version after build
-
 # Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service
 RUN mkdir /var/run/sshd
 RUN echo 'root:root' | chpasswd

paddle/scripts/docker/Dockerfile.gpu

@@ -13,8 +13,8 @@ RUN apt-get update && \
     apt-get install -y automake clang-3.8 llvm-3.8 libclang-3.8-dev && \
     apt-get clean -y
 
-RUN pip install --upgrade pip
-RUN pip install 'protobuf==3.1.0.post1' && \
+RUN pip install --upgrade pip && \
+    pip install -U protobuf && \
     pip install -U wheel pillow BeautifulSoup && \
     pip install -U docopt PyYAML sphinx && \
     pip install -U sphinx_rtd_theme recommonmark jupyter
@@ -23,10 +23,12 @@ RUN curl -sSL https://cmake.org/files/v3.4/cmake-3.4.1.tar.gz | tar -xz && \
     cd cmake-3.4.1 && ./bootstrap && make -j4 && make install && \
     cd .. && rm -rf cmake-3.4.1
 
+ARG BUILD_AND_INSTALL
 ARG WITH_AVX
 ARG WITH_DOC
 ARG WITH_STYLE_CHECK
 
+ENV BUILD_AND_INSTALL=${BUILD_AND_INSTALL:-OFF}
 ENV WITH_GPU=ON
 ENV WITH_AVX=${WITH_AVX:-ON}
 ENV WITH_DOC=${WITH_DOC:-ON}
@@ -37,10 +39,6 @@ COPY . /paddle/
 RUN /paddle/paddle/scripts/docker/build.sh
 VOLUME ["/usr/share/nginx/html/data", "/usr/share/nginx/html/paddle"]
 
-RUN echo 'export LD_LIBRARY_PATH=/usr/lib64:${LD_LIBRARY_PATH}' >> /etc/profile
-RUN pip install /usr/local/opt/paddle/share/wheels/*.whl
-RUN paddle version  # print version after build
-
 # Configure OpenSSH server. c.f. https://docs.docker.com/engine/examples/running_ssh_service
 RUN mkdir /var/run/sshd
 RUN echo 'root:root' | chpasswd

paddle/scripts/docker/build.sh

@@ -8,42 +8,50 @@ function abort(){
 trap 'abort' 0
 set -e
 
-if [ ${WITH_GPU} == 'ON' ]; then
-  ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so /usr/lib/libcudnn.so
+# If Dockerfile.* sets BUILD_AND_INSTALL to 'ON', it would have copied
+# source tree to /paddle, and this scripts should build it into
+# /paddle/build.
+if [[ ${BUILD_AND_INSTALL:-ON} == 'ON' ]]; then
+    if [[ ${WITH_GPU:-OFF} == 'ON' ]]; then
+	ln -s /usr/lib/x86_64-linux-gnu/libcudnn.so /usr/lib/libcudnn.so
+    fi
+
+    mkdir -p /paddle/build # -p means no error if exists
+    cd /paddle/build
+    cmake .. \
+	  -DWITH_DOC=ON \
+	  -DWITH_GPU=${WITH_GPU:-OFF} \
+	  -DWITH_AVX=${WITH_AVX:-OFF} \
+	  -DWITH_SWIG_PY=ON \
+	  -DCUDNN_ROOT=/usr/ \
+	  -DWITH_STYLE_CHECK=OFF \
+	  -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
+    make -j `nproc`
+    make install
+
+    # Install woboq_codebrowser.
+    git clone https://github.com/woboq/woboq_codebrowser /woboq
+    cd /woboq
+    cmake -DLLVM_CONFIG_EXECUTABLE=/usr/bin/llvm-config-3.8 \
+	  -DCMAKE_BUILD_TYPE=Release \
+	  .
+    make
+
+    export WOBOQ_OUT=/usr/share/nginx/html/paddle
+    export BUILD_DIR=/paddle/build
+    mkdir -p $WOBOQ_OUT
+    cp -rv /woboq/data $WOBOQ_OUT/../data
+    /woboq/generator/codebrowser_generator \
+	-b /paddle/build \
+	-a \
+	-o $WOBOQ_OUT \
+	-p paddle:/paddle
+    /woboq/indexgenerator/codebrowser_indexgenerator $WOBOQ_OUT
+    cd /woboq
+    make clean
+
+    pip install /usr/local/opt/paddle/share/wheels/*.whl
+    paddle version
 fi
 
-mkdir -p /paddle/build # -p means no error if exists
-cd /paddle/build
-cmake .. \
-      -DWITH_DOC=ON \
-      -DWITH_GPU=${WITH_GPU} \
-      -DWITH_AVX=${WITH_AVX} \
-      -DWITH_SWIG_PY=ON \
-      -DCUDNN_ROOT=/usr/ \
-      -DWITH_STYLE_CHECK=OFF \
-      -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
-make -j `nproc`
-make install
-
-# Install woboq_codebrowser.
-git clone https://github.com/woboq/woboq_codebrowser /woboq
-cd /woboq
-cmake -DLLVM_CONFIG_EXECUTABLE=/usr/bin/llvm-config-3.8 \
-      -DCMAKE_BUILD_TYPE=Release \
-      .
-make
-
-export WOBOQ_OUT=/usr/share/nginx/html/paddle
-export BUILD_DIR=/paddle/build
-mkdir -p $WOBOQ_OUT
-cp -rv /woboq/data $WOBOQ_OUT/../data
-/woboq/generator/codebrowser_generator \
-    -b /paddle/build \
-    -a \
-    -o $WOBOQ_OUT \
-    -p paddle:/paddle
-/woboq/indexgenerator/codebrowser_indexgenerator $WOBOQ_OUT
-cd /woboq
-make clean
-rm -rf /paddle/build
 trap : 0