Nothing else. Not even Python and GCC, because you can install all build tools into a Docker image. We run all the tools by running this image.
## General Process
1. Retrieve source code.
```bash
git clone https://github.com/paddlepaddle/paddle
```
2. Install build tools into a Docker image.
```bash
cd paddle; docker build -t paddle:dev .
```
Please be aware of the `.` at the end of the command, which refers to the [`./Dockerfile` file](https://github.com/PaddlePaddle/Paddle/blob/develop/Dockerfile). `docker build` follows instructions in this file to create a Docker image named `paddle:dev`, and installs building tools into it.
3. Build from source.
This following command starts a Docker container that executes the Docker image `paddle:dev`, mapping the current directory to `/paddle/` in the container, and runs the default entry-point [`build.sh`](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh) as specified in the Dockefile. `build.sh` invokes `cmake` and `make` to build PaddlePaddle source code, which had been mapped to `/paddle`, and writes outputs to `/paddle/build`, which maps to `build` in the current source directory on the computer.
```bash
docker run -v $PWD:/paddle paddle:dev
```
Above command builds a CUDA-enabled version. If we want to build a CPU-only version, we can type
```bash
docker run -e WITH_GPU=OFF -v $PWD:/paddle paddle:dev
```
4. Run unit tests.
To run all unit tests using the first GPU of a node:
Sometimes, we might want to clean all thirt-party dependents and built binaries. To do so, just
```bash
rm -rf build
```
## Docker, Or Not?
- What is Docker?
If you haven't heard of it, consider it something like Python's virtualenv.
- Docker or virtual machine?
Some people compare Docker with VMs, but Docker doesn't virtualize any hardware nor running a guest OS, which means there is no compromise on the performance.
- Why Docker?
Using a Docker image of build tools standardizes the building environment, which makes it easier for others to reproduce your problems and to help.
Also, some build tools don't run on Windows or Mac or BSD, but Docker runs almost everywhere, so developers can use whatever computer they want.
- Can I choose not to use Docker?
Sure, you don't have to install build tools into a Docker image; instead, you can install them in your local computer. This document exists because Docker would make the development way easier.
- How difficult is it to learn Docker?
It takes you ten minutes to read [an introductory article](https://docs.docker.com/get-started) and saves you more than one hour to install all required build tools, configure them, especially when new versions of PaddlePaddle require some new tools. Not even to mention the time saved when other people trying to reproduce the issue you have.
- Can I use my favorite IDE?
Yes, of course. The source code resides on your local computer, and you can edit it using whatever editor you like.
Many PaddlePaddle developers are using Emacs. They add the following few lines into their `~/.emacs` configure file:
```emacs
(global-set-key "\C-cc" 'compile)
(setq compile-command
"docker run --rm -it -v $(git rev-parse --show-toplevel):/paddle paddle:dev")
```
so they could type `Ctrl-C` and `c` to build PaddlePaddle from source.
- Does Docker do parallel building?
Our building Docker image runs a [Bash script](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh), which calls `make -j$(nproc)` to starts as many processes as the number of your CPU cores.
## Some Gotchas
- Docker requires sudo
An owner of a computer has the administrative privilege, a.k.a., sudo, and Docker requires this privilege to work properly. If you use a shared computer for development, please ask the administrator to install and configure Docker. We will do our best to support rkt, another container technology that doesn't require sudo.
- Docker on Windows/MacOS builds slowly
On Windows and MacOS, Docker containers run in a Linux VM. You might want to give this VM some more memory and CPUs so to make the building efficient. Please refer to [this issue](https://github.com/PaddlePaddle/Paddle/issues/627) for details.
- Not enough disk space
Examples in this article uses option `--rm` with the `docker run` command. This option ensures that stopped containers do not exist on hard disks. We can use `docker ps -a` to list all containers, including stopped. Sometimes `docker build` generates some intermediate dangling images, which also take disk space. To clean them, please refer to [this article](https://zaiste.net/posts/removing_docker_containers/).
If you haven't heard of it, consider it something like Python's virtualenv.
- Docker or virtual machine?
Some people compare Docker with VMs, but Docker doesn't virtualize any hardware nor running a guest OS, which means there is no compromise on the performance.
- Why Docker?
Using a Docker image of build tools standardizes the building environment, which makes it easier for others to reproduce your problems and to help.
Also, some build tools don't run on Windows or Mac or BSD, but Docker runs almost everywhere, so developers can use whatever computer they want.
- Can I choose not to use Docker?
Sure, you don't have to install build tools into a Docker image; instead, you can install them on your local computer. This document exists because Docker would make the development way easier.
- How difficult is it to learn Docker?
It takes you ten minutes to read [an introductory article](https://docs.docker.com/get-started) and saves you more than one hour to install all required build tools, configure them, especially when new versions of PaddlePaddle require some new tools. Not even to mention the time saved when other people trying to reproduce the issue you have.
- Can I use my favorite IDE?
Yes, of course. The source code resides on your local computer, and you can edit it using whatever editor you like.
Many PaddlePaddle developers are using Emacs. They add the following few lines into their `~/.emacs` configure file:
```emacs
(global-set-key "\C-cc" 'compile)
(setq compile-command
"docker run --rm -it -v $(git rev-parse --show-toplevel):/paddle paddle:dev")
```
so they could type `Ctrl-C` and `c` to build PaddlePaddle from source.
- Does Docker do parallel building?
Our building Docker image runs a [Bash script](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh), which calls `make -j$(nproc)` to starts as many processes as the number of your CPU cores.
- Docker requires sudo
An owner of a computer has the administrative privilege, a.k.a., sudo, and Docker requires this privilege to work properly. If you use a shared computer for development, please ask the administrator to install and configure Docker. We will do our best to support rkt, another container technology that doesn't require sudo.
- Docker on Windows/MacOS builds slowly
On Windows and MacOS, Docker containers run in a Linux VM. You might want to give this VM some more memory and CPUs so to make the building efficient. Please refer to [this issue](https://github.com/PaddlePaddle/Paddle/issues/627) for details.
- Not enough disk space
Examples in this article use option `--rm` with the `docker run` command. This option ensures that stopped containers do not exist on hard disks. We can use `docker ps -a` to list all containers, including stopped. Sometimes `docker build` generates some intermediate dangling images, which also take disk space. To clean them, please refer to [this article](https://zaiste.net/posts/removing_docker_containers/).
.. _compile_deps:
Compile Dependencies
Appendix: Compile Dependencies
----------------
PaddlePaddle need the following dependencies when compiling, other dependencies
...
...
@@ -97,17 +163,13 @@ will be downloaded automatically.
.. _build_options:
Build Options
Appendix: Build Options
----------------
Build options include whether build binaries for CPU or GPU, which BLAS
library to use etc. You may pass these settings when running cmake.
For detailed cmake tutorial please refer to `here <https://cmake.org/cmake-tutorial>`_ 。
.. _build_options_bool:
Bool Type Options
----------------
You can add :code:`-D` argument to pass such options, like:
<liclass="toctree-l3"><aclass="reference internal"href="../howto/cluster/multi_cluster/index_en.html">Use different clusters</a><ul>
<liclass="toctree-l4"><aclass="reference internal"href="../howto/cluster/multi_cluster/fabric_en.html">Cluster Training Using Fabric</a></li>
<liclass="toctree-l4"><aclass="reference internal"href="../howto/cluster/multi_cluster/openmpi_en.html">Cluster Training Using OpenMPI</a></li>
<liclass="toctree-l4"><aclass="reference internal"href="../howto/cluster/multi_cluster/k8s_en.html">PaddlePaddle On Kubernetes</a></li>
<liclass="toctree-l4"><aclass="reference internal"href="../howto/cluster/multi_cluster/k8s_aws_en.html">Distributed PaddlePaddle Training on AWS with Kubernetes</a></li>
<spanid="build-using-docker"></span><h1>Build using Docker<aclass="headerlink"href="#build-using-docker"title="Permalink to this headline">¶</a></h1>
<divclass="section"id="what-developers-need">
<spanid="what-developers-need"></span><h2>What Developers Need<aclass="headerlink"href="#what-developers-need"title="Permalink to this headline">¶</a></h2>
<p>Please be aware of the <codeclass="docutils literal"><spanclass="pre">.</span></code> at the end of the command, which refers to the <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/Dockerfile"><codeclass="docutils literal"><spanclass="pre">./Dockerfile</span></code> file</a>. <codeclass="docutils literal"><spanclass="pre">docker</span><spanclass="pre">build</span></code> follows instructions in this file to create a Docker image named <codeclass="docutils literal"><spanclass="pre">paddle:dev</span></code>, and installs building tools into it.</p>
</li>
<li><pclass="first">Build from source.</p>
<p>This following command starts a Docker container that executes the Docker image <codeclass="docutils literal"><spanclass="pre">paddle:dev</span></code>, mapping the current directory to <codeclass="docutils literal"><spanclass="pre">/paddle/</span></code> in the container, and runs the default entry-point <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh"><codeclass="docutils literal"><spanclass="pre">build.sh</span></code></a> as specified in the Dockefile. <codeclass="docutils literal"><spanclass="pre">build.sh</span></code> invokes <codeclass="docutils literal"><spanclass="pre">cmake</span></code> and <codeclass="docutils literal"><spanclass="pre">make</span></code> to build PaddlePaddle source code, which had been mapped to <codeclass="docutils literal"><spanclass="pre">/paddle</span></code>, and writes outputs to <codeclass="docutils literal"><spanclass="pre">/paddle/build</span></code>, which maps to <codeclass="docutils literal"><spanclass="pre">build</span></code> in the current source directory on the computer.</p>
<divclass="highlight-bash"><divclass="highlight"><pre><span></span>docker run -v <spanclass="nv">$PWD</span>:/paddle paddle:dev
</pre></div>
</div>
<p>Above command builds a CUDA-enabled version. If we want to build a CPU-only version, we can type</p>
<divclass="highlight-bash"><divclass="highlight"><pre><span></span>docker run -e <spanclass="nv">WITH_GPU</span><spanclass="o">=</span>OFF -v <spanclass="nv">$PWD</span>:/paddle paddle:dev
</pre></div>
</div>
</li>
<li><pclass="first">Run unit tests.</p>
<p>To run all unit tests using the first GPU of a node:</p>
<p>If we used <codeclass="docutils literal"><spanclass="pre">WITH_GPU=OFF</span></code> at build time, it generates only CPU-based unit tests, and we don’t need nvidia-docker to run them. We can just run</p>
<divclass="highlight-bash"><divclass="highlight"><pre><span></span>docker run -v <spanclass="nv">$PWD</span>:/paddle paddle:dev bash -c <spanclass="s2">"cd /paddle/build; ctest"</span>
</pre></div>
</div>
<p>Sometimes we want to run a specific unit test, say <codeclass="docutils literal"><spanclass="pre">memory_test</span></code>, we can run</p>
<spanid="docker-or-not"></span><h2>Docker, Or Not?<aclass="headerlink"href="#docker-or-not"title="Permalink to this headline">¶</a></h2>
<ul>
<li><pclass="first">What is Docker?</p>
<p>If you haven’t heard of it, consider it something like Python’s virtualenv.</p>
</li>
<li><pclass="first">Docker or virtual machine?</p>
<p>Some people compare Docker with VMs, but Docker doesn’t virtualize any hardware nor running a guest OS, which means there is no compromise on the performance.</p>
</li>
<li><pclass="first">Why Docker?</p>
<p>Using a Docker image of build tools standardizes the building environment, which makes it easier for others to reproduce your problems and to help.</p>
<p>Also, some build tools don’t run on Windows or Mac or BSD, but Docker runs almost everywhere, so developers can use whatever computer they want.</p>
</li>
<li><pclass="first">Can I choose not to use Docker?</p>
<p>Sure, you don’t have to install build tools into a Docker image; instead, you can install them in your local computer. This document exists because Docker would make the development way easier.</p>
</li>
<li><pclass="first">How difficult is it to learn Docker?</p>
<p>It takes you ten minutes to read <aclass="reference external"href="https://docs.docker.com/get-started">an introductory article</a> and saves you more than one hour to install all required build tools, configure them, especially when new versions of PaddlePaddle require some new tools. Not even to mention the time saved when other people trying to reproduce the issue you have.</p>
</li>
<li><pclass="first">Can I use my favorite IDE?</p>
<p>Yes, of course. The source code resides on your local computer, and you can edit it using whatever editor you like.</p>
<p>Many PaddlePaddle developers are using Emacs. They add the following few lines into their <codeclass="docutils literal"><spanclass="pre">~/.emacs</span></code> configure file:</p>
<spanclass="s">"docker run --rm -it -v $(git rev-parse --show-toplevel):/paddle paddle:dev"</span><spanclass="p">)</span>
</pre></div>
</div>
<p>so they could type <codeclass="docutils literal"><spanclass="pre">Ctrl-C</span></code> and <codeclass="docutils literal"><spanclass="pre">c</span></code> to build PaddlePaddle from source.</p>
</li>
<li><pclass="first">Does Docker do parallel building?</p>
<p>Our building Docker image runs a <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh">Bash script</a>, which calls <codeclass="docutils literal"><spanclass="pre">make</span><spanclass="pre">-j$(nproc)</span></code> to starts as many processes as the number of your CPU cores.</p>
</li>
</ul>
</div>
<divclass="section"id="some-gotchas">
<spanid="some-gotchas"></span><h2>Some Gotchas<aclass="headerlink"href="#some-gotchas"title="Permalink to this headline">¶</a></h2>
<ul>
<li><pclass="first">Docker requires sudo</p>
<p>An owner of a computer has the administrative privilege, a.k.a., sudo, and Docker requires this privilege to work properly. If you use a shared computer for development, please ask the administrator to install and configure Docker. We will do our best to support rkt, another container technology that doesn’t require sudo.</p>
</li>
<li><pclass="first">Docker on Windows/MacOS builds slowly</p>
<p>On Windows and MacOS, Docker containers run in a Linux VM. You might want to give this VM some more memory and CPUs so to make the building efficient. Please refer to <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/issues/627">this issue</a> for details.</p>
</li>
<li><pclass="first">Not enough disk space</p>
<p>Examples in this article uses option <codeclass="docutils literal"><spanclass="pre">--rm</span></code> with the <codeclass="docutils literal"><spanclass="pre">docker</span><spanclass="pre">run</span></code> command. This option ensures that stopped containers do not exist on hard disks. We can use <codeclass="docutils literal"><spanclass="pre">docker</span><spanclass="pre">ps</span><spanclass="pre">-a</span></code> to list all containers, including stopped. Sometimes <codeclass="docutils literal"><spanclass="pre">docker</span><spanclass="pre">build</span></code> generates some intermediate dangling images, which also take disk space. To clean them, please refer to <aclass="reference external"href="https://zaiste.net/posts/removing_docker_containers/">this article</a>.</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
<spanid="faq-docker"></span><h2>Frequently Asked Questions<aclass="headerlink"href="#frequently-asked-questions"title="Permalink to this headline">¶</a></h2>
<ul>
<li><pclass="first">What is Docker?</p>
<p>If you haven’t heard of it, consider it something like Python’s virtualenv.</p>
</li>
<li><pclass="first">Docker or virtual machine?</p>
<p>Some people compare Docker with VMs, but Docker doesn’t virtualize any hardware nor running a guest OS, which means there is no compromise on the performance.</p>
</li>
<li><pclass="first">Why Docker?</p>
<p>Using a Docker image of build tools standardizes the building environment, which makes it easier for others to reproduce your problems and to help.</p>
<p>Also, some build tools don’t run on Windows or Mac or BSD, but Docker runs almost everywhere, so developers can use whatever computer they want.</p>
</li>
<li><pclass="first">Can I choose not to use Docker?</p>
<p>Sure, you don’t have to install build tools into a Docker image; instead, you can install them on your local computer. This document exists because Docker would make the development way easier.</p>
</li>
<li><pclass="first">How difficult is it to learn Docker?</p>
<blockquote>
<div><p>It takes you ten minutes to read [an introductory article](<aclass="reference external"href="https://docs.docker.com/get-started">https://docs.docker.com/get-started</a>) and saves you more than one hour to install all required build tools, configure them, especially when new versions of PaddlePaddle require some new tools. Not even to mention the time saved when other people trying to reproduce the issue you have.</p>
</div></blockquote>
</li>
<li><pclass="first">Can I use my favorite IDE?</p>
<p>Yes, of course. The source code resides on your local computer, and you can edit it using whatever editor you like.</p>
<p>Many PaddlePaddle developers are using Emacs. They add the following few lines into their <cite>~/.emacs</cite> configure file:</p>
<p>so they could type <cite>Ctrl-C</cite> and <cite>c</cite> to build PaddlePaddle from source.</p>
</li>
<li><pclass="first">Does Docker do parallel building?</p>
<p>Our building Docker image runs a [Bash script](<aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh">https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh</a>), which calls <cite>make -j$(nproc)</cite> to starts as many processes as the number of your CPU cores.</p>
</li>
<li><pclass="first">Docker requires sudo</p>
<p>An owner of a computer has the administrative privilege, a.k.a., sudo, and Docker requires this privilege to work properly. If you use a shared computer for development, please ask the administrator to install and configure Docker. We will do our best to support rkt, another container technology that doesn’t require sudo.</p>
</li>
<li><pclass="first">Docker on Windows/MacOS builds slowly</p>
<p>On Windows and MacOS, Docker containers run in a Linux VM. You might want to give this VM some more memory and CPUs so to make the building efficient. Please refer to [this issue](<aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/issues/627">https://github.com/PaddlePaddle/Paddle/issues/627</a>) for details.</p>
</li>
<li><pclass="first">Not enough disk space</p>
<p>Examples in this article use option <cite>–rm</cite> with the <cite>docker run</cite> command. This option ensures that stopped containers do not exist on hard disks. We can use <cite>docker ps -a</cite> to list all containers, including stopped. Sometimes <cite>docker build</cite> generates some intermediate dangling images, which also take disk space. To clean them, please refer to [this article](<aclass="reference external"href="https://zaiste.net/posts/removing_docker_containers/">https://zaiste.net/posts/removing_docker_containers/</a>).</p>
<spanid="compile-deps"></span><h2>Appendix: Compile Dependencies<aclass="headerlink"href="#appendix-compile-dependencies"title="Permalink to this headline">¶</a></h2>
<p>PaddlePaddle need the following dependencies when compiling, other dependencies
<caption><spanclass="caption-text">PaddlePaddle Compile Dependencies</span><aclass="headerlink"href="#id3"title="Permalink to this table">¶</a></caption>
<caption><spanclass="caption-text">PaddlePaddle Compile Dependencies</span><aclass="headerlink"href="#id11"title="Permalink to this table">¶</a></caption>
<colgroup>
<colwidth="18%"/>
<colwidth="27%"/>
...
...
@@ -289,20 +341,17 @@ will be downloaded automatically.</p>
</tbody>
</table>
</div>
<divclass="section"id="build-options">
<spanid="id1"></span><h2>Build Options<aclass="headerlink"href="#build-options"title="Permalink to this headline">¶</a></h2>
<divclass="section"id="appendix-build-options">
<spanid="build-options"></span><h2>Appendix: Build Options<aclass="headerlink"href="#appendix-build-options"title="Permalink to this headline">¶</a></h2>
<p>Build options include whether build binaries for CPU or GPU, which BLAS
library to use etc. You may pass these settings when running cmake.
For detailed cmake tutorial please refer to <aclass="reference external"href="https://cmake.org/cmake-tutorial">here</a> 。</p>
</div>
<divclass="section"id="bool-type-options">
<spanid="build-options-bool"></span><h2>Bool Type Options<aclass="headerlink"href="#bool-type-options"title="Permalink to this headline">¶</a></h2>
<p>You can add <codeclass="code docutils literal"><spanclass="pre">-D</span></code> argument to pass such options, like:</p>
