A few months ago when we were trying to replace CMake with Bazel, @emailweixu suggested that we rewrite those handy Bazel functions using CMake. Now it seems that it's the right time to get this done, as we are facing problems from the porting of Majel and the development of new the parameter server using Go and C++.
Here are some initial thoughts. Your comments are welcome!
### Required CMake Function
I think we need only the following few CMake functions to make a project description mean and clean:
| C++ | CUDA C++ | Go |
|---|---|---|
| cc_library | nv_library | go_library |
| cc_binary | nv_binary | go_binary |
| cc_test | nv_test | go_test |
- The `_library` functions generate .a files from source code.
- The `_binary` functions generate executable binary files.
- The `_test` functions generate executable unit test files. They work like `_binary` but links `-lgtest` and `-lgtest_main`.
The difference between `nv_` functions and `cc_` functions is that the former use `nvcc` instead of the system-default C++ compiler.
Both `nv_` and `cc_` functions enables C++11 (-std=c++11).
Also,
- to describe external dependencies, we need `external_library`.
- to build shared libraries, we need `shared_library`.
### An Example Project
Suppose that we have aforementioned functions defined in our `/cmake` directory. The following example `CMakeLists.txt` describes a project including the following source files:
- tensor.h
- tensor.cc
- tensor_test.cc
- ops.h
- ops.cu
- ops_test.cu
- api.go
- api_test.go
Suppose that ops.cu depends on CUDNN.
```cmake
# cc_binary parses tensor.cc and figures out that target also depend
# on tensor.h.
cc_binary(tensor
SRCS
tensor.cc)
# The dependency to target tensor implies that if any of
# tensor{.h,.cc,_test.cc} is changed, tensor_test need to be re-built.
cc_test(tensor_test
SRCS
tensor_test.cc
DEPS
tensor)
# I don't have a clear idea what parameters external_library need to
# have. @gangliao as a CMake expert would have better ideas.
external_library(cudnn
....)
# Suppose that ops.cu depends on external target CUDNN. Also, ops.cu
# include global functions that take Tensor as their parameters, so
# ops depend on tensor. This implies that if any of tensor.{h.cc},
# ops.{h,cu} is changed, ops need to be re-built.
nv_library(ops
SRCS
ops.cu
DEPS
tensor
cudnn) # cudnn is defined later.
nv_test(ops_test
SRCS
ops_test.cu
DEPS
ops)
# Because api.go defines a GO wrapper to ops and tensor, it depends on
# both. This implies that if any of tensor.{h,cc}, ops.{h,cu}, or
# api.go is changed, api need to be re-built.
go_library(api
SRCS
api.go
DEPS
tensor # Because ops depend on tensor, this line is optional.
ops)
go_test(api_test
SRCS
api_test.go
DEPS
api)
# This builds libapi.so. shared_library might use CMake target
# api_shared so to distinguish it from above target api.
shared_library(api
DEPS
api)
```
### Implementation
As above example CMakeLists.txt executes, each function invocation adds "nodes" to a dependency graph. It also use this graph to generate CMake commands including `add_executable`, `add_dependencies`, `target_link_libraries`, and `add_test`.
<liclass="toctree-l2"><aclass="reference internal"href="../../getstarted/build_and_install/index_en.html">Install and Build</a><ul>
<liclass="toctree-l3"><aclass="reference internal"href="../../getstarted/build_and_install/docker_install_en.html">PaddlePaddle in Docker Containers</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../../howto/usage/k8s/k8s_en.html">Paddle On Kubernetes</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../../howto/usage/k8s/k8s_aws_en.html">Distributed PaddlePaddle Training on AWS with Kubernetes</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="../../howto/dev/new_layer_en.html">Write New Layers</a></li>
<p>A few months ago when we were trying to replace CMake with Bazel, @emailweixu suggested that we rewrite those handy Bazel functions using CMake. Now it seems that it’s the right time to get this done, as we are facing problems from the porting of Majel and the development of new the parameter server using Go and C++.</p>
<p>Here are some initial thoughts. Your comments are welcome!</p>
<divclass="section"id="required-cmake-function">
<spanid="required-cmake-function"></span><h1>Required CMake Function<aclass="headerlink"href="#required-cmake-function"title="Permalink to this headline">¶</a></h1>
<p>I think we need only the following few CMake functions to make a project description mean and clean:</p>
<p>| C++ | CUDA C++ | Go |
|—|—|—|
| cc_library | nv_library | go_library |
| cc_binary | nv_binary | go_binary |
| cc_test | nv_test | go_test |</p>
<ulclass="simple">
<li>The <codeclass="docutils literal"><spanclass="pre">_library</span></code> functions generate .a files from source code.</li>
<li>The <codeclass="docutils literal"><spanclass="pre">_test</span></code> functions generate executable unit test files. They work like <codeclass="docutils literal"><spanclass="pre">_binary</span></code> but links <codeclass="docutils literal"><spanclass="pre">-lgtest</span></code> and <codeclass="docutils literal"><spanclass="pre">-lgtest_main</span></code>.</li>
</ul>
<p>The difference between <codeclass="docutils literal"><spanclass="pre">nv_</span></code> functions and <codeclass="docutils literal"><spanclass="pre">cc_</span></code> functions is that the former use <codeclass="docutils literal"><spanclass="pre">nvcc</span></code> instead of the system-default C++ compiler.</p>
<p>Both <codeclass="docutils literal"><spanclass="pre">nv_</span></code> and <codeclass="docutils literal"><spanclass="pre">cc_</span></code> functions enables C++11 (-std=c++11).</p>
<p>Also,</p>
<ulclass="simple">
<li>to describe external dependencies, we need <codeclass="docutils literal"><spanclass="pre">external_library</span></code>.</li>
<li>to build shared libraries, we need <codeclass="docutils literal"><spanclass="pre">shared_library</span></code>.</li>
</ul>
</div>
<divclass="section"id="an-example-project">
<spanid="an-example-project"></span><h1>An Example Project<aclass="headerlink"href="#an-example-project"title="Permalink to this headline">¶</a></h1>
<p>Suppose that we have aforementioned functions defined in our <codeclass="docutils literal"><spanclass="pre">/cmake</span></code> directory. The following example <codeclass="docutils literal"><spanclass="pre">CMakeLists.txt</span></code> describes a project including the following source files:</p>
<ulclass="simple">
<li>tensor.h</li>
<li>tensor.cc</li>
<li>tensor_test.cc</li>
<li>ops.h</li>
<li>ops.cu</li>
<li>ops_test.cu</li>
<li>api.go</li>
<li>api_test.go</li>
</ul>
<p>Suppose that ops.cu depends on CUDNN.</p>
<divclass="highlight-cmake"><divclass="highlight"><pre><span></span><spanclass="c"># cc_binary parses tensor.cc and figures out that target also depend</span>
<spanid="implementation"></span><h1>Implementation<aclass="headerlink"href="#implementation"title="Permalink to this headline">¶</a></h1>
<p>As above example CMakeLists.txt executes, each function invocation adds “nodes” to a dependency graph. It also use this graph to generate CMake commands including <codeclass="docutils literal"><spanclass="pre">add_executable</span></code>, <codeclass="docutils literal"><spanclass="pre">add_dependencies</span></code>, <codeclass="docutils literal"><spanclass="pre">target_link_libraries</span></code>, and <codeclass="docutils literal"><spanclass="pre">add_test</span></code>.</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
A few months ago when we were trying to replace CMake with Bazel, @emailweixu suggested that we rewrite those handy Bazel functions using CMake. Now it seems that it's the right time to get this done, as we are facing problems from the porting of Majel and the development of new the parameter server using Go and C++.
Here are some initial thoughts. Your comments are welcome!
### Required CMake Function
I think we need only the following few CMake functions to make a project description mean and clean:
| C++ | CUDA C++ | Go |
|---|---|---|
| cc_library | nv_library | go_library |
| cc_binary | nv_binary | go_binary |
| cc_test | nv_test | go_test |
- The `_library` functions generate .a files from source code.
- The `_binary` functions generate executable binary files.
- The `_test` functions generate executable unit test files. They work like `_binary` but links `-lgtest` and `-lgtest_main`.
The difference between `nv_` functions and `cc_` functions is that the former use `nvcc` instead of the system-default C++ compiler.
Both `nv_` and `cc_` functions enables C++11 (-std=c++11).
Also,
- to describe external dependencies, we need `external_library`.
- to build shared libraries, we need `shared_library`.
### An Example Project
Suppose that we have aforementioned functions defined in our `/cmake` directory. The following example `CMakeLists.txt` describes a project including the following source files:
- tensor.h
- tensor.cc
- tensor_test.cc
- ops.h
- ops.cu
- ops_test.cu
- api.go
- api_test.go
Suppose that ops.cu depends on CUDNN.
```cmake
# cc_binary parses tensor.cc and figures out that target also depend
# on tensor.h.
cc_binary(tensor
SRCS
tensor.cc)
# The dependency to target tensor implies that if any of
# tensor{.h,.cc,_test.cc} is changed, tensor_test need to be re-built.
cc_test(tensor_test
SRCS
tensor_test.cc
DEPS
tensor)
# I don't have a clear idea what parameters external_library need to
# have. @gangliao as a CMake expert would have better ideas.
external_library(cudnn
....)
# Suppose that ops.cu depends on external target CUDNN. Also, ops.cu
# include global functions that take Tensor as their parameters, so
# ops depend on tensor. This implies that if any of tensor.{h.cc},
# ops.{h,cu} is changed, ops need to be re-built.
nv_library(ops
SRCS
ops.cu
DEPS
tensor
cudnn) # cudnn is defined later.
nv_test(ops_test
SRCS
ops_test.cu
DEPS
ops)
# Because api.go defines a GO wrapper to ops and tensor, it depends on
# both. This implies that if any of tensor.{h,cc}, ops.{h,cu}, or
# api.go is changed, api need to be re-built.
go_library(api
SRCS
api.go
DEPS
tensor # Because ops depend on tensor, this line is optional.
ops)
go_test(api_test
SRCS
api_test.go
DEPS
api)
# This builds libapi.so. shared_library might use CMake target
# api_shared so to distinguish it from above target api.
shared_library(api
DEPS
api)
```
### Implementation
As above example CMakeLists.txt executes, each function invocation adds "nodes" to a dependency graph. It also use this graph to generate CMake commands including `add_executable`, `add_dependencies`, `target_link_libraries`, and `add_test`.
<p>A few months ago when we were trying to replace CMake with Bazel, @emailweixu suggested that we rewrite those handy Bazel functions using CMake. Now it seems that it’s the right time to get this done, as we are facing problems from the porting of Majel and the development of new the parameter server using Go and C++.</p>
<p>Here are some initial thoughts. Your comments are welcome!</p>
<li>The <codeclass="docutils literal"><spanclass="pre">_test</span></code> functions generate executable unit test files. They work like <codeclass="docutils literal"><spanclass="pre">_binary</span></code> but links <codeclass="docutils literal"><spanclass="pre">-lgtest</span></code> and <codeclass="docutils literal"><spanclass="pre">-lgtest_main</span></code>.</li>
</ul>
<p>The difference between <codeclass="docutils literal"><spanclass="pre">nv_</span></code> functions and <codeclass="docutils literal"><spanclass="pre">cc_</span></code> functions is that the former use <codeclass="docutils literal"><spanclass="pre">nvcc</span></code> instead of the system-default C++ compiler.</p>
<p>Both <codeclass="docutils literal"><spanclass="pre">nv_</span></code> and <codeclass="docutils literal"><spanclass="pre">cc_</span></code> functions enables C++11 (-std=c++11).</p>
<p>Also,</p>
<ulclass="simple">
<li>to describe external dependencies, we need <codeclass="docutils literal"><spanclass="pre">external_library</span></code>.</li>
<li>to build shared libraries, we need <codeclass="docutils literal"><spanclass="pre">shared_library</span></code>.</li>
</ul>
</div>
<divclass="section"id="an-example-project">
<spanid="an-example-project"></span><h1>An Example Project<aclass="headerlink"href="#an-example-project"title="永久链接至标题">¶</a></h1>
<p>Suppose that we have aforementioned functions defined in our <codeclass="docutils literal"><spanclass="pre">/cmake</span></code> directory. The following example <codeclass="docutils literal"><spanclass="pre">CMakeLists.txt</span></code> describes a project including the following source files:</p>
<ulclass="simple">
<li>tensor.h</li>
<li>tensor.cc</li>
<li>tensor_test.cc</li>
<li>ops.h</li>
<li>ops.cu</li>
<li>ops_test.cu</li>
<li>api.go</li>
<li>api_test.go</li>
</ul>
<p>Suppose that ops.cu depends on CUDNN.</p>
<divclass="highlight-cmake"><divclass="highlight"><pre><span></span><spanclass="c"># cc_binary parses tensor.cc and figures out that target also depend</span>
<p>As above example CMakeLists.txt executes, each function invocation adds “nodes” to a dependency graph. It also use this graph to generate CMake commands including <codeclass="docutils literal"><spanclass="pre">add_executable</span></code>, <codeclass="docutils literal"><spanclass="pre">add_dependencies</span></code>, <codeclass="docutils literal"><spanclass="pre">target_link_libraries</span></code>, and <codeclass="docutils literal"><spanclass="pre">add_test</span></code>.</p>
Built with <ahref="http://sphinx-doc.org/">Sphinx</a> using a <ahref="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <ahref="https://readthedocs.org">Read the Docs</a>.
