Skip to content

P
Paddle

BaiXuePrincess / Paddle
与 Fork 源项目一致

Fork自 PaddlePaddle / Paddle

通知 1
Star 1
Fork 0
P
Paddle
切换分支/标签
查找文件 普通视图历史永久链接Permalink
cluster_train.html 18.7 KB
Newer Older
Y
Update docs  
Yu Yang 已提交
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">


<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>Cluster Training &mdash; PaddlePaddle  documentation</title>
    
    <link rel="stylesheet" href="../../_static/classic.css" type="text/css" />
    <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../../',
        VERSION:     '',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="../../_static/jquery.js"></script>
    <script type="text/javascript" src="../../_static/underscore.js"></script>
    <script type="text/javascript" src="../../_static/doctools.js"></script>
    <script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
    <link rel="top" title="PaddlePaddle  documentation" href="../../index.html" />
    <link rel="up" title="Cluster Train" href="../index.html" />
Y
Upload docs  
Yu Yang 已提交
29 
    <link rel="next" title="Layer Documents" href="../../layer.html" />
Y
Update docs  
Yu Yang 已提交
30 31 32 33 34 35 36 37 38 39 40 41 
    <link rel="prev" title="Cluster Train" href="../index.html" /> 
  </head>
  <body role="document">
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../../genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="../../py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
Y
Upload docs  
Yu Yang 已提交
42 43 44 
        <li class="right" >
          <a href="../../layer.html" title="Layer Documents"
             accesskey="N">next</a> |</li>
Y
Update docs  
Yu Yang 已提交
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 
        <li class="right" >
          <a href="../index.html" title="Cluster Train"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="../../index.html">PaddlePaddle  documentation</a> &raquo;</li>
          <li class="nav-item nav-item-1"><a href="../index.html" accesskey="U">Cluster Train</a> &raquo;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="cluster-training">
<span id="cluster-training"></span><h1>Cluster Training<a class="headerlink" href="#cluster-training" title="Permalink to this headline"></a></h1>
Y
Upload docs  
Yu Yang 已提交
60 61 
<p>We provide some simple scripts <code class="docutils literal"><span class="pre">paddle/scripts/cluster_train</span></code> to help you to launch cluster training Job to harness PaddlePaddle&#8217;s distributed trainning. For MPI and other cluster scheduler refer this naive script to implement more robust cluster training platform by yourself.</p>
<p>The following cluster demo is based on RECOMMENDATION local training demo in PaddlePaddle <code class="docutils literal"><span class="pre">demo/recommendation</span></code> directory.  Assuming you enter the <code class="docutils literal"><span class="pre">paddle/scripts/cluster_train/</span></code> directory.</p>
Y
Update docs  
Yu Yang 已提交
62 63 64 65 66 67 
<div class="section" id="pre-requirements">
<span id="pre-requirements"></span><h2>Pre-requirements<a class="headerlink" href="#pre-requirements" title="Permalink to this headline"></a></h2>
<p>Firstly,</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install fabric
</pre></div>
</div>
Y
Upload docs  
Yu Yang 已提交
68 69 
<p>Secondly, go through installing scripts to install PaddlePaddle at all nodes to make sure demo can run as local mode. For CUDA enabled training, we assume that CUDA is installed in <code class="docutils literal"><span class="pre">/usr/local/cuda</span></code>, otherwise missed cuda runtime libraries error could be reported at cluster runtime. In one word, the local training environment should be well prepared for the simple scripts.</p>
<p>Then you should prepare same ROOT_DIR directory in all nodes. ROOT_DIR is from in cluster_train/conf.py. Assuming that the ROOT_DIR = /home/paddle, you can create <code class="docutils literal"><span class="pre">paddle</span></code> user account as well, at last <code class="docutils literal"><span class="pre">paddle.py</span></code> can ssh connections to all nodes with <code class="docutils literal"><span class="pre">paddle</span></code> user automatically.</p>
Y
Update docs  
Yu Yang 已提交
70 71 72 73 74 75 76 77 
<p>At last you can create ssh mutual trust relationship between all nodes for easy ssh login, otherwise <code class="docutils literal"><span class="pre">password</span></code> should be provided at runtime from <code class="docutils literal"><span class="pre">paddle.py</span></code>.</p>
</div>
<div class="section" id="prepare-job-workspace">
<span id="prepare-job-workspace"></span><h2>Prepare Job Workspace<a class="headerlink" href="#prepare-job-workspace" title="Permalink to this headline"></a></h2>
<p><code class="docutils literal"><span class="pre">Job</span> <span class="pre">workspace</span></code> is defined as one package directory which contains dependency libraries, train data, test data, model config file and all other related file dependencies.</p>
<p>These <code class="docutils literal"><span class="pre">train/test</span></code> data should be prepared before launching cluster job. To  satisfy the requirement that train/test data are placed in different directory from workspace, PADDLE refers train/test data according to index file named as <code class="docutils literal"><span class="pre">train.list/test.list</span></code> which are used in model config file. So the train/test data also contains train.list/test.list two list file. All local training demo already provides scripts to help you create these two files,  and all nodes in cluster job will handle files with same logical code in normal condition.</p>
<p>Generally, you can use same model file from local training for cluster training. What you should have in mind that, the <code class="docutils literal"><span class="pre">batch_size</span></code> set in <code class="docutils literal"><span class="pre">setting</span></code> function in model file means batch size in <code class="docutils literal"><span class="pre">each</span></code> node of cluster job instead of total batch size if synchronization SGD was used.</p>
<p>Following steps are based on demo/recommendation demo in demo directory.</p>
Y
Upload docs  
Yu Yang 已提交
78 
<p>You just go through demo/recommendation tutorial doc until <code class="docutils literal"><span class="pre">Train</span></code> section, and at last you will get train/test data and model configuration file. Finaly, just use demo/recommendation as workspace for cluster training.</p>
Y
Update docs  
Yu Yang 已提交
79 80 
<p>At last your workspace should look like as follow:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span>.
Y
Upload docs  
Yu Yang 已提交
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 
|-- common_utils.py
|-- data
|   |-- config.json
|   |-- config_generator.py
|   |-- meta.bin
|   |-- meta_config.json
|   |-- meta_generator.py
|   |-- ml-1m
|   |-- ml_data.sh
|   |-- ratings.dat.test
|   |-- ratings.dat.train
|   |-- split.py
|   |-- test.list
|   `-- train.list
|-- dataprovider.py
|-- evaluate.sh
|-- prediction.py
|-- preprocess.sh
|-- requirements.txt
|-- run.sh
`-- trainer_config.py
Y
Update docs  
Yu Yang 已提交
102 103 
</pre></div>
</div>
Y
Upload docs  
Yu Yang 已提交
104 105 
<p>Not all of these files are needed for cluster training, but it&#8217;s not necessary to remove useless files.</p>
<p><code class="docutils literal"><span class="pre">trainer_config.py</span></code>
Y
Update docs  
Yu Yang 已提交
106 107 108 
Indicates the model config file.</p>
<p><code class="docutils literal"><span class="pre">train.list</span></code> and <code class="docutils literal"><span class="pre">test.list</span></code>
File index. It stores all relative or absolute file paths of all train/test data at current node.</p>
Y
Upload docs  
Yu Yang 已提交
109 110 111 112 
<p><code class="docutils literal"><span class="pre">dataprovider.py</span></code>
used to read train/test samples. It&#8217;s same as local training.</p>
<p><code class="docutils literal"><span class="pre">data</span></code>
all files in data directory are refered by train.list/test.list which are refered by data provider.</p>
Y
Update docs  
Yu Yang 已提交
113 114 115 
</div>
<div class="section" id="prepare-cluster-job-configuration">
<span id="prepare-cluster-job-configuration"></span><h2>Prepare Cluster Job Configuration<a class="headerlink" href="#prepare-cluster-job-configuration" title="Permalink to this headline"></a></h2>
Y
Upload docs  
Yu Yang 已提交
116 
<p>The options below must be carefully set in cluster_train/conf.py</p>
Y
Update docs  
Yu Yang 已提交
117 118 119 120 121 122 
<p><code class="docutils literal"><span class="pre">HOSTS</span></code>  all nodes hostname or ip that will run cluster job. You can also append user and ssh port with hostname, such as root&#64;192.168.100.17:9090.</p>
<p><code class="docutils literal"><span class="pre">ROOT_DIR</span></code> workspace ROOT directory for placing JOB workspace directory</p>
<p><code class="docutils literal"><span class="pre">PADDLE_NIC</span></code> the NIC(Network Interface Card) interface name for cluster communication channel, such as eth0 for ethternet, ib0 for infiniband.</p>
<p><code class="docutils literal"><span class="pre">PADDLE_PORT</span></code> port number for cluster commnunication channel</p>
<p><code class="docutils literal"><span class="pre">PADDLE_PORTS_NUM</span></code> the number of port used for cluster communication channle. if the number of cluster nodes is small(less than 5~6nodes), recommend you set it to larger, such as 2 ~ 8, for better network performance.</p>
<p><code class="docutils literal"><span class="pre">PADDLE_PORTS_NUM_FOR_SPARSE</span></code> the number of port used for sparse updater cluster commnunication channel. if sparse remote update is used, set it like <code class="docutils literal"><span class="pre">PADDLE_PORTS_NUM</span></code></p>
Y
Upload docs  
Yu Yang 已提交
123 
<p><code class="docutils literal"><span class="pre">LD_LIBRARY_PATH</span></code> set addtional LD_LIBRARY_PATH for cluster job. You can use it to set CUDA libraries path.</p>
Y
Update docs  
Yu Yang 已提交
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 
<p>Default Configuration as follow:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">HOSTS</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s2">&quot;root@192.168.100.17&quot;</span><span class="p">,</span>
        <span class="s2">&quot;root@192.168.100.18&quot;</span><span class="p">,</span>
        <span class="p">]</span>

<span class="sd">&#39;&#39;&#39;</span>
<span class="sd">workspace configuration</span>
<span class="sd">&#39;&#39;&#39;</span>

<span class="c1">#root dir for workspace</span>
<span class="n">ROOT_DIR</span> <span class="o">=</span> <span class="s2">&quot;/home/paddle&quot;</span>

<span class="sd">&#39;&#39;&#39;</span>
<span class="sd">network configuration</span>
<span class="sd">&#39;&#39;&#39;</span>
<span class="c1">#pserver nics</span>
<span class="n">PADDLE_NIC</span> <span class="o">=</span> <span class="s2">&quot;eth0&quot;</span>
<span class="c1">#pserver port</span>
<span class="n">PADDLE_PORT</span> <span class="o">=</span> <span class="mi">7164</span>
<span class="c1">#pserver ports num</span>
<span class="n">PADDLE_PORTS_NUM</span> <span class="o">=</span> <span class="mi">2</span>
<span class="c1">#pserver sparse ports num</span>
<span class="n">PADDLE_PORTS_NUM_FOR_SPARSE</span> <span class="o">=</span> <span class="mi">2</span>
Y
Upload docs  
Yu Yang 已提交
148 149 150 

<span class="c1">#environments setting for all processes in cluster job</span>
<span class="n">LD_LIBRARY_PATH</span><span class="o">=</span><span class="s2">&quot;/usr/local/cuda/lib64:/usr/lib64&quot;</span>
Y
Update docs  
Yu Yang 已提交
151 152 153 154 155 156 157 158 159 
</pre></div>
</div>
<div class="section" id="launching-cluster-job">
<span id="launching-cluster-job"></span><h3>Launching Cluster Job<a class="headerlink" href="#launching-cluster-job" title="Permalink to this headline"></a></h3>
<p><code class="docutils literal"><span class="pre">paddle.py</span></code> provides automatical scripts to start all PaddlePaddle cluster processes in different nodes. By default, all command line options can set as <code class="docutils literal"><span class="pre">paddle.py</span></code> command options and <code class="docutils literal"><span class="pre">paddle.py</span></code> will transparently and automatically set these options to PaddlePaddle lower level processes.</p>
<p><code class="docutils literal"><span class="pre">paddle.py</span></code>provides two distinguished command option for easy job launching.</p>
<p><code class="docutils literal"><span class="pre">job_dispatch_package</span></code>  set it with local <code class="docutils literal"><span class="pre">workspace</span></code>directory, it will be dispatched to all nodes set in conf.py. It could be helpful for frequent hacking workspace files, otherwise frequent mulit-nodes workspace deployment could make your crazy.
<code class="docutils literal"><span class="pre">job_workspace</span></code>  set it with already deployed workspace directory, <code class="docutils literal"><span class="pre">paddle.py</span></code> will skip dispatch stage to directly launch cluster job with all nodes. It could help to reduce heavy
dispatch latency.</p>
Y
Upload docs  
Yu Yang 已提交
160 
<p><code class="docutils literal"><span class="pre">cluster_train/run.sh</span></code> provides command line sample to run <code class="docutils literal"><span class="pre">demo/recommendation</span></code> cluster job, just modify <code class="docutils literal"><span class="pre">job_dispatch_package</span></code> and <code class="docutils literal"><span class="pre">job_workspace</span></code> with your defined directory, then:</p>
Y
Update docs  
Yu Yang 已提交
161 162 163 164 165 166 167 
<div class="highlight-python"><div class="highlight"><pre><span></span>sh run.sh
</pre></div>
</div>
<p>The cluster Job will start in several seconds.</p>
</div>
<div class="section" id="kill-cluster-job">
<span id="kill-cluster-job"></span><h3>Kill Cluster Job<a class="headerlink" href="#kill-cluster-job" title="Permalink to this headline"></a></h3>
Y
Upload docs  
Yu Yang 已提交
168 
<p><code class="docutils literal"><span class="pre">paddle.py</span></code> can capture <code class="docutils literal"><span class="pre">Ctrl</span> <span class="pre">+</span> <span class="pre">C</span></code> SIGINT signal to automatically kill all processes launched by it. So just stop <code class="docutils literal"><span class="pre">paddle.py</span></code> to kill cluster job. You should mannally kill job if program crashed.</p>
Y
Update docs  
Yu Yang 已提交
169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 
</div>
<div class="section" id="check-cluster-training-result">
<span id="check-cluster-training-result"></span><h3>Check Cluster Training Result<a class="headerlink" href="#check-cluster-training-result" title="Permalink to this headline"></a></h3>
<p>Check log in $workspace/log for details, each node owns same log structure.</p>
<p><code class="docutils literal"><span class="pre">paddle_trainer.INFO</span></code>
It provides almost all interal output log for training,  same as local training. Check runtime model convergence here.</p>
<p><code class="docutils literal"><span class="pre">paddle_pserver2.INFO</span></code>
It provides pserver running log, which could help to diagnose distributed error.</p>
<p><code class="docutils literal"><span class="pre">server.log</span></code>
It provides stderr and stdout of pserver process. Check error log if training crashs.</p>
<p><code class="docutils literal"><span class="pre">train.log</span></code>
It provides stderr and stdout of trainer process. Check error log if training crashs.</p>
</div>
<div class="section" id="check-model-output">
<span id="check-model-output"></span><h3>Check Model Output<a class="headerlink" href="#check-model-output" title="Permalink to this headline"></a></h3>
<p>After one pass finished, model files will be writed in <code class="docutils literal"><span class="pre">output</span></code> directory in node 0.
<code class="docutils literal"><span class="pre">nodefile</span></code> in workspace indicates the node id of current cluster job.</p>
</div>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h3><a href="../../index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Cluster Training</a><ul>
<li><a class="reference internal" href="#pre-requirements">Pre-requirements</a></li>
<li><a class="reference internal" href="#prepare-job-workspace">Prepare Job Workspace</a></li>
<li><a class="reference internal" href="#prepare-cluster-job-configuration">Prepare Cluster Job Configuration</a><ul>
<li><a class="reference internal" href="#launching-cluster-job">Launching Cluster Job</a></li>
<li><a class="reference internal" href="#kill-cluster-job">Kill Cluster Job</a></li>
<li><a class="reference internal" href="#check-cluster-training-result">Check Cluster Training Result</a></li>
<li><a class="reference internal" href="#check-model-output">Check Model Output</a></li>
</ul>
</li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="../index.html"
                        title="previous chapter">Cluster Train</a></p>
Y
Upload docs  
Yu Yang 已提交
215 216 217 
  <h4>Next topic</h4>
  <p class="topless"><a href="../../layer.html"
                        title="next chapter">Layer Documents</a></p>
Y
Update docs  
Yu Yang 已提交
218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="../../_sources/cluster/opensource/cluster_train.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="../../search.html" method="get">
      <input type="text" name="q" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    <p class="searchtip" style="font-size: 90%">
    Enter search terms or a module, class or function name.
    </p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../../genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="../../py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
Y
Upload docs  
Yu Yang 已提交
251 252 253 
        <li class="right" >
          <a href="../../layer.html" title="Layer Documents"
             >next</a> |</li>
Y
Update docs  
Yu Yang 已提交
254 255 256 257 258 259 260 261 262 263 264 265 266 
        <li class="right" >
          <a href="../index.html" title="Cluster Train"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="../../index.html">PaddlePaddle  documentation</a> &raquo;</li>
          <li class="nav-item nav-item-1"><a href="../index.html" >Cluster Train</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &copy; Copyright 2016, PaddlePaddle developers.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.5.
    </div>
  </body>
</html>