Skip to content

P
Paddle

PaddlePaddle / Paddle
大约 1 年 前同步成功

通知 2299
Star 20931
Fork 5422
P
Paddle
提交 82e956b5 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: b15b2637

上级 8503f028
显示空白变更内容
内联 并排
Showing with 2 addition and 688 deletion
develop/doc/searchindex.js
浏览文件 @ 82e956b5
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc/tutorials/rec/ml_regression_en.html
浏览文件 @ 82e956b5
......@@ -212,24 +212,6 @@ i.e, specific <strong>WHAT</strong> type it is in each feature file.</p>
<p>The field config file of ml-1m shows in <code class="code docutils literal"><span class="pre">demo/recommendation/data/config.json</span></code>.
It specifics the field types and file names: 1) there are four types of field for user file: id, gender, age and occupation;
2) the filename is &#8220;users.dat&#8221;, and the delimiter of file is &#8221;::&#8221;.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">&quot;user&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;users.dat&quot;</span><span class="p">,</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;id&quot;</span><span class="p">,</span> <span class="s2">&quot;gender&quot;</span><span class="p">,</span> <span class="s2">&quot;age&quot;</span><span class="p">,</span> <span class="s2">&quot;occupation&quot;</span><span class="p">]</span>
<span class="p">},</span>
<span class="s2">&quot;movie&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;movies.dat&quot;</span><span class="p">,</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;id&quot;</span><span class="p">,</span> <span class="s2">&quot;title&quot;</span><span class="p">,</span> <span class="s2">&quot;genres&quot;</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="preprocess-data">
......@@ -261,89 +243,6 @@ python config_generator.py config.json &gt; meta_config.json
</pre></div>
</div>
<p>The meta config file shows below:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">&quot;meta&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;movie&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;id&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;regex&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;pattern&quot;</span><span class="p">:</span> <span class="s2">&quot;^(.*)</span><span class="se">\\</span><span class="s2">((</span><span class="se">\\</span><span class="s2">d+)</span><span class="se">\\</span><span class="s2">)$&quot;</span><span class="p">,</span>
<span class="s2">&quot;group_id&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;strip&quot;</span><span class="p">:</span> <span class="n">true</span>
<span class="p">},</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;seq_type&quot;</span><span class="p">:</span> <span class="s2">&quot;sequence&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;char_based&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;title&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">1</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;one_hot_dense&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;|&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;split&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;genres&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;split&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;movies.dat&quot;</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s2">&quot;user&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;id&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;char_based&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;gender&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">1</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;sort&quot;</span><span class="p">:</span> <span class="n">true</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;whole_content&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;age&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">2</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;sort&quot;</span><span class="p">:</span> <span class="s2">&quot;true&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;whole_content&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;occupation&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">3</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;split&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;users.dat&quot;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<p>There are two kinds of features in meta: movie and user.</p>
<ul class="simple">
<li><dl class="first docutils">
......@@ -497,92 +396,6 @@ cp ml-1m/ratings.dat.test .
<p>The network structure shows below.</p>
<img alt="rec_regression_network" class="align-center" src="../../_images/rec_regression_network.png" />
<p>The demo&#8217;s neural network config file <code class="code docutils literal"><span class="pre">trainer_config.py</span></code> show as below.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="kn">import</span> <span class="o">*</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">cPickle</span> <span class="kn">as</span> <span class="nn">pickle</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">pickle</span>
<span class="n">is_predict</span> <span class="o">=</span> <span class="n">get_config_arg</span><span class="p">(</span><span class="s1">&#39;is_predict&#39;</span><span class="p">,</span> <span class="nb">bool</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
<span class="n">META_FILE</span> <span class="o">=</span> <span class="s1">&#39;data/meta.bin&#39;</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">META_FILE</span><span class="p">,</span> <span class="s1">&#39;rb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="c1"># load meta file</span>
<span class="n">meta</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">settings</span><span class="p">(</span>
<span class="n">batch_size</span><span class="o">=</span><span class="mi">1600</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">1e-3</span><span class="p">,</span> <span class="n">learning_method</span><span class="o">=</span><span class="n">RMSPropOptimizer</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">construct_feature</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Construct movie/user features.</span>
<span class="sd"> This method read from meta data. Then convert feature to neural network due</span>
<span class="sd"> to feature type. The map relation as follow.</span>
<span class="sd"> * id: embedding =&gt; fc</span>
<span class="sd"> * embedding:</span>
<span class="sd"> is_sequence: embedding =&gt; context_projection =&gt; fc =&gt; pool</span>
<span class="sd"> not sequence: embedding =&gt; fc</span>
<span class="sd"> * one_hot_dense: fc =&gt; fc</span>
<span class="sd"> Then gather all features vector, and use a fc layer to combined them as</span>
<span class="sd"> return.</span>
<span class="sd"> :param name: &#39;movie&#39; or &#39;user&#39;</span>
<span class="sd"> :type name: basestring</span>
<span class="sd"> :return: combined feature output</span>
<span class="sd"> :rtype: LayerOutput</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">__meta__</span> <span class="o">=</span> <span class="n">meta</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="s1">&#39;__meta__&#39;</span><span class="p">][</span><span class="s1">&#39;raw_meta&#39;</span><span class="p">]</span>
<span class="n">fusion</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">each_meta</span> <span class="ow">in</span> <span class="n">__meta__</span><span class="p">:</span>
<span class="n">type_name</span> <span class="o">=</span> <span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;type&#39;</span><span class="p">]</span>
<span class="n">slot_name</span> <span class="o">=</span> <span class="n">each_meta</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="s1">&#39;</span><span class="si">%s</span><span class="s1">_id&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_name</span> <span class="o">==</span> <span class="s1">&#39;id&#39;</span><span class="p">:</span>
<span class="n">slot_dim</span> <span class="o">=</span> <span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;max&#39;</span><span class="p">]</span>
<span class="n">embedding</span> <span class="o">=</span> <span class="n">embedding_layer</span><span class="p">(</span>
<span class="nb">input</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span>
<span class="n">slot_name</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">slot_dim</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">embedding</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">type_name</span> <span class="o">==</span> <span class="s1">&#39;embedding&#39;</span><span class="p">:</span>
<span class="n">is_seq</span> <span class="o">=</span> <span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;seq&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;sequence&#39;</span>
<span class="n">slot_dim</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;dict&#39;</span><span class="p">])</span>
<span class="n">din</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">slot_name</span><span class="p">,</span> <span class="n">slot_dim</span><span class="p">)</span>
<span class="n">embedding</span> <span class="o">=</span> <span class="n">embedding_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">din</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="k">if</span> <span class="n">is_seq</span><span class="p">:</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="n">text_conv_pool</span><span class="p">(</span>
<span class="nb">input</span><span class="o">=</span><span class="n">embedding</span><span class="p">,</span> <span class="n">context_len</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">hidden_size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">embedding</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">type_name</span> <span class="o">==</span> <span class="s1">&#39;one_hot_dense&#39;</span><span class="p">:</span>
<span class="n">slot_dim</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;dict&#39;</span><span class="p">])</span>
<span class="n">hidden</span> <span class="o">=</span> <span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span><span class="n">slot_name</span><span class="p">,</span> <span class="n">slot_dim</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">hidden</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">return</span> <span class="n">fc_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;</span><span class="si">%s</span><span class="s2">_fusion&quot;</span> <span class="o">%</span> <span class="n">name</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">fusion</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">movie_feature</span> <span class="o">=</span> <span class="n">construct_feature</span><span class="p">(</span><span class="s2">&quot;movie&quot;</span><span class="p">)</span>
<span class="n">user_feature</span> <span class="o">=</span> <span class="n">construct_feature</span><span class="p">(</span><span class="s2">&quot;user&quot;</span><span class="p">)</span>
<span class="n">similarity</span> <span class="o">=</span> <span class="n">cos_sim</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="n">movie_feature</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="n">user_feature</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">is_predict</span><span class="p">:</span>
<span class="n">outputs</span><span class="p">(</span><span class="n">mse_cost</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">similarity</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span><span class="s1">&#39;rating&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">)))</span>
<span class="n">define_py_data_sources2</span><span class="p">(</span>
<span class="s1">&#39;data/train.list&#39;</span><span class="p">,</span>
<span class="s1">&#39;data/test.list&#39;</span><span class="p">,</span>
<span class="n">module</span><span class="o">=</span><span class="s1">&#39;dataprovider&#39;</span><span class="p">,</span>
<span class="n">obj</span><span class="o">=</span><span class="s1">&#39;process&#39;</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;meta&#39;</span><span class="p">:</span> <span class="n">meta</span><span class="p">})</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">outputs</span><span class="p">(</span><span class="n">similarity</span><span class="p">)</span>
</pre></div>
</div>
<p>In this <code class="code docutils literal"><span class="pre">trainer_config.py</span></code>, we just map each feature type to
a feature vector, following shows how to map each feature to a vector shows below.</p>
<ul class="simple">
......@@ -624,82 +437,6 @@ features.</p>
</div>
<div class="section" id="data-provider">
<h3>Data Provider<a class="headerlink" href="#data-provider" title="Permalink to this headline"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">import</span> <span class="nn">common_utils</span> <span class="c1"># parse</span>
<span class="k">def</span> <span class="nf">__list_to_map__</span><span class="p">(</span><span class="n">lst</span><span class="p">):</span>
<span class="n">ret_val</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="k">for</span> <span class="n">each</span> <span class="ow">in</span> <span class="n">lst</span><span class="p">:</span>
<span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="o">=</span> <span class="n">each</span>
<span class="n">ret_val</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span>
<span class="k">return</span> <span class="n">ret_val</span>
<span class="k">def</span> <span class="nf">hook</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">meta</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Init hook is invoked before process data. It will set obj.slots and store</span>
<span class="sd"> data meta.</span>
<span class="sd"> :param obj: global object. It will passed to process routine.</span>
<span class="sd"> :type obj: object</span>
<span class="sd"> :param meta: the meta file object, which passed from trainer_config. Meta</span>
<span class="sd"> file record movie/user features.</span>
<span class="sd"> :param kwargs: unused other arguments.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">del</span> <span class="n">kwargs</span> <span class="c1"># unused kwargs</span>
<span class="c1"># Header define slots that used for paddle.</span>
<span class="c1"># first part is movie features.</span>
<span class="c1"># second part is user features.</span>
<span class="c1"># final part is rating score.</span>
<span class="c1"># header is a list of [USE_SEQ_OR_NOT?, SlotType]</span>
<span class="n">movie_headers</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">common_utils</span><span class="o">.</span><span class="n">meta_to_header</span><span class="p">(</span><span class="n">meta</span><span class="p">,</span> <span class="s1">&#39;movie&#39;</span><span class="p">))</span>
<span class="n">settings</span><span class="o">.</span><span class="n">movie_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">h</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">h</span> <span class="ow">in</span> <span class="n">movie_headers</span><span class="p">]</span>
<span class="n">headers</span> <span class="o">=</span> <span class="n">movie_headers</span>
<span class="n">user_headers</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">common_utils</span><span class="o">.</span><span class="n">meta_to_header</span><span class="p">(</span><span class="n">meta</span><span class="p">,</span> <span class="s1">&#39;user&#39;</span><span class="p">))</span>
<span class="n">settings</span><span class="o">.</span><span class="n">user_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">h</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">h</span> <span class="ow">in</span> <span class="n">user_headers</span><span class="p">]</span>
<span class="n">headers</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">user_headers</span><span class="p">)</span>
<span class="n">headers</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="s2">&quot;rating&quot;</span><span class="p">,</span> <span class="n">dense_vector</span><span class="p">(</span><span class="mi">1</span><span class="p">)))</span> <span class="c1"># Score</span>
<span class="c1"># slot types.</span>
<span class="n">settings</span><span class="o">.</span><span class="n">input_types</span> <span class="o">=</span> <span class="n">__list_to_map__</span><span class="p">(</span><span class="n">headers</span><span class="p">)</span>
<span class="n">settings</span><span class="o">.</span><span class="n">meta</span> <span class="o">=</span> <span class="n">meta</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">init_hook</span><span class="o">=</span><span class="n">hook</span><span class="p">,</span> <span class="n">cache</span><span class="o">=</span><span class="n">CacheType</span><span class="o">.</span><span class="n">CACHE_PASS_IN_MEM</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>
<span class="c1"># Get a rating from file.</span>
<span class="n">user_id</span><span class="p">,</span> <span class="n">movie_id</span><span class="p">,</span> <span class="n">score</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39;::&#39;</span><span class="p">)[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c1"># Scale score to [-5, +5]</span>
<span class="n">score</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">score</span><span class="p">)</span> <span class="o">*</span> <span class="mi">2</span> <span class="o">-</span> <span class="mf">5.0</span>
<span class="c1"># Get movie/user features by movie_id, user_id</span>
<span class="n">movie_meta</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s1">&#39;movie&#39;</span><span class="p">][</span><span class="n">movie_id</span><span class="p">]</span>
<span class="n">user_meta</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s1">&#39;user&#39;</span><span class="p">][</span><span class="n">user_id</span><span class="p">]</span>
<span class="n">outputs</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">&#39;movie_id&#39;</span><span class="p">,</span> <span class="n">movie_id</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)]</span>
<span class="c1"># Then add movie features</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">each_meta</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">movie_meta</span><span class="p">):</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">settings</span><span class="o">.</span><span class="n">movie_names</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">],</span> <span class="n">each_meta</span><span class="p">))</span>
<span class="c1"># Then add user id.</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="s1">&#39;user_id&#39;</span><span class="p">,</span> <span class="n">user_id</span> <span class="o">-</span> <span class="mi">1</span><span class="p">))</span>
<span class="c1"># Then add user features.</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">each_meta</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">user_meta</span><span class="p">):</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">settings</span><span class="o">.</span><span class="n">user_names</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">],</span> <span class="n">each_meta</span><span class="p">))</span>
<span class="c1"># Finally, add score</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="s1">&#39;rating&#39;</span><span class="p">,</span> <span class="p">[</span><span class="n">score</span><span class="p">]))</span>
<span class="c1"># Return data to paddle</span>
<span class="k">yield</span> <span class="n">__list_to_map__</span><span class="p">(</span><span class="n">outputs</span><span class="p">)</span>
</pre></div>
</div>
<p>The data provider just read the meta.bin and rating file, yield each sample for training.
In this <code class="code docutils literal"><span class="pre">dataprovider.py</span></code>, we should set:</p>
<ul class="simple">
......@@ -714,18 +451,6 @@ In this <code class="code docutils literal"><span class="pre">dataprovider.py</s
<h2>Train<a class="headerlink" href="#train" title="Permalink to this headline"></a></h2>
<p>After prepare data, config network, writting data provider, now we can run paddle training.</p>
<p>The <code class="code docutils literal"><span class="pre">run.sh</span></code> is shown as follow:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>paddle train <span class="se">\</span>
--config<span class="o">=</span>trainer_config.py <span class="se">\</span>
--save_dir<span class="o">=</span>./output <span class="se">\</span>
--use_gpu<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
--trainer_count<span class="o">=</span><span class="m">4</span><span class="se">\</span>
--test_all_data_in_one_period<span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
--log_period<span class="o">=</span><span class="m">100</span> <span class="se">\</span>
--dot_period<span class="o">=</span><span class="m">1</span> <span class="se">\</span>
--num_passes<span class="o">=</span><span class="m">50</span> <span class="m">2</span>&gt;<span class="p">&amp;</span><span class="m">1</span> <span class="p">|</span> tee <span class="s1">&#39;log.txt&#39;</span>
paddle usage -l log.txt -e <span class="nv">$?</span> -n <span class="s2">&quot;recommendation&quot;</span> &gt;/dev/null <span class="m">2</span>&gt;<span class="p">&amp;</span><span class="m">1</span>
</pre></div>
</div>
<p>It just start a paddle training process, write the log to <code class="code docutils literal"><span class="pre">log.txt</span></code>,
then print it on screen.</p>
<p>Each command line argument in <code class="code docutils literal"><span class="pre">run.sh</span></code>, please refer to the <a class="reference internal" href="../../howto/usage/cmd_parameter/index_en.html#cmd-line-index"><span class="std std-ref">Set Command-line Parameters</span></a> page. The short description of these arguments is shown as follow.</p>
......
develop/doc_cn/searchindex.js
浏览文件 @ 82e956b5
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc_cn/tutorials/quick_start/index_cn.html
浏览文件 @ 82e956b5
......@@ -257,147 +257,11 @@
<li>process：PaddlePaddle调用process函数来读取数据。每次读取一条数据后，process函数会用yield语句输出这条数据，从而能够被PaddlePaddle 捕获 (harvest)。</li>
</ul>
<p><code class="docutils literal"><span class="pre">dataprovider_bow.py</span></code> 文件给出了完整例子：</p>
<div class="highlight-python"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># initializer is called by the framework during initialization.</span>
<span class="c1"># It allows the user to describe the data types and setup the</span>
<span class="c1"># necessary data structure for later use.</span>
<span class="c1"># `settings` is an object. initializer need to properly fill settings.input_types.</span>
<span class="c1"># initializer can also store other data structures needed to be used at process().</span>
<span class="c1"># In this example, dictionary is stored in settings.</span>
<span class="c1"># `dictionay` and `kwargs` are arguments passed from trainer_config.lr.py</span>
<span class="hll"><span class="k">def</span> <span class="nf">initializer</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">dictionary</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
</span> <span class="c1"># Put the word dictionary into settings</span>
<span class="n">settings</span><span class="o">.</span><span class="n">word_dict</span> <span class="o">=</span> <span class="n">dictionary</span>
<span class="c1"># setting.input_types specifies what the data types the data provider</span>
<span class="c1"># generates.</span>
<span class="n">settings</span><span class="o">.</span><span class="n">input_types</span> <span class="o">=</span> <span class="p">{</span>
<span class="c1"># The first input is a sparse_binary_vector,</span>
<span class="c1"># which means each dimension of the vector is either 0 or 1. It is the</span>
<span class="c1"># bag-of-words (BOW) representation of the texts.</span>
<span class="s1">&#39;word&#39;</span><span class="p">:</span> <span class="n">sparse_binary_vector</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">dictionary</span><span class="p">)),</span>
<span class="c1"># The second input is an integer. It represents the category id of the</span>
<span class="c1"># sample. 2 means there are two labels in the dataset.</span>
<span class="c1"># (1 for positive and 0 for negative)</span>
<span class="s1">&#39;label&#39;</span><span class="p">:</span> <span class="n">integer_value</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1"># Delaring a data provider. It has an initializer &#39;data_initialzer&#39;.</span>
<span class="c1"># It will cache the generated data of the first pass in memory, so that</span>
<span class="c1"># during later pass, no on-the-fly data generation will be needed.</span>
<span class="c1"># `setting` is the same object used by initializer()</span>
<span class="c1"># `file_name` is the name of a file listed train_list or test_list file given</span>
<span class="c1"># to define_py_data_sources2(). See trainer_config.lr.py.</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">init_hook</span><span class="o">=</span><span class="n">initializer</span><span class="p">,</span> <span class="n">cache</span><span class="o">=</span><span class="n">CacheType</span><span class="o">.</span><span class="n">CACHE_PASS_IN_MEM</span><span class="p">)</span>
<span class="hll"><span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">file_name</span><span class="p">):</span>
</span> <span class="c1"># Open the input data file.</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_name</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="c1"># Read each line.</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>
<span class="c1"># Each line contains the label and text of the comment, separated by \t.</span>
<span class="n">label</span><span class="p">,</span> <span class="n">comment</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\t</span><span class="s1">&#39;</span><span class="p">)</span>
<span class="c1"># Split the words into a list.</span>
<span class="n">words</span> <span class="o">=</span> <span class="n">comment</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="c1"># convert the words into a list of ids by looking them up in word_dict.</span>
<span class="n">word_vector</span> <span class="o">=</span> <span class="p">[</span><span class="n">settings</span><span class="o">.</span><span class="n">word_dict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="n">UNK_IDX</span><span class="p">)</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">words</span><span class="p">]</span>
<span class="c1"># Return the features for the current comment. The first is a list</span>
<span class="c1"># of ids representing a 0-1 binary sparse vector of the text,</span>
<span class="c1"># the second is the integer id of the label.</span>
<span class="k">yield</span> <span class="p">{</span><span class="s1">&#39;word&#39;</span><span class="p">:</span> <span class="n">word_vector</span><span class="p">,</span> <span class="s1">&#39;label&#39;</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">label</span><span class="p">)}</span>
</pre></div>
</td></tr></table></div>
<p>详细内容请参见 <a class="reference internal" href="../../api/v1/data_provider/dataprovider_cn.html#api-dataprovider"><span class="std std-ref">DataProvider的介绍</span></a> 。</p>
</div>
<div class="section" id="id8">
<h3>配置中的数据加载定义<a class="headerlink" href="#id8" title="永久链接至标题"></a></h3>
<p>在模型配置中通过 <code class="docutils literal"><span class="pre">define_py_data_sources2</span></code> 接口来加载数据：</p>
<div class="highlight-python"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">dict_file</span> <span class="o">=</span> <span class="s2">&quot;./data/dict.txt&quot;</span>
<span class="n">word_dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">dict_file</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="n">w</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">word_dict</span><span class="p">[</span><span class="n">w</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span>
<span class="n">is_predict</span> <span class="o">=</span> <span class="n">get_config_arg</span><span class="p">(</span><span class="s1">&#39;is_predict&#39;</span><span class="p">,</span> <span class="nb">bool</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
<span class="n">trn</span> <span class="o">=</span> <span class="s1">&#39;data/train.list&#39;</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">is_predict</span> <span class="k">else</span> <span class="bp">None</span>
<span class="n">tst</span> <span class="o">=</span> <span class="s1">&#39;data/test.list&#39;</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">is_predict</span> <span class="k">else</span> <span class="s1">&#39;data/pred.list&#39;</span>
<span class="n">process</span> <span class="o">=</span> <span class="s1">&#39;process&#39;</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">is_predict</span> <span class="k">else</span> <span class="s1">&#39;process_predict&#39;</span>
<span class="hll"><span class="n">define_py_data_sources2</span><span class="p">(</span>
</span> <span class="n">train_list</span><span class="o">=</span><span class="n">trn</span><span class="p">,</span>
<span class="n">test_list</span><span class="o">=</span><span class="n">tst</span><span class="p">,</span>
<span class="n">module</span><span class="o">=</span><span class="s2">&quot;dataprovider_emb&quot;</span><span class="p">,</span>
<span class="n">obj</span><span class="o">=</span><span class="n">process</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;dictionary&quot;</span><span class="p">:</span> <span class="n">word_dict</span><span class="p">})</span>
</pre></div>
</td></tr></table></div>
<p>以下是对上述数据加载的解释：</p>
<ul class="simple">
<li>data/train.list,data/test.list: 指定训练数据和测试数据</li>
......
develop/doc_cn/tutorials/rec/ml_regression_cn.html
浏览文件 @ 82e956b5
......@@ -221,24 +221,6 @@
<li>用户文件中有四种类型的字段: 编号，性别，年龄和职业；</li>
<li>文件名称为&#8221;users.dat&#8221;，文件的分隔符为&#8221;::&#8221;。</li>
</ol>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">&quot;user&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;users.dat&quot;</span><span class="p">,</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;id&quot;</span><span class="p">,</span> <span class="s2">&quot;gender&quot;</span><span class="p">,</span> <span class="s2">&quot;age&quot;</span><span class="p">,</span> <span class="s2">&quot;occupation&quot;</span><span class="p">]</span>
<span class="p">},</span>
<span class="s2">&quot;movie&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;movies.dat&quot;</span><span class="p">,</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;id&quot;</span><span class="p">,</span> <span class="s2">&quot;title&quot;</span><span class="p">,</span> <span class="s2">&quot;genres&quot;</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="id4">
......@@ -270,89 +252,6 @@ python config_generator.py config.json &gt; meta_config.json
</pre></div>
</div>
<p>生成的meta配置文件如下所示：</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="s2">&quot;meta&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;movie&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;id&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;regex&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;pattern&quot;</span><span class="p">:</span> <span class="s2">&quot;^(.*)</span><span class="se">\\</span><span class="s2">((</span><span class="se">\\</span><span class="s2">d+)</span><span class="se">\\</span><span class="s2">)$&quot;</span><span class="p">,</span>
<span class="s2">&quot;group_id&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;strip&quot;</span><span class="p">:</span> <span class="n">true</span>
<span class="p">},</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;seq_type&quot;</span><span class="p">:</span> <span class="s2">&quot;sequence&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;char_based&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;title&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">1</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;one_hot_dense&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;|&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;split&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;genres&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;split&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;movies.dat&quot;</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s2">&quot;user&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;fields&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;id&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;char_based&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;gender&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">1</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;sort&quot;</span><span class="p">:</span> <span class="n">true</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;whole_content&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;age&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">2</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;embedding&quot;</span><span class="p">,</span>
<span class="s2">&quot;dict&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;sort&quot;</span><span class="p">:</span> <span class="s2">&quot;true&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;whole_content&quot;</span>
<span class="p">},</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;occupation&quot;</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="mi">3</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="s2">&quot;file&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;delimiter&quot;</span><span class="p">:</span> <span class="s2">&quot;::&quot;</span><span class="p">,</span>
<span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="s2">&quot;split&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;users.dat&quot;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<p>在meta文件中有两种特征: 电影和用户。</p>
<ul class="simple">
<li><dl class="first docutils">
......@@ -506,92 +405,6 @@ cp ml-1m/ratings.dat.test .
<p>网络结构如下图所示：</p>
<img alt="rec_regression_network" class="align-center" src="../../_images/rec_regression_network.png" />
<p>该示例的神经网络配置文件 <code class="code docutils literal"><span class="pre">trainer_config.py</span></code> 如下所示：</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="kn">import</span> <span class="o">*</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">cPickle</span> <span class="kn">as</span> <span class="nn">pickle</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">pickle</span>
<span class="n">is_predict</span> <span class="o">=</span> <span class="n">get_config_arg</span><span class="p">(</span><span class="s1">&#39;is_predict&#39;</span><span class="p">,</span> <span class="nb">bool</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
<span class="n">META_FILE</span> <span class="o">=</span> <span class="s1">&#39;data/meta.bin&#39;</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">META_FILE</span><span class="p">,</span> <span class="s1">&#39;rb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="c1"># load meta file</span>
<span class="n">meta</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">settings</span><span class="p">(</span>
<span class="n">batch_size</span><span class="o">=</span><span class="mi">1600</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">1e-3</span><span class="p">,</span> <span class="n">learning_method</span><span class="o">=</span><span class="n">RMSPropOptimizer</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">construct_feature</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Construct movie/user features.</span>
<span class="sd"> This method read from meta data. Then convert feature to neural network due</span>
<span class="sd"> to feature type. The map relation as follow.</span>
<span class="sd"> * id: embedding =&gt; fc</span>
<span class="sd"> * embedding:</span>
<span class="sd"> is_sequence: embedding =&gt; context_projection =&gt; fc =&gt; pool</span>
<span class="sd"> not sequence: embedding =&gt; fc</span>
<span class="sd"> * one_hot_dense: fc =&gt; fc</span>
<span class="sd"> Then gather all features vector, and use a fc layer to combined them as</span>
<span class="sd"> return.</span>
<span class="sd"> :param name: &#39;movie&#39; or &#39;user&#39;</span>
<span class="sd"> :type name: basestring</span>
<span class="sd"> :return: combined feature output</span>
<span class="sd"> :rtype: LayerOutput</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">__meta__</span> <span class="o">=</span> <span class="n">meta</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="s1">&#39;__meta__&#39;</span><span class="p">][</span><span class="s1">&#39;raw_meta&#39;</span><span class="p">]</span>
<span class="n">fusion</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">each_meta</span> <span class="ow">in</span> <span class="n">__meta__</span><span class="p">:</span>
<span class="n">type_name</span> <span class="o">=</span> <span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;type&#39;</span><span class="p">]</span>
<span class="n">slot_name</span> <span class="o">=</span> <span class="n">each_meta</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="s1">&#39;</span><span class="si">%s</span><span class="s1">_id&#39;</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_name</span> <span class="o">==</span> <span class="s1">&#39;id&#39;</span><span class="p">:</span>
<span class="n">slot_dim</span> <span class="o">=</span> <span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;max&#39;</span><span class="p">]</span>
<span class="n">embedding</span> <span class="o">=</span> <span class="n">embedding_layer</span><span class="p">(</span>
<span class="nb">input</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span>
<span class="n">slot_name</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">slot_dim</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">embedding</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">type_name</span> <span class="o">==</span> <span class="s1">&#39;embedding&#39;</span><span class="p">:</span>
<span class="n">is_seq</span> <span class="o">=</span> <span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;seq&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s1">&#39;sequence&#39;</span>
<span class="n">slot_dim</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;dict&#39;</span><span class="p">])</span>
<span class="n">din</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">slot_name</span><span class="p">,</span> <span class="n">slot_dim</span><span class="p">)</span>
<span class="n">embedding</span> <span class="o">=</span> <span class="n">embedding_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">din</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="k">if</span> <span class="n">is_seq</span><span class="p">:</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span>
<span class="n">text_conv_pool</span><span class="p">(</span>
<span class="nb">input</span><span class="o">=</span><span class="n">embedding</span><span class="p">,</span> <span class="n">context_len</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">hidden_size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">embedding</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">type_name</span> <span class="o">==</span> <span class="s1">&#39;one_hot_dense&#39;</span><span class="p">:</span>
<span class="n">slot_dim</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">each_meta</span><span class="p">[</span><span class="s1">&#39;dict&#39;</span><span class="p">])</span>
<span class="n">hidden</span> <span class="o">=</span> <span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span><span class="n">slot_name</span><span class="p">,</span> <span class="n">slot_dim</span><span class="p">),</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">fusion</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fc_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">hidden</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">))</span>
<span class="k">return</span> <span class="n">fc_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;</span><span class="si">%s</span><span class="s2">_fusion&quot;</span> <span class="o">%</span> <span class="n">name</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">fusion</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">movie_feature</span> <span class="o">=</span> <span class="n">construct_feature</span><span class="p">(</span><span class="s2">&quot;movie&quot;</span><span class="p">)</span>
<span class="n">user_feature</span> <span class="o">=</span> <span class="n">construct_feature</span><span class="p">(</span><span class="s2">&quot;user&quot;</span><span class="p">)</span>
<span class="n">similarity</span> <span class="o">=</span> <span class="n">cos_sim</span><span class="p">(</span><span class="n">a</span><span class="o">=</span><span class="n">movie_feature</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="n">user_feature</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">is_predict</span><span class="p">:</span>
<span class="n">outputs</span><span class="p">(</span><span class="n">mse_cost</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">similarity</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span><span class="s1">&#39;rating&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">)))</span>
<span class="n">define_py_data_sources2</span><span class="p">(</span>
<span class="s1">&#39;data/train.list&#39;</span><span class="p">,</span>
<span class="s1">&#39;data/test.list&#39;</span><span class="p">,</span>
<span class="n">module</span><span class="o">=</span><span class="s1">&#39;dataprovider&#39;</span><span class="p">,</span>
<span class="n">obj</span><span class="o">=</span><span class="s1">&#39;process&#39;</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;meta&#39;</span><span class="p">:</span> <span class="n">meta</span><span class="p">})</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">outputs</span><span class="p">(</span><span class="n">similarity</span><span class="p">)</span>
</pre></div>
</div>
<p>在文件 <code class="code docutils literal"><span class="pre">trainer_config.py</span></code> 中，我们仅仅是将每个特征种类映射到一个特征向量中，以下
展示了如何将每个特征映射到一个向量。</p>
<ul class="simple">
......@@ -631,82 +444,6 @@ cp ml-1m/ratings.dat.test .
</div>
<div class="section" id="id9">
<h3>数据提供脚本<a class="headerlink" href="#id9" title="永久链接至标题"></a></h3>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">import</span> <span class="nn">common_utils</span> <span class="c1"># parse</span>
<span class="k">def</span> <span class="nf">__list_to_map__</span><span class="p">(</span><span class="n">lst</span><span class="p">):</span>
<span class="n">ret_val</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="k">for</span> <span class="n">each</span> <span class="ow">in</span> <span class="n">lst</span><span class="p">:</span>
<span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="o">=</span> <span class="n">each</span>
<span class="n">ret_val</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span>
<span class="k">return</span> <span class="n">ret_val</span>
<span class="k">def</span> <span class="nf">hook</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">meta</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Init hook is invoked before process data. It will set obj.slots and store</span>
<span class="sd"> data meta.</span>
<span class="sd"> :param obj: global object. It will passed to process routine.</span>
<span class="sd"> :type obj: object</span>
<span class="sd"> :param meta: the meta file object, which passed from trainer_config. Meta</span>
<span class="sd"> file record movie/user features.</span>
<span class="sd"> :param kwargs: unused other arguments.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">del</span> <span class="n">kwargs</span> <span class="c1"># unused kwargs</span>
<span class="c1"># Header define slots that used for paddle.</span>
<span class="c1"># first part is movie features.</span>
<span class="c1"># second part is user features.</span>
<span class="c1"># final part is rating score.</span>
<span class="c1"># header is a list of [USE_SEQ_OR_NOT?, SlotType]</span>
<span class="n">movie_headers</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">common_utils</span><span class="o">.</span><span class="n">meta_to_header</span><span class="p">(</span><span class="n">meta</span><span class="p">,</span> <span class="s1">&#39;movie&#39;</span><span class="p">))</span>
<span class="n">settings</span><span class="o">.</span><span class="n">movie_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">h</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">h</span> <span class="ow">in</span> <span class="n">movie_headers</span><span class="p">]</span>
<span class="n">headers</span> <span class="o">=</span> <span class="n">movie_headers</span>
<span class="n">user_headers</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">common_utils</span><span class="o">.</span><span class="n">meta_to_header</span><span class="p">(</span><span class="n">meta</span><span class="p">,</span> <span class="s1">&#39;user&#39;</span><span class="p">))</span>
<span class="n">settings</span><span class="o">.</span><span class="n">user_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">h</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">h</span> <span class="ow">in</span> <span class="n">user_headers</span><span class="p">]</span>
<span class="n">headers</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">user_headers</span><span class="p">)</span>
<span class="n">headers</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="s2">&quot;rating&quot;</span><span class="p">,</span> <span class="n">dense_vector</span><span class="p">(</span><span class="mi">1</span><span class="p">)))</span> <span class="c1"># Score</span>
<span class="c1"># slot types.</span>
<span class="n">settings</span><span class="o">.</span><span class="n">input_types</span> <span class="o">=</span> <span class="n">__list_to_map__</span><span class="p">(</span><span class="n">headers</span><span class="p">)</span>
<span class="n">settings</span><span class="o">.</span><span class="n">meta</span> <span class="o">=</span> <span class="n">meta</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">init_hook</span><span class="o">=</span><span class="n">hook</span><span class="p">,</span> <span class="n">cache</span><span class="o">=</span><span class="n">CacheType</span><span class="o">.</span><span class="n">CACHE_PASS_IN_MEM</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>
<span class="c1"># Get a rating from file.</span>
<span class="n">user_id</span><span class="p">,</span> <span class="n">movie_id</span><span class="p">,</span> <span class="n">score</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39;::&#39;</span><span class="p">)[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c1"># Scale score to [-5, +5]</span>
<span class="n">score</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">score</span><span class="p">)</span> <span class="o">*</span> <span class="mi">2</span> <span class="o">-</span> <span class="mf">5.0</span>
<span class="c1"># Get movie/user features by movie_id, user_id</span>
<span class="n">movie_meta</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s1">&#39;movie&#39;</span><span class="p">][</span><span class="n">movie_id</span><span class="p">]</span>
<span class="n">user_meta</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">meta</span><span class="p">[</span><span class="s1">&#39;user&#39;</span><span class="p">][</span><span class="n">user_id</span><span class="p">]</span>
<span class="n">outputs</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">&#39;movie_id&#39;</span><span class="p">,</span> <span class="n">movie_id</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)]</span>
<span class="c1"># Then add movie features</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">each_meta</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">movie_meta</span><span class="p">):</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">settings</span><span class="o">.</span><span class="n">movie_names</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">],</span> <span class="n">each_meta</span><span class="p">))</span>
<span class="c1"># Then add user id.</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="s1">&#39;user_id&#39;</span><span class="p">,</span> <span class="n">user_id</span> <span class="o">-</span> <span class="mi">1</span><span class="p">))</span>
<span class="c1"># Then add user features.</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">each_meta</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">user_meta</span><span class="p">):</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">settings</span><span class="o">.</span><span class="n">user_names</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">],</span> <span class="n">each_meta</span><span class="p">))</span>
<span class="c1"># Finally, add score</span>
<span class="n">outputs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="s1">&#39;rating&#39;</span><span class="p">,</span> <span class="p">[</span><span class="n">score</span><span class="p">]))</span>
<span class="c1"># Return data to paddle</span>
<span class="k">yield</span> <span class="n">__list_to_map__</span><span class="p">(</span><span class="n">outputs</span><span class="p">)</span>
</pre></div>
</div>
<p>数据提供脚本仅仅是读取meta.bin和评分文件，生成训练需要的样本。
在脚本 <code class="code docutils literal"><span class="pre">dataprovider.py</span></code> 中，我们需要设置：</p>
<ul class="simple">
......@@ -721,18 +458,6 @@ cp ml-1m/ratings.dat.test .
<h2>训练<a class="headerlink" href="#id10" title="永久链接至标题"></a></h2>
<p>准备好数据，配置了网络，编写好数据提供脚本后，现在我们可以开始paddle训练了。</p>
<p>代码 <code class="code docutils literal"><span class="pre">run.sh</span></code> 如下：</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>paddle train <span class="se">\</span>
--config<span class="o">=</span>trainer_config.py <span class="se">\</span>
--save_dir<span class="o">=</span>./output <span class="se">\</span>
--use_gpu<span class="o">=</span><span class="nb">false</span> <span class="se">\</span>
--trainer_count<span class="o">=</span><span class="m">4</span><span class="se">\</span>
--test_all_data_in_one_period<span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
--log_period<span class="o">=</span><span class="m">100</span> <span class="se">\</span>
--dot_period<span class="o">=</span><span class="m">1</span> <span class="se">\</span>
--num_passes<span class="o">=</span><span class="m">50</span> <span class="m">2</span>&gt;<span class="p">&amp;</span><span class="m">1</span> <span class="p">|</span> tee <span class="s1">&#39;log.txt&#39;</span>
paddle usage -l log.txt -e <span class="nv">$?</span> -n <span class="s2">&quot;recommendation&quot;</span> &gt;/dev/null <span class="m">2</span>&gt;<span class="p">&amp;</span><span class="m">1</span>
</pre></div>
</div>
<p>该脚本仅仅是开始一个paddle训练过程，将日志写入文件 <code class="code docutils literal"><span class="pre">log.txt</span></code> ，然后
打印在屏幕上。</p>
<p>脚本 <code class="code docutils literal"><span class="pre">run.sh</span></code> 中的每一行命令，请参考页面 <a class="reference internal" href="../../howto/usage/cmd_parameter/index_cn.html#cmd-line-index"><span class="std std-ref">设置命令行参数</span></a> 。
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或