Skip to content

P
Paddle

GoAI / Paddle
与 Fork 源项目一致

Fork自 PaddlePaddle / Paddle

通知 1
Star 1
Fork 0
P
Paddle

体验新版 GitCode，发现更多精彩内容 >>

提交 42aba945 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: b9b75377

上级 5c49efc0
展开全部 隐藏空白更改
内联 并排
Showing with 238 addition and 43 deletion
develop/doc/api/v1/data_provider/pydataprovider2_en.html
浏览文件 @ 42aba945
......@@ -239,7 +239,20 @@ label of an image. The second part contains 28x28 pixel float values.</p>
</pre></div>
</div>
<p>The corresponding dataprovider is shown as below:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<span class="c1"># Define a py data provider</span>
......@@ -294,7 +307,20 @@ sample by using keywords <code class="code docutils literal"><span class="pre">y
<code class="code docutils literal"><span class="pre">generator</span></code>.</p>
<p>Only a few lines of codes need to be added into the training configuration file,
you can take this as an example.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<span class="n">define_py_data_sources2</span><span class="p">(</span>
<span class="n">train_list</span><span class="o">=</span><span class="s1">&#39;train.list&#39;</span><span class="p">,</span>
......@@ -334,7 +360,33 @@ the <code class="code docutils literal"><span class="pre">dataprovider</span></c
21
22
23
24</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
24
25
26
27
28
29
30
31
32
33
34
35
36
37</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<span class="c1"># Define a py data provider</span>
......@@ -395,7 +447,20 @@ negative sentiment (marked by 0 and 1 respectively).</p>
</pre></div>
</div>
<p>The corresponding data provider can be found in the path below:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<span class="k">def</span> <span class="nf">on_init</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">dictionary</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
......@@ -455,7 +520,20 @@ configuration file, and it maps word string to word id.</li>
</ul>
<p>To pass these parameters into DataProvider, the following lines should be added
into trainer configuration file.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<span class="n">dictionary</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="o">...</span> <span class="c1"># read dictionary from outside</span>
......
develop/doc/searchindex.js
浏览文件 @ 42aba945
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc_cn/api/v1/data_provider/pydataprovider2_cn.html
浏览文件 @ 42aba945
......@@ -282,7 +282,20 @@
</div>
<div class="section" id="dataprovider">
<h3><a class="toc-backref" href="#id14">dataprovider的使用</a><a class="headerlink" href="#dataprovider" title="永久链接至标题"></a></h3>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<span class="c1"># Define a py data provider</span>
......@@ -314,8 +327,8 @@
<li><p class="first">其次，定义一个Python的 <a class="reference external" href="http://www.learnpython.org/en/Decorators">Decorator</a> <a class="reference internal" href="#provider">&#64;provider</a> 。用于将下一行的数据输入函数标记成一个PyDataProvider2，同时设置它的input_types属性。</p>
<ul>
<li><p class="first"><a class="reference internal" href="#input-types">input_types</a>：设置这个PyDataProvider2返回什么样的数据。本例根据网络配置中 <code class="docutils literal"><span class="pre">data_layer</span></code> 的名字，显式指定返回的是一个28*28维的稠密浮点数向量和一个[0-9]的10维整数标签。</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">img</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;pixel&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">784</span><span class="p">)</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;label&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
</pre></div>
</div>
</li>
......@@ -343,13 +356,13 @@
<div class="section" id="id2">
<h3><a class="toc-backref" href="#id15">网络配置中的调用</a><a class="headerlink" href="#id2" title="永久链接至标题"></a></h3>
<p>在网络配置里，只需要一行代码就可以调用这个PyDataProvider2，如，</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<span class="n">define_py_data_sources2</span><span class="p">(</span>
<span class="n">train_list</span><span class="o">=</span><span class="s1">&#39;train.list&#39;</span><span class="p">,</span>
<span class="n">test_list</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">module</span><span class="o">=</span><span class="s1">&#39;mnist_provider&#39;</span><span class="p">,</span>
<span class="n">obj</span><span class="o">=</span><span class="s1">&#39;process&#39;</span><span class="p">)</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
</pre></div>
</div>
<p>训练数据是 <code class="docutils literal"><span class="pre">train.list</span></code> ，没有测试数据，调用的PyDataProvider2是 <code class="docutils literal"><span class="pre">mnist_provider</span></code> 模块中的 <code class="docutils literal"><span class="pre">process</span></code> 函数。</p>
......@@ -386,7 +399,20 @@
<li>其中 <code class="docutils literal"><span class="pre">input_types</span></code> 和在 <a class="reference internal" href="#provider">&#64;provider</a> 中配置的效果一致。本例中的输入特征是词ID的序列，因此使用 <code class="docutils literal"><span class="pre">integer_value_sequence</span></code> 类型来设置。</li>
<li>将 <code class="docutils literal"><span class="pre">dictionary</span></code> 存入settings对象，在 <code class="docutils literal"><span class="pre">process</span></code> 函数中使用。 dictionary是从网络配置中传入的dict对象，即一个将单词字符串映射到单词ID的字典。</li>
</ul>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
<span class="k">def</span> <span class="nf">on_init</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">dictionary</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
......@@ -439,8 +465,21 @@
<li>在配置中需要读取外部字典。</li>
<li>在声明DataProvider的时候传入dictionary作为参数。</li>
</ul>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="hll"><span class="c1">#See the License for the specific language governing permissions and</span>
</span><span class="hll"><span class="c1">#limitations under the License.</span>
</span><span class="hll"><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
</span>
<span class="n">dictionary</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="o">...</span> <span class="c1"># read dictionary from outside</span>
......@@ -450,10 +489,10 @@
<span class="n">module</span><span class="o">=</span><span class="s1">&#39;sentimental_provider&#39;</span><span class="p">,</span>
<span class="n">obj</span><span class="o">=</span><span class="s1">&#39;process&#39;</span><span class="p">,</span>
<span class="c1"># above codes same as mnist sample.</span>
<span class="hll"> <span class="n">args</span><span class="o">=</span><span class="p">{</span> <span class="c1"># pass to provider.</span>
</span><span class="hll"> <span class="s1">&#39;dictionary&#39;</span><span class="p">:</span> <span class="n">dictionary</span>
</span><span class="hll"> <span class="p">})</span>
</span></pre></div>
<span class="n">args</span><span class="o">=</span><span class="p">{</span> <span class="c1"># pass to provider.</span>
<span class="s1">&#39;dictionary&#39;</span><span class="p">:</span> <span class="n">dictionary</span>
<span class="p">})</span>
</pre></div>
</div>
</div>
</div>
......
develop/doc_cn/faq/local/index_cn.html
浏览文件 @ 42aba945
......@@ -279,7 +279,20 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
<p>所以，减小这个内存池即可减小内存占用，同时也可以加速开始训练前数据载入的过程。但是，这
个内存池实际上决定了shuffle的粒度。所以，如果将这个内存池减小，又要保证数据是随机的，
那么最好将数据文件在每次读取之前做一次shuffle。可能的代码为</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nd">@provider</span><span class="p">(</span><span class="n">min_pool_size</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">min_pool_size</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;shuf </span><span class="si">%s</span><span class="s1"> &gt; </span><span class="si">%s</span><span class="s1">.shuf&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">filename</span><span class="p">))</span> <span class="c1"># shuffle before.</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">&#39;</span><span class="si">%s</span><span class="s1">.shuf&#39;</span> <span class="o">%</span> <span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
......@@ -322,7 +335,20 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
<h3><a class="toc-backref" href="#id15">减少数据载入的耗时</a><a class="headerlink" href="#id6" title="永久链接至标题"></a></h3>
<p>使用<code class="code docutils literal"><span class="pre">pydataprovider</span></code>时，可以减少缓存池的大小，同时设置内存缓存功能，即可以极大的加速数据载入流程。
<code class="code docutils literal"><span class="pre">DataProvider</span></code> 缓存池的减小，和之前减小通过减小缓存池来减小内存占用的原理一致。</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="nd">@provider</span><span class="p">(</span><span class="n">min_pool_size</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">min_pool_size</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;shuf </span><span class="si">%s</span><span class="s1"> &gt; </span><span class="si">%s</span><span class="s1">.shuf&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">filename</span><span class="p">))</span> <span class="c1"># shuffle before.</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">&#39;</span><span class="si">%s</span><span class="s1">.shuf&#39;</span> <span class="o">%</span> <span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
......@@ -337,7 +363,20 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
<p>PaddlePaddle支持Sparse的训练，sparse训练需要训练特征是 <code class="code docutils literal"><span class="pre">sparse_binary_vector</span></code> 、 <code class="code docutils literal"><span class="pre">sparse_vector</span></code> 、或者 <code class="code docutils literal"><span class="pre">integer_value</span></code> 的任一一种。同时，与这个训练数据交互的Layer，需要将其Parameter设置成 sparse 更新模式，即设置 <code class="code docutils literal"><span class="pre">sparse_update=True</span></code></p>
<p>这里使用简单的 <code class="code docutils literal"><span class="pre">word2vec</span></code> 训练语言模型距离，具体使用方法为:</p>
<p>使用一个词前两个词和后两个词，来预测这个中间的词。这个任务的DataProvider为:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">DICT_DIM</span> <span class="o">=</span> <span class="mi">3000</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="n">DICT_DIM</span> <span class="o">=</span> <span class="mi">3000</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">input_types</span><span class="o">=</span><span class="p">[</span><span class="n">integer_sequence</span><span class="p">(</span><span class="n">DICT_DIM</span><span class="p">),</span> <span class="n">integer_value</span><span class="p">(</span><span class="n">DICT_DIM</span><span class="p">)])</span>
......@@ -350,7 +389,20 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
</pre></div>
</div>
<p>这个任务的配置为:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">...</span> <span class="c1"># the settings and define data provider is omitted.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="o">...</span> <span class="c1"># the settings and define data provider is omitted.</span>
<span class="n">DICT_DIM</span> <span class="o">=</span> <span class="mi">3000</span> <span class="c1"># dictionary dimension.</span>
<span class="n">word_ids</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="s1">&#39;word_ids&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">DICT_DIM</span><span class="p">)</span>
......
develop/doc_cn/getstarted/concepts/use_concepts_cn.html
浏览文件 @ 42aba945
......@@ -429,7 +429,33 @@ trainer.train<span class="o">(</span>
54
55
56
57</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle.v2</span> <span class="k">as</span> <span class="nn">paddle</span>
57
58
59
60
61
62
63
64
65
66
67
68
69
70</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="kn">import</span> <span class="nn">paddle.v2</span> <span class="k">as</span> <span class="nn">paddle</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="c1"># init paddle</span>
......
develop/doc_cn/howto/deep_model/rnn/hrnn_rnn_api_compare_cn.html
浏览文件 @ 42aba945
......@@ -628,24 +628,24 @@
15
16
17
18</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">step</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">x2</span><span class="p">):</span>
18</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="n">hidden_dim</span> <span class="o">=</span> <span class="mi">8</span>
<span class="n">label_dim</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">speaker1</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;word1&quot;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">dict_dim</span><span class="p">)</span>
<span class="n">speaker2</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;word2&quot;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">dict_dim</span><span class="p">)</span>
<span class="n">emb1</span> <span class="o">=</span> <span class="n">embedding_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">speaker1</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">word_dim</span><span class="p">)</span>
<span class="n">emb2</span> <span class="o">=</span> <span class="n">embedding_layer</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">speaker2</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">word_dim</span><span class="p">)</span>
<span class="c1"># This hierachical RNN is designed to be equivalent to the RNN in</span>
<span class="c1"># sequence_nest_rnn_multi_unequalength_inputs.conf</span>
<span class="k">def</span> <span class="nf">step</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">x2</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">calrnn</span><span class="p">(</span><span class="n">y</span><span class="p">):</span>
<span class="n">mem</span> <span class="o">=</span> <span class="n">memory</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;rnn_state_&#39;</span> <span class="o">+</span> <span class="n">y</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">hidden_dim</span><span class="p">)</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">fc_layer</span><span class="p">(</span>
<span class="nb">input</span><span class="o">=</span><span class="p">[</span><span class="n">y</span><span class="p">,</span> <span class="n">mem</span><span class="p">],</span>
<span class="n">size</span><span class="o">=</span><span class="n">hidden_dim</span><span class="p">,</span>
<span class="n">act</span><span class="o">=</span><span class="n">TanhActivation</span><span class="p">(),</span>
<span class="n">bias_attr</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s1">&#39;rnn_state_&#39;</span> <span class="o">+</span> <span class="n">y</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">return</span> <span class="n">out</span>
<span class="n">encoder1</span> <span class="o">=</span> <span class="n">calrnn</span><span class="p">(</span><span class="n">x1</span><span class="p">)</span>
<span class="n">encoder2</span> <span class="o">=</span> <span class="n">calrnn</span><span class="p">(</span><span class="n">x2</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">encoder1</span><span class="p">,</span> <span class="n">encoder2</span><span class="p">]</span>
<span class="n">encoder1_rep</span><span class="p">,</span> <span class="n">encoder2_rep</span> <span class="o">=</span> <span class="n">recurrent_group</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;stepout&quot;</span><span class="p">,</span> <span class="n">step</span><span class="o">=</span><span class="n">step</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="p">[</span><span class="n">emb1</span><span class="p">,</span> <span class="n">emb2</span><span class="p">])</span>
</pre></div>
</td></tr></table></div>
<ul class="simple">
......
develop/doc_cn/searchindex.js
浏览文件 @ 42aba945
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或