Skip to content

P
Paddle

PaddlePaddle / Paddle
大约 1 年 前同步成功

通知 2299
Star 20931
Fork 5422
P
Paddle
提交 f2cb48f1 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: e983cc90

上级 d0881548
隐藏空白更改
内联 并排
Showing with 168 addition and 149 deletion
develop/doc/api/v1/data_provider/pydataprovider2_en.html
浏览文件 @ f2cb48f1
......@@ -239,19 +239,20 @@ label of an image. The second part contains 28x28 pixel float values.</p>
</pre></div>
</div>
<p>The corresponding dataprovider is shown as below:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
......@@ -307,19 +308,20 @@ sample by using keywords <code class="code docutils literal"><span class="pre">y
<code class="code docutils literal"><span class="pre">generator</span></code>.</p>
<p>Only a few lines of codes need to be added into the training configuration file,
you can take this as an example.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<span class="n">define_py_data_sources2</span><span class="p">(</span>
......@@ -373,19 +375,21 @@ the <code class="code docutils literal"><span class="pre">dataprovider</span></c
34
35
36
37</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
37
38</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
......@@ -447,19 +451,20 @@ negative sentiment (marked by 0 and 1 respectively).</p>
</pre></div>
</div>
<p>The corresponding data provider can be found in the path below:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
......@@ -520,19 +525,20 @@ configuration file, and it maps word string to word id.</li>
</ul>
<p>To pass these parameters into DataProvider, the following lines should be added
into trainer configuration file.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<span class="n">dictionary</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
......
develop/doc_cn/api/v1/data_provider/pydataprovider2_cn.html
浏览文件 @ f2cb48f1
......@@ -282,19 +282,20 @@
</div>
<div class="section" id="dataprovider">
<h3><a class="toc-backref" href="#id14">dataprovider的使用</a><a class="headerlink" href="#dataprovider" title="永久链接至标题"></a></h3>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
......@@ -327,8 +328,8 @@
<li><p class="first">其次，定义一个Python的 <a class="reference external" href="http://www.learnpython.org/en/Decorators">Decorator</a> <a class="reference internal" href="#provider">&#64;provider</a> 。用于将下一行的数据输入函数标记成一个PyDataProvider2，同时设置它的input_types属性。</p>
<ul>
<li><p class="first"><a class="reference internal" href="#input-types">input_types</a>：设置这个PyDataProvider2返回什么样的数据。本例根据网络配置中 <code class="docutils literal"><span class="pre">data_layer</span></code> 的名字，显式指定返回的是一个28*28维的稠密浮点数向量和一个[0-9]的10维整数标签。</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
</pre></div>
</div>
</li>
......@@ -356,13 +357,13 @@
<div class="section" id="id2">
<h3><a class="toc-backref" href="#id15">网络配置中的调用</a><a class="headerlink" href="#id2" title="永久链接至标题"></a></h3>
<p>在网络配置里，只需要一行代码就可以调用这个PyDataProvider2，如，</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
</pre></div>
</div>
<p>训练数据是 <code class="docutils literal"><span class="pre">train.list</span></code> ，没有测试数据，调用的PyDataProvider2是 <code class="docutils literal"><span class="pre">mnist_provider</span></code> 模块中的 <code class="docutils literal"><span class="pre">process</span></code> 函数。</p>
......@@ -399,19 +400,20 @@
<li>其中 <code class="docutils literal"><span class="pre">input_types</span></code> 和在 <a class="reference internal" href="#provider">&#64;provider</a> 中配置的效果一致。本例中的输入特征是词ID的序列，因此使用 <code class="docutils literal"><span class="pre">integer_value_sequence</span></code> 类型来设置。</li>
<li>将 <code class="docutils literal"><span class="pre">dictionary</span></code> 存入settings对象，在 <code class="docutils literal"><span class="pre">process</span></code> 函数中使用。 dictionary是从网络配置中传入的dict对象，即一个将单词字符串映射到单词ID的字典。</li>
</ul>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
......@@ -465,21 +467,22 @@
<li>在配置中需要读取外部字典。</li>
<li>在声明DataProvider的时候传入dictionary作为参数。</li>
</ul>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="hll"><span class="c1">#See the License for the specific language governing permissions and</span>
</span><span class="hll"><span class="c1">#limitations under the License.</span>
</span><span class="hll"><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="hll"><span class="c1"># See the License for the specific language governing permissions and</span>
</span><span class="hll"><span class="c1"># limitations under the License.</span>
</span><span class="hll">
</span><span class="kn">from</span> <span class="nn">paddle.trainer_config_helpers</span> <span class="k">import</span> <span class="o">*</span>
<span class="n">dictionary</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="o">...</span> <span class="c1"># read dictionary from outside</span>
......
develop/doc_cn/faq/local/index_cn.html
浏览文件 @ f2cb48f1
......@@ -279,19 +279,21 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
<p>所以，减小这个内存池即可减小内存占用，同时也可以加速开始训练前数据载入的过程。但是，这
个内存池实际上决定了shuffle的粒度。所以，如果将这个内存池减小，又要保证数据是随机的，
那么最好将数据文件在每次读取之前做一次shuffle。可能的代码为</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">min_pool_size</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;shuf </span><span class="si">%s</span><span class="s1"> &gt; </span><span class="si">%s</span><span class="s1">.shuf&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">filename</span><span class="p">))</span> <span class="c1"># shuffle before.</span>
......@@ -335,19 +337,21 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
<h3><a class="toc-backref" href="#id15">减少数据载入的耗时</a><a class="headerlink" href="#id6" title="永久链接至标题"></a></h3>
<p>使用<code class="code docutils literal"><span class="pre">pydataprovider</span></code>时，可以减少缓存池的大小，同时设置内存缓存功能，即可以极大的加速数据载入流程。
<code class="code docutils literal"><span class="pre">DataProvider</span></code> 缓存池的减小，和之前减小通过减小缓存池来减小内存占用的原理一致。</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="nd">@provider</span><span class="p">(</span><span class="n">min_pool_size</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="s1">&#39;shuf </span><span class="si">%s</span><span class="s1"> &gt; </span><span class="si">%s</span><span class="s1">.shuf&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">filename</span><span class="p">))</span> <span class="c1"># shuffle before.</span>
......@@ -363,19 +367,20 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
<p>PaddlePaddle支持Sparse的训练，sparse训练需要训练特征是 <code class="code docutils literal"><span class="pre">sparse_binary_vector</span></code> 、 <code class="code docutils literal"><span class="pre">sparse_vector</span></code> 、或者 <code class="code docutils literal"><span class="pre">integer_value</span></code> 的任一一种。同时，与这个训练数据交互的Layer，需要将其Parameter设置成 sparse 更新模式，即设置 <code class="code docutils literal"><span class="pre">sparse_update=True</span></code></p>
<p>这里使用简单的 <code class="code docutils literal"><span class="pre">word2vec</span></code> 训练语言模型距离，具体使用方法为:</p>
<p>使用一个词前两个词和后两个词，来预测这个中间的词。这个任务的DataProvider为:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="n">DICT_DIM</span> <span class="o">=</span> <span class="mi">3000</span>
......@@ -389,19 +394,20 @@ PaddlePaddle的内存占用主要分为如下几个方面:</p>
</pre></div>
</div>
<p>这个任务的配置为:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="o">...</span> <span class="c1"># the settings and define data provider is omitted.</span>
<span class="n">DICT_DIM</span> <span class="o">=</span> <span class="mi">3000</span> <span class="c1"># dictionary dimension.</span>
<span class="n">word_ids</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="s1">&#39;word_ids&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">DICT_DIM</span><span class="p">)</span>
......
develop/doc_cn/getstarted/concepts/use_concepts_cn.html
浏览文件 @ f2cb48f1
......@@ -442,19 +442,21 @@ trainer.train<span class="o">(</span>
67
68
69
70</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
70
71</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">import</span> <span class="nn">paddle.v2</span> <span class="k">as</span> <span class="nn">paddle</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
......@@ -545,19 +547,21 @@ trainer.train<span class="o">(</span>
28
29
30
31</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
31
32</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="c1"># Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserve.</span>
<span class="c1">#</span>
<span class="c1">#Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1">#you may not use this file except in compliance with the License.</span>
<span class="c1">#You may obtain a copy of the License at</span>
<span class="c1"># Licensed under the Apache License, Version 2.0 (the &quot;License&quot;);</span>
<span class="c1"># you may not use this file except in compliance with the License.</span>
<span class="c1"># You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1">#Unless required by applicable law or agreed to in writing, software</span>
<span class="c1">#distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1">#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1">#See the License for the specific language governing permissions and</span>
<span class="c1">#limitations under the License.</span>
<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
<span class="c1"># distributed under the License is distributed on an &quot;AS IS&quot; BASIS,</span>
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="kn">import</span> <span class="nn">paddle.v2</span> <span class="k">as</span> <span class="nn">paddle</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或