Deploy to GitHub Pages: 90b9cba7

14768828 · Travis CI · 4b97303a · 14768828 · 14768828 · 14768828
10 changed file
--- a/doc/_sources/ui/data_provider/pydataprovider2.txt
+++ b/doc/_sources/ui/data_provider/pydataprovider2.txt
@@ -24,7 +24,7 @@ A small part of the original data as an example is shown as below:

 .. literalinclude:: ../../../doc_cn/ui/data_provider/mnist_train.txt

-Each line of the data contains two parts, separated by ';'. The first part is
+Each line of the data contains two parts, separated by :code:`;`. The first part is
 label of an image. The second part contains 28x28 pixel float values.

 Just write path of the above data into train.list. It looks like this:
@@ -74,7 +74,20 @@ you can take this as an example.

 .. literalinclude:: ../../../doc_cn/ui/data_provider/mnist_config.py

-Here we specify training data by 'train.list', and no testing data is specified.
+Here we specify training data by :code:`train.list`, and no testing data is specified.
+The method which actually provide data is :code:`process`.
+
+User also can use another style to provide data, which defines the
+:code:`data_layer`'s name explicitly when `yield`. For example,
+the :code:`dataprovider` is shown as below.
+
+.. literalinclude:: ../../../doc_cn/ui/data_provider/mnist_provider.dict.py
+   :linenos:
+
+If user did't give the :code:`data_layer`'s name, PaddlePaddle will use
+the order of :code:`data_layer` definition roughly to determine which feature to
+which :code:`data_layer`. This order may be not correct, so TO DEFINE THE
+:code:`data_layer`'s NAMES EXPLICITLY IS THE RECOMMANDED WAY TO PROVIDER DATA.

 Now, this simple example of using PyDataProvider is finished.
 The only thing that the user should know is how to generte **one sample** from
@@ -93,7 +106,7 @@ DataProvider for the sequential model
 -------------------------------------
 A sequence model takes sequences as its input. A sequence is made up of several
 timesteps. The so-called timestep, is not necessary to have something to do
-with 'time'. It can also be explained to that the order of data are taken into
+with time. It can also be explained to that the order of data are taken into
 consideration into model design and training.
 For example, the sentence can be interpreted as a kind of sequence data in NLP
 tasks.
@@ -155,23 +168,7 @@ Reference
 @provider
 +++++++++

-'@provider' is a Python `Decorator`_, it can construct a PyDataProvider in
-PaddlePaddle from a user defined function. Its parameters are:
-
-* `input_types`_ defines format of the data input.
-* should_shuffle defines whether to shuffle data or not. By default, it is set
-  true during training, and false during testing.
-* pool_size is the memory pool size (in sample number) in DataProvider.
-  -1 means no limit.
-* can_over_batch_size defines whether PaddlePaddle can store little more
-  samples than pool_size. It is better to set True to avoid some deadlocks.
-* calc_batch_size is a function define how to calculate batch size. This is
-  usefull in sequential model, that defines batch size is counted upon sequence
-  or token. By default, each sample or sequence counts to 1 when calculating
-  batch size.
-* cache is a data cache strategy, see `cache`_.
-* Init_hook function is invoked once the data provider is initialized,
-  see `init_hook`_.
+.. autofunction:: paddle.trainer.PyDataProvider2.provider

 input_types
 +++++++++++

--- a/doc/genindex.html
+++ b/doc/genindex.html
@@ -6389,7 +6389,7 @@
  </dt>

      
-  <dt><a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle12DataProvider6createERK10DataConfigb">paddle::DataProvider::create (C++ function)</a>
+  <dt><a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle12DataProvider6createERK10DataConfigRK11ModelConfigb">paddle::DataProvider::create (C++ function)</a>, <a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle12DataProvider6createERK10DataConfigb">[1]</a>
  </dt>

      
@@ -9672,12 +9672,12 @@
  <dt><a href="source/math/matrix/matrix.html#_CPPv2N6paddle12MatrixOffset5cCol_E">paddle::MatrixOffset::cCol_ (C++ member)</a>
  </dt>

-  </dl></td>
-  <td style="width: 33%" valign="top"><dl>
      
  <dt><a href="source/math/matrix/matrix.html#_CPPv2N6paddle12MatrixOffset5cRow_E">paddle::MatrixOffset::cRow_ (C++ member)</a>
  </dt>

+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
      
  <dt><a href="source/math/matrix/matrix.html#_CPPv2N6paddle12MatrixOffset5dCol_E">paddle::MatrixOffset::dCol_ (C++ member)</a>
  </dt>
@@ -10143,7 +10143,7 @@
  </dt>

      
-  <dt><a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle17MultiDataProvider17MultiDataProviderERK10DataConfigb">paddle::MultiDataProvider::MultiDataProvider (C++ function)</a>
+  <dt><a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle17MultiDataProvider17MultiDataProviderERK10DataConfigRK11ModelConfigb">paddle::MultiDataProvider::MultiDataProvider (C++ function)</a>
  </dt>

      
@@ -13163,7 +13163,7 @@
  </dt>

      
-  <dt><a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle15PyDataProvider215PyDataProvider2ERK10DataConfigb">paddle::PyDataProvider2::PyDataProvider2 (C++ function)</a>
+  <dt><a href="source/gserver/dataprovider/dataproviders.html#_CPPv2N6paddle15PyDataProvider215PyDataProvider2ERK10DataConfigRK11ModelConfigb">paddle::PyDataProvider2::PyDataProvider2 (C++ function)</a>
  </dt>

      
@@ -16814,6 +16814,10 @@
  <dt><a href="source/api/api.html#_CPPv2N25ParameterTraverseCallbackD0Ev">ParameterTraverseCallback::~ParameterTraverseCallback (C++ function)</a>
  </dt>

+      
+  <dt><a href="ui/data_provider/pydataprovider2.html#paddle.trainer.PyDataProvider2.provider">provider() (in module paddle.trainer.PyDataProvider2)</a>
+  </dt>
+
  </dl></td>
 </tr></table>


--- a/doc/objects.inv
+++ b/doc/objects.inv
--- a/doc/searchindex.js
+++ b/doc/searchindex.js
--- a/doc/source/gserver/dataprovider/dataproviders.html
+++ b/doc/source/gserver/dataprovider/dataproviders.html
@@ -164,16 +164,22 @@
 <div class="breathe-sectiondef container">
 <p class="breathe-sectiondef-title rubric">Public Static Functions</p>
 <dl class="function">
-<dt id="_CPPv2N6paddle12DataProvider6createERK10DataConfigb">
-<span id="paddle::DataProvider::create__DataConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1DataProvider_1ae6b6c293f49094fa37e06504da998248"></span><a class="reference internal" href="#_CPPv2N6paddle12DataProviderE" title="paddle::DataProvider">DataProvider</a> *<code class="descname">create</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, bool <em>useGpu</em> = FLAGS_use_gpu<span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle12DataProvider6createERK10DataConfigb" title="Permalink to this definition">¶</a></dt>
+<dt id="_CPPv2N6paddle12DataProvider6createERK10DataConfigRK11ModelConfigb">
+<span id="paddle::DataProvider::create__DataConfigCR.ModelConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1DataProvider_1a761ef962e6ef45b14562632db6be39c9"></span><a class="reference internal" href="#_CPPv2N6paddle12DataProviderE" title="paddle::DataProvider">DataProvider</a> *<code class="descname">create</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, <em class="property">const</em> <a class="reference internal" href="../../api/api.html#_CPPv211ModelConfig" title="ModelConfig">ModelConfig</a> &amp;<em>modelConfig</em>, bool <em>useGpu</em> = FLAGS_use_gpu<span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle12DataProvider6createERK10DataConfigRK11ModelConfigb" title="Permalink to this definition">¶</a></dt>
 <dd></dd></dl>

+<dl class="function">
+<dt id="_CPPv2N6paddle12DataProvider6createERK10DataConfigb">
+<span id="paddle::DataProvider::create__DataConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1DataProvider_1ad782dc59f7366c19ba4375101159ba95"></span><em class="property">static</em> <a class="reference internal" href="#_CPPv2N6paddle12DataProviderE" title="paddle::DataProvider">DataProvider</a> *<code class="descname">create</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, bool <em>useGpu</em><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle12DataProvider6createERK10DataConfigb" title="Permalink to this definition">¶</a></dt>
+<dd><p>create only used for unittest. </p>
+</dd></dl>
+
 </div>
 <div class="breathe-sectiondef container">
 <p class="breathe-sectiondef-title rubric">Public Static Attributes</p>
 <dl class="member">
 <dt id="_CPPv2N6paddle12DataProvider10registrar_E">
-<span id="paddle::DataProvider::registrar___ClassRegistrar:DataProvider.DataConfig.b:"></span><span class="target" id="paddleclasspaddle_1_1DataProvider_1acc7ff8754097b2ecdd6c85ba98e78e18"></span>ClassRegistrar&lt;<a class="reference internal" href="#_CPPv2N6paddle12DataProviderE" title="paddle::DataProvider">DataProvider</a>, DataConfig, bool&gt; <code class="descname">registrar_</code><a class="headerlink" href="#_CPPv2N6paddle12DataProvider10registrar_E" title="Permalink to this definition">¶</a></dt>
+<span id="paddle::DataProvider::registrar___ClassRegistrar:DataProvider.DataConfig.ModelConfig.b:"></span><span class="target" id="paddleclasspaddle_1_1DataProvider_1ae40e5169aa51da3fb0e903ff75b8ab01"></span>ClassRegistrar&lt;<a class="reference internal" href="#_CPPv2N6paddle12DataProviderE" title="paddle::DataProvider">DataProvider</a>, DataConfig, <a class="reference internal" href="../../api/api.html#_CPPv211ModelConfig" title="ModelConfig">ModelConfig</a>, bool&gt; <code class="descname">registrar_</code><a class="headerlink" href="#_CPPv2N6paddle12DataProvider10registrar_E" title="Permalink to this definition">¶</a></dt>
 <dd></dd></dl>

 </div>
@@ -357,8 +363,8 @@
 <div class="breathe-sectiondef container">
 <p class="breathe-sectiondef-title rubric">Public Functions</p>
 <dl class="function">
-<dt id="_CPPv2N6paddle17MultiDataProvider17MultiDataProviderERK10DataConfigb">
-<span id="paddle::MultiDataProvider::MultiDataProvider__DataConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1MultiDataProvider_1a9335b9b57d19fccd0b374746b0e28336"></span><code class="descname">MultiDataProvider</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, bool <em>useGpu</em><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle17MultiDataProvider17MultiDataProviderERK10DataConfigb" title="Permalink to this definition">¶</a></dt>
+<dt id="_CPPv2N6paddle17MultiDataProvider17MultiDataProviderERK10DataConfigRK11ModelConfigb">
+<span id="paddle::MultiDataProvider::MultiDataProvider__DataConfigCR.ModelConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1MultiDataProvider_1acd993858e31a0f829e7ca3034bcdb655"></span><code class="descname">MultiDataProvider</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, <em class="property">const</em> <a class="reference internal" href="../../api/api.html#_CPPv211ModelConfig" title="ModelConfig">ModelConfig</a> &amp;<em>modelConfig</em>, bool <em>useGpu</em><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle17MultiDataProvider17MultiDataProviderERK10DataConfigRK11ModelConfigb" title="Permalink to this definition">¶</a></dt>
 <dd></dd></dl>

 <dl class="function">
@@ -977,8 +983,8 @@ virtual <span class="target" id="paddleclasspaddle_1_1SparseValueScanner_1a7ce57
 <div class="breathe-sectiondef container">
 <p class="breathe-sectiondef-title rubric">Public Functions</p>
 <dl class="function">
-<dt id="_CPPv2N6paddle15PyDataProvider215PyDataProvider2ERK10DataConfigb">
-<span id="paddle::PyDataProvider2::PyDataProvider2__DataConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1PyDataProvider2_1a493a2635a8b83aeb9be3ce6db0d3aa55"></span><code class="descname">PyDataProvider2</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, bool <em>useGpu</em><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle15PyDataProvider215PyDataProvider2ERK10DataConfigb" title="Permalink to this definition">¶</a></dt>
+<dt id="_CPPv2N6paddle15PyDataProvider215PyDataProvider2ERK10DataConfigRK11ModelConfigb">
+<span id="paddle::PyDataProvider2::PyDataProvider2__DataConfigCR.ModelConfigCR.b"></span><span class="target" id="paddleclasspaddle_1_1PyDataProvider2_1a2281a7aca7c68247414746864688e7bb"></span><code class="descname">PyDataProvider2</code><span class="sig-paren">(</span><em class="property">const</em> DataConfig &amp;<em>config</em>, <em class="property">const</em> <a class="reference internal" href="../../api/api.html#_CPPv211ModelConfig" title="ModelConfig">ModelConfig</a> &amp;<em>modelConfig</em>, bool <em>useGpu</em><span class="sig-paren">)</span><a class="headerlink" href="#_CPPv2N6paddle15PyDataProvider215PyDataProvider2ERK10DataConfigRK11ModelConfigb" title="Permalink to this definition">¶</a></dt>
 <dd><p>Ctor </p>
 </dd></dl>


--- a/doc/ui/data_provider/pydataprovider2.html
+++ b/doc/ui/data_provider/pydataprovider2.html
--- a/doc_cn/_sources/ui/data_provider/pydataprovider2.txt
+++ b/doc_cn/_sources/ui/data_provider/pydataprovider2.txt
@@ -56,6 +56,14 @@ process函数调用多次 :code:`yield` 即可。 :code:`yield` 是Python的一
 这里说明了训练数据是 'train.list'，而没有测试数据。引用的DataProvider是 'mnist_provider' 
 这个模块中的 'process' 函数。

+同时，根据模型配置文件中 :code:`data_layer` 的名字，用户也可以显式指定返回的数据对应关系。例如:
+
+.. literalinclude:: mnist_provider.dict.py
+   :linenos:
+
+如果用户不指定返回数据的对应关系，那么PaddlePaddle会粗略的根据layer的声明顺序，
+来确定对应关系。这个对应关系可能不正确。所以推荐使用显式指定返回值和数据对应关系。
+
 至此，简单的PyDataProvider样例就说明完毕了。对于用户来说，讲数据发送给PaddlePaddle，仅仅需要
 知道如何从 **一个文件** 里面读取 **一条** 样本。而PaddlePaddle进程帮助用户做了

@@ -119,11 +127,13 @@ DataProvider创建的时候执行。这个初始化函数具有如下参数:
 @provider
 +++++++++

-'@provider'是一个Python的 `Decorator`_ ，他可以将某一个函数标记成一个PyDataProvider。它包含的参数有:
+:code:`@provider` 是一个Python的 `Decorator`_ ，他可以将某一个函数标记成一个PyDataProvider。它包含的参数有:

 *  `input_types`_ 是数据输入格式。具体有哪些格式，参考 `input_types`_ 。
 *  should_shuffle 是个DataProvider是不是要做shuffle，如果不设置的话，训练的时候默认shuffle，
-   测试的时候默认不shuffle
+   测试的时候默认不shuffle。
+*  min_pool_size 是设置DataProvider在内存中最小暂存的数据条数。这个也是PaddlePaddle所能够保证的shuffle粒度。
+   设置成-1的话，会预先读取全部数据到内存中。
 *  pool_size 是设置DataProvider在内存中暂存的数据条数。设置成-1的话，即不在乎内存暂存多少条数据。
 *  can_over_batch_size 表示是否允许Paddle暂存略微多余pool_size的数据。这样做可以避免很多死锁问题。
   一般推荐设置成True
@@ -131,6 +141,11 @@ DataProvider创建的时候执行。这个初始化函数具有如下参数:
   是一个batch size，但是有时为了计算均衡性，可以将一条数据设置成多个batch size
 *  cache 是数据缓存的策略，参考 `cache`_
 *  init_hook 是初始化时调用的函数，参考 `init_hook`_
+*  use_dynamic_order 如果是true的话，可以返回一个dict，key是data_layer的名字，value是特征值。同时，也可以
+   返回一个list或者tuple。如果是false的话，只能够返回list或者tuple
+*  check 设置成true的话，会根据input_types检查数据的合法性。
+*  check_fail_continue 如果设置成true的话，即使在check中数据不合法，也会扔到这条数据，继续训练。 如果
+   check是false的话，没有作用。

 input_types
 +++++++++++
@@ -190,3 +205,55 @@ DataProvider提供了两种简单的Cache策略。他们是
 * CacheType.NO_CACHE 不缓存任何数据，每次都会从python端读取数据
 * CacheType.CACHE_PASS_IN_MEM 第一个pass会从python端读取数据，剩下的pass会直接从内存里
  读取数据。 
+
+
+注意事项
+--------
+
+可能的内存泄露问题
++++++++++++++++++
+
+PaddlePaddle将train.list中的每一行，都传递给process函数，从而生成多个generator。
+即如果train.list中，有100个训练文件，即会生成100个generator。这个本身不是一个很
+严重的问题。
+
+但是，如果在训练时，每一条训练数据都是一个文件，并且，训练数据非常多的情况下，就
+会生成多个generator。每个generator在没有调用的时候，是几乎不占内存的。但是，当调
+用过一次的时候，generator便会存下当前的上下文(Context)。而这个Context可能会非常
+大。并且，generator至少调用两次才会知道是否停止。所以，即使在process里面只会有一
+个yield，也需要两次随机选择到同样的generator的时候，才会释放该段内存。
+
+..  code-block:: python
+
+    def func():
+        yield 0
+
+    f = func()  # 创建generator
+    tmp = next(f)  # 调用一次，返回0
+    tmp = next(f)  # 调用第二次的时候，才会Stop Iteration
+
+而如果按顺序调用这些generator就不会出现这个问题。
+
+所以最佳实践推荐不要将每一个样本都放入train.list。而是将样本的地址放入另一个文本
+文件，train.list写入那个文本文件的地址。 或者在python generator的上下文中尽量留
+下非常少的变量引用。例如
+
+..  code-block:: python
+
+    def real_process(fn):
+        # ... read from fn
+        return result   # 当函数返回的时候，python可以解除掉内部变量的引用。
+
+    def process(fn):
+        yield real_process(fn)
+
+这个问题是PyDataProvider读数据时候的逻辑问题，基本上不能整体修正。
+
+
+内存不够用的情况
++++++++++++++++
+
+PyDataProvider2会尽量使用内存。所以如果对于内存比较小的机器，推荐设置
+:code:`pool_size` 变量，而这个变量推荐大于训练的batch size，并且在内存足够
+的情况下越大越好。
+
--- a/doc_cn/searchindex.js
+++ b/doc_cn/searchindex.js
--- a/doc_cn/ui/data_provider/index.html
+++ b/doc_cn/ui/data_provider/index.html
@@ -61,6 +61,11 @@
 <li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#cache">cache</a></li>
 </ul>
 </li>
+<li class="toctree-l2"><a class="reference internal" href="pydataprovider2.html#id4">注意事项</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#id5">可能的内存泄露问题</a></li>
+<li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#id6">内存不够用的情况</a></li>
+</ul>
+</li>
 </ul>
 </li>
 <li class="toctree-l1"><a class="reference internal" href="write_new_dataprovider.html">自定义一个DataProvider</a></li>

--- a/doc_cn/ui/data_provider/pydataprovider2.html
+++ b/doc_cn/ui/data_provider/pydataprovider2.html
@@ -140,10 +140,66 @@ process函数调用多次 <code class="code docutils literal"><span class="pre">
                        <span class="n">test_list</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
                        <span class="n">module</span><span class="o">=</span><span class="s1">&#39;mnist_provider&#39;</span><span class="p">,</span>
                        <span class="n">obj</span><span class="o">=</span><span class="s1">&#39;process&#39;</span><span class="p">)</span>
+<span class="n">img</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;pixel&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">784</span><span class="p">)</span>
+<span class="n">label</span> <span class="o">=</span> <span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">&#39;label&#39;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
 </pre></div>
 </div>
 <p>这里说明了训练数据是 &#8216;train.list&#8217;，而没有测试数据。引用的DataProvider是 &#8216;mnist_provider&#8217;
 这个模块中的 &#8216;process&#8217; 函数。</p>
+<p>同时，根据模型配置文件中 <code class="code docutils literal"><span class="pre">data_layer</span></code> 的名字，用户也可以显式指定返回的数据对应关系。例如:</p>
+<div class="highlight-default"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre> 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25</pre></div></td><td class="code"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">paddle.trainer.PyDataProvider2</span> <span class="k">import</span> <span class="o">*</span>
+
+
+<span class="c1"># Define a py data provider</span>
+<span class="nd">@provider</span><span class="p">(</span><span class="n">input_types</span><span class="o">=</span><span class="p">[</span>
+    <span class="n">dense_vector</span><span class="p">(</span><span class="mi">28</span> <span class="o">*</span> <span class="mi">28</span><span class="p">),</span>
+    <span class="n">integer_value</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
+<span class="p">])</span>
+<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>  <span class="c1"># settings is not used currently.</span>
+    <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span>  <span class="c1"># open one of training file</span>
+
+    <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>  <span class="c1"># read each line</span>
+        <span class="n">label</span><span class="p">,</span> <span class="n">pixel</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39;;&#39;</span><span class="p">)</span>
+
+        <span class="c1"># get features and label</span>
+        <span class="n">pixels_str</span> <span class="o">=</span> <span class="n">pixel</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">)</span>
+
+        <span class="n">pixels_float</span> <span class="o">=</span> <span class="p">[]</span>
+        <span class="k">for</span> <span class="n">each_pixel_str</span> <span class="ow">in</span> <span class="n">pixels_str</span><span class="p">:</span>
+            <span class="n">pixels_float</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">each_pixel_str</span><span class="p">))</span>
+
+        <span class="c1"># give data to paddle.</span>
+        <span class="k">yield</span> <span class="p">{</span> <span class="s2">&quot;pixel&quot;</span><span class="p">:</span> <span class="n">pixels_float</span><span class="p">,</span> <span class="s1">&#39;label&#39;</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">label</span><span class="p">)</span> <span class="p">}</span>
+
+    <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>  <span class="c1"># close file</span>
+</pre></div>
+</td></tr></table></div>
+<p>如果用户不指定返回数据的对应关系，那么PaddlePaddle会粗略的根据layer的声明顺序，
+来确定对应关系。这个对应关系可能不正确。所以推荐使用显式指定返回值和数据对应关系。</p>
 <p>至此，简单的PyDataProvider样例就说明完毕了。对于用户来说，讲数据发送给PaddlePaddle，仅仅需要
 知道如何从 <strong>一个文件</strong> 里面读取 <strong>一条</strong> 样本。而PaddlePaddle进程帮助用户做了</p>
 <ul class="simple">
@@ -256,11 +312,13 @@ DataProvider创建的时候执行。这个初始化函数具有如下参数:</p>
 <h2>参考(Reference)<a class="headerlink" href="#reference" title="Permalink to this headline">¶</a></h2>
 <div class="section" id="provider">
 <h3>&#64;provider<a class="headerlink" href="#provider" title="Permalink to this headline">¶</a></h3>
-<p><a class="reference external" href="mailto:'&#37;&#52;&#48;provider">'<span>&#64;</span>provider</a>&#8216;是一个Python的 <a class="reference external" href="http://www.learnpython.org/en/Decorators">Decorator</a> ，他可以将某一个函数标记成一个PyDataProvider。它包含的参数有:</p>
+<p><code class="code docutils literal"><span class="pre">&#64;provider</span></code> 是一个Python的 <a class="reference external" href="http://www.learnpython.org/en/Decorators">Decorator</a> ，他可以将某一个函数标记成一个PyDataProvider。它包含的参数有:</p>
 <ul class="simple">
 <li><a class="reference internal" href="#input-types">input_types</a> 是数据输入格式。具体有哪些格式，参考 <a class="reference internal" href="#input-types">input_types</a> 。</li>
 <li>should_shuffle 是个DataProvider是不是要做shuffle，如果不设置的话，训练的时候默认shuffle，
-测试的时候默认不shuffle</li>
+测试的时候默认不shuffle。</li>
+<li>min_pool_size 是设置DataProvider在内存中最小暂存的数据条数。这个也是PaddlePaddle所能够保证的shuffle粒度。
+设置成-1的话，会预先读取全部数据到内存中。</li>
 <li>pool_size 是设置DataProvider在内存中暂存的数据条数。设置成-1的话，即不在乎内存暂存多少条数据。</li>
 <li>can_over_batch_size 表示是否允许Paddle暂存略微多余pool_size的数据。这样做可以避免很多死锁问题。
 一般推荐设置成True</li>
@@ -268,6 +326,11 @@ DataProvider创建的时候执行。这个初始化函数具有如下参数:</p>
 是一个batch size，但是有时为了计算均衡性，可以将一条数据设置成多个batch size</li>
 <li>cache 是数据缓存的策略，参考 <a class="reference internal" href="#cache">cache</a></li>
 <li>init_hook 是初始化时调用的函数，参考 <a class="reference internal" href="#init-hook">init_hook</a></li>
+<li>use_dynamic_order 如果是true的话，可以返回一个dict，key是data_layer的名字，value是特征值。同时，也可以
+返回一个list或者tuple。如果是false的话，只能够返回list或者tuple</li>
+<li>check 设置成true的话，会根据input_types检查数据的合法性。</li>
+<li>check_fail_continue 如果设置成true的话，即使在check中数据不合法，也会扔到这条数据，继续训练。 如果
+check是false的话，没有作用。</li>
 </ul>
 </div>
 <div class="section" id="input-types">
@@ -371,6 +434,47 @@ DataProvider</li>
 </ul>
 </div>
 </div>
+<div class="section" id="id4">
+<h2>注意事项<a class="headerlink" href="#id4" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="id5">
+<h3>可能的内存泄露问题<a class="headerlink" href="#id5" title="Permalink to this headline">¶</a></h3>
+<p>PaddlePaddle将train.list中的每一行，都传递给process函数，从而生成多个generator。
+即如果train.list中，有100个训练文件，即会生成100个generator。这个本身不是一个很
+严重的问题。</p>
+<p>但是，如果在训练时，每一条训练数据都是一个文件，并且，训练数据非常多的情况下，就
+会生成多个generator。每个generator在没有调用的时候，是几乎不占内存的。但是，当调
+用过一次的时候，generator便会存下当前的上下文(Context)。而这个Context可能会非常
+大。并且，generator至少调用两次才会知道是否停止。所以，即使在process里面只会有一
+个yield，也需要两次随机选择到同样的generator的时候，才会释放该段内存。</p>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">func</span><span class="p">():</span>
+    <span class="k">yield</span> <span class="mi">0</span>
+
+<span class="n">f</span> <span class="o">=</span> <span class="n">func</span><span class="p">()</span>  <span class="c1"># 创建generator</span>
+<span class="n">tmp</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>  <span class="c1"># 调用一次，返回0</span>
+<span class="n">tmp</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>  <span class="c1"># 调用第二次的时候，才会Stop Iteration</span>
+</pre></div>
+</div>
+<p>而如果按顺序调用这些generator就不会出现这个问题。</p>
+<p>所以最佳实践推荐不要将每一个样本都放入train.list。而是将样本的地址放入另一个文本
+文件，train.list写入那个文本文件的地址。 或者在python generator的上下文中尽量留
+下非常少的变量引用。例如</p>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">real_process</span><span class="p">(</span><span class="n">fn</span><span class="p">):</span>
+    <span class="c1"># ... read from fn</span>
+    <span class="k">return</span> <span class="n">result</span>   <span class="c1"># 当函数返回的时候，python可以解除掉内部变量的引用。</span>
+
+<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">fn</span><span class="p">):</span>
+    <span class="k">yield</span> <span class="n">real_process</span><span class="p">(</span><span class="n">fn</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>这个问题是PyDataProvider读数据时候的逻辑问题，基本上不能整体修正。</p>
+</div>
+<div class="section" id="id6">
+<h3>内存不够用的情况<a class="headerlink" href="#id6" title="Permalink to this headline">¶</a></h3>
+<p>PyDataProvider2会尽量使用内存。所以如果对于内存比较小的机器，推荐设置
+<code class="code docutils literal"><span class="pre">pool_size</span></code> 变量，而这个变量推荐大于训练的batch size，并且在内存足够
+的情况下越大越好。</p>
+</div>
+</div>
 </div>


@@ -391,6 +495,11 @@ DataProvider</li>
 <li><a class="reference internal" href="#cache">cache</a></li>
 </ul>
 </li>
+<li><a class="reference internal" href="#id4">注意事项</a><ul>
+<li><a class="reference internal" href="#id5">可能的内存泄露问题</a></li>
+<li><a class="reference internal" href="#id6">内存不够用的情况</a></li>
+</ul>
+</li>
 </ul>
 </li>
 </ul>