Deploy to GitHub Pages: 1398854f

424055f0 · Travis CI · c8e3e20e · 424055f0 · 424055f0 · 424055f0
6 changed file
--- a/develop/doc/api/v2/data/data_reader.html
+++ b/develop/doc/api/v2/data/data_reader.html
@@ -730,60 +730,33 @@ And this function contains a buffered decorator.
 :rtype: callable</p>
 </dd></dl>

-<dl class="function">
+<dl class="class">
 <dt>
-<code class="descclassname">paddle.v2.reader.</code><code class="descname">pipe_reader</code><span class="sig-paren">(</span><em>left_cmd</em>, <em>parser</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em>, <em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt>
-<dd><blockquote>
-<div><p>pipe_reader read data by stream from a command, take it&#8217;s
+<em class="property">class </em><code class="descclassname">paddle.v2.reader.</code><code class="descname">PipeReader</code><span class="sig-paren">(</span><em>command</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em><span class="sig-paren">)</span></dt>
+<dd><p>PipeReader read data by stream from a command, take it&#8217;s
 stdout into a pipe buffer and redirect it to the parser to
 parse, then yield data as your desired format.</p>
 <p>You can using standard linux command or call another program
 to read data, from HDFS, Ceph, URL, AWS S3 etc:</p>
-<p>cmd = &#8220;hadoop fs -cat /path/to/some/file&#8221;
-cmd = &#8220;cat sample_file.tar.gz&#8221;
-cmd = &#8220;curl <a class="reference external" href="http://someurl">http://someurl</a>&#8221;
-cmd = &#8220;python print_s3_bucket.py&#8221;</p>
-<p>A sample parser:</p>
-<dl class="docutils">
-<dt>def sample_parser(lines):</dt>
-<dd><p class="first"># parse each line as one sample data,
-# return a list of samples as batches.
-ret = []
-for l in lines:</p>
-<blockquote>
-<div>ret.append(l.split(&#8221; &#8221;)[1:5])</div></blockquote>
-<p class="last">return ret</p>
-</dd>
-</dl>
-<table class="docutils field-list" frame="void" rules="none">
+<p>An example:</p>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">example_reader</span><span class="p">():</span>
+    <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">myfiles</span><span class="p">:</span>
+        <span class="n">pr</span> <span class="o">=</span> <span class="n">PipeReader</span><span class="p">(</span><span class="s2">&quot;cat </span><span class="si">%s</span><span class="s2">&quot;</span><span class="o">%</span><span class="n">f</span><span class="p">)</span>
+        <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">pr</span><span class="o">.</span><span class="n">get_line</span><span class="p">():</span>
+            <span class="n">sample</span> <span class="o">=</span> <span class="n">l</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)</span>
+            <span class="k">yield</span> <span class="n">sample</span>
+</pre></div>
+</div>
+<dl class="method">
+<dt>
+<code class="descname">get_line</code><span class="sig-paren">(</span><em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt>
+<dd><blockquote>
+<div><table class="docutils field-list" frame="void" rules="none">
 <col class="field-name" />
 <col class="field-body" />
 <tbody valign="top">
-<tr class="field-odd field"><th class="field-name">param left_cmd:</th><td class="field-body">command to excute to get stdout from.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type left_cmd:</th><td class="field-body">string</td>
-</tr>
-<tr class="field-odd field"><th class="field-name">param parser:</th><td class="field-body">parser function to parse lines of data.
-if cut_lines is True, parser will receive list
-of lines.
-if cut_lines is False, parser will receive a
-raw buffer each time.
-parser should return a list of parsed values.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type parser:</th><td class="field-body">callable</td>
-</tr>
-<tr class="field-odd field"><th class="field-name">param bufsize:</th><td class="field-body">the buffer size used for the stdout pipe.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type bufsize:</th><td class="field-body">int</td>
-</tr>
-<tr class="field-odd field"><th class="field-name" colspan="2">param file_type:</th></tr>
-<tr class="field-odd field"><td>&#160;</td><td class="field-body">can be plain/gzip, stream buffer data type.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type file_type:</th><td class="field-body">string</td>
-</tr>
 <tr class="field-odd field"><th class="field-name" colspan="2">param cut_lines:</th></tr>
-<tr class="field-odd field"><td>&#160;</td><td class="field-body">whether to pass lines instead of raw buffer
-to the parser</td>
+<tr class="field-odd field"><td>&#160;</td><td class="field-body">cut buffer to lines</td>
 </tr>
 <tr class="field-even field"><th class="field-name">type cut_lines:</th><td class="field-body">bool</td>
 </tr>
@@ -802,9 +775,9 @@ to the parser</td>
 <tr class="field-odd field"><th class="field-name" colspan="2">type line_break:</th></tr>
 <tr class="field-odd field"><td>&#160;</td><td class="field-body">string</td>
 </tr>
-<tr class="field-even field"><th class="field-name">return:</th><td class="field-body">the reader generator.</td>
+<tr class="field-even field"><th class="field-name">return:</th><td class="field-body">one line or a buffer of bytes</td>
 </tr>
-<tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">callable</td>
+<tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">string</td>
 </tr>
 </tbody>
 </table>
@@ -812,6 +785,8 @@ to the parser</td>
 </dl>
 </dd></dl>

+</dd></dl>
+
 </div>
 <p>Creator package contains some simple reader creator, which could
 be used in user program.</p>

--- a/develop/doc/searchindex.js
+++ b/develop/doc/searchindex.js
--- a/develop/doc_cn/_sources/faq/build_and_install/index_cn.rst.txt
+++ b/develop/doc_cn/_sources/faq/build_and_install/index_cn.rst.txt
@@ -109,3 +109,31 @@ PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二
 解决办法是：

 * 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。
+
+8. 下载MKLML库失败
+------------------
+
+..  code-block:: bash
+
+    make[2]: *** [third_party/mklml/src/extern_mklml-stamp/extern_mklml-download] 错误 4
+    make[1]: *** [CMakeFiles/extern_mklml.dir/all] 错误 2
+    make[1]: *** 正在等待未完成的任务....
+
+原因：网速或SSL链接原因，导致MKLML库下载不成功。
+
+解决办法是：手动下载并安装，具体步骤如下。
+
+..  code-block:: bash
+
+    // 1. 进入对应的目录
+    cd build/third_party/mklml/src/extern_mklml
+
+    // 2. 查看包的大小， 正常情况下是75M，如果小于75M，即下载失败：
+    du -sh mklml_lnx_2018.0.1.20171007.tgz
+
+    // 3. 手动下载且解压缩，并手动生成download成功标签：
+    wget --no-check-certificate https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz -c -O mklml_lnx_2018.0.1.20171007.tgz 
+    tar zxf mklml_lnx_2018.0.1.20171007.tgz
+    touch ../extern_mklml-stamp/extern_mklml-download
+
+    // 4. 接着编译即可
--- a/develop/doc_cn/api/v2/data/data_reader.html
+++ b/develop/doc_cn/api/v2/data/data_reader.html
@@ -743,60 +743,33 @@ And this function contains a buffered decorator.
 :rtype: callable</p>
 </dd></dl>

-<dl class="function">
+<dl class="class">
 <dt>
-<code class="descclassname">paddle.v2.reader.</code><code class="descname">pipe_reader</code><span class="sig-paren">(</span><em>left_cmd</em>, <em>parser</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em>, <em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt>
-<dd><blockquote>
-<div><p>pipe_reader read data by stream from a command, take it&#8217;s
+<em class="property">class </em><code class="descclassname">paddle.v2.reader.</code><code class="descname">PipeReader</code><span class="sig-paren">(</span><em>command</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em><span class="sig-paren">)</span></dt>
+<dd><p>PipeReader read data by stream from a command, take it&#8217;s
 stdout into a pipe buffer and redirect it to the parser to
 parse, then yield data as your desired format.</p>
 <p>You can using standard linux command or call another program
 to read data, from HDFS, Ceph, URL, AWS S3 etc:</p>
-<p>cmd = &#8220;hadoop fs -cat /path/to/some/file&#8221;
-cmd = &#8220;cat sample_file.tar.gz&#8221;
-cmd = &#8220;curl <a class="reference external" href="http://someurl">http://someurl</a>&#8221;
-cmd = &#8220;python print_s3_bucket.py&#8221;</p>
-<p>A sample parser:</p>
-<dl class="docutils">
-<dt>def sample_parser(lines):</dt>
-<dd><p class="first"># parse each line as one sample data,
-# return a list of samples as batches.
-ret = []
-for l in lines:</p>
-<blockquote>
-<div>ret.append(l.split(&#8221; &#8221;)[1:5])</div></blockquote>
-<p class="last">return ret</p>
-</dd>
-</dl>
-<table class="docutils field-list" frame="void" rules="none">
+<p>An example:</p>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">example_reader</span><span class="p">():</span>
+    <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">myfiles</span><span class="p">:</span>
+        <span class="n">pr</span> <span class="o">=</span> <span class="n">PipeReader</span><span class="p">(</span><span class="s2">&quot;cat </span><span class="si">%s</span><span class="s2">&quot;</span><span class="o">%</span><span class="n">f</span><span class="p">)</span>
+        <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">pr</span><span class="o">.</span><span class="n">get_line</span><span class="p">():</span>
+            <span class="n">sample</span> <span class="o">=</span> <span class="n">l</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)</span>
+            <span class="k">yield</span> <span class="n">sample</span>
+</pre></div>
+</div>
+<dl class="method">
+<dt>
+<code class="descname">get_line</code><span class="sig-paren">(</span><em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt>
+<dd><blockquote>
+<div><table class="docutils field-list" frame="void" rules="none">
 <col class="field-name" />
 <col class="field-body" />
 <tbody valign="top">
-<tr class="field-odd field"><th class="field-name">param left_cmd:</th><td class="field-body">command to excute to get stdout from.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type left_cmd:</th><td class="field-body">string</td>
-</tr>
-<tr class="field-odd field"><th class="field-name">param parser:</th><td class="field-body">parser function to parse lines of data.
-if cut_lines is True, parser will receive list
-of lines.
-if cut_lines is False, parser will receive a
-raw buffer each time.
-parser should return a list of parsed values.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type parser:</th><td class="field-body">callable</td>
-</tr>
-<tr class="field-odd field"><th class="field-name">param bufsize:</th><td class="field-body">the buffer size used for the stdout pipe.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type bufsize:</th><td class="field-body">int</td>
-</tr>
-<tr class="field-odd field"><th class="field-name" colspan="2">param file_type:</th></tr>
-<tr class="field-odd field"><td>&#160;</td><td class="field-body">can be plain/gzip, stream buffer data type.</td>
-</tr>
-<tr class="field-even field"><th class="field-name">type file_type:</th><td class="field-body">string</td>
-</tr>
 <tr class="field-odd field"><th class="field-name" colspan="2">param cut_lines:</th></tr>
-<tr class="field-odd field"><td>&#160;</td><td class="field-body">whether to pass lines instead of raw buffer
-to the parser</td>
+<tr class="field-odd field"><td>&#160;</td><td class="field-body">cut buffer to lines</td>
 </tr>
 <tr class="field-even field"><th class="field-name">type cut_lines:</th><td class="field-body">bool</td>
 </tr>
@@ -815,9 +788,9 @@ to the parser</td>
 <tr class="field-odd field"><th class="field-name" colspan="2">type line_break:</th></tr>
 <tr class="field-odd field"><td>&#160;</td><td class="field-body">string</td>
 </tr>
-<tr class="field-even field"><th class="field-name">return:</th><td class="field-body">the reader generator.</td>
+<tr class="field-even field"><th class="field-name">return:</th><td class="field-body">one line or a buffer of bytes</td>
 </tr>
-<tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">callable</td>
+<tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">string</td>
 </tr>
 </tbody>
 </table>
@@ -825,6 +798,8 @@ to the parser</td>
 </dl>
 </dd></dl>

+</dd></dl>
+
 </div>
 <p>Creator package contains some simple reader creator, which could
 be used in user program.</p>

--- a/develop/doc_cn/faq/build_and_install/index_cn.html
+++ b/develop/doc_cn/faq/build_and_install/index_cn.html
@@ -239,6 +239,7 @@
 <li><a class="reference internal" href="#import-paddle-v2-as-paddle-importerror-no-module-named-v2" id="id7">5. 编译安装后执行 import paddle.v2 as paddle 报ImportError: No module named v2</a></li>
 <li><a class="reference internal" href="#illegal-instruction" id="id8">6. 遇到“非法指令”或者是“illegal instruction”</a></li>
 <li><a class="reference internal" href="#python" id="id9">7.  python相关的单元测试都过不了</a></li>
+<li><a class="reference internal" href="#mklml" id="id10">8. 下载MKLML库失败</a></li>
 </ul>
 </li>
 </ul>
@@ -328,6 +329,30 @@ Please uninstall paddle package before start unittest. Try to <span class="s1">&
 <li>卸载PaddlePaddle包 <code class="code docutils literal"><span class="pre">pip</span> <span class="pre">uninstall</span> <span class="pre">paddle</span></code>, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 <code class="code docutils literal"><span class="pre">/python</span></code> 目录下的python包。同时，即便设置 <code class="code docutils literal"><span class="pre">PYTHONPATH</span></code> 到 <code class="code docutils literal"><span class="pre">/python</span></code> 也没用，因为python的搜索路径是优先已经安装的python包。</li>
 </ul>
 </div>
+<div class="section" id="mklml">
+<h2><a class="toc-backref" href="#id10">8. 下载MKLML库失败</a><a class="headerlink" href="#mklml" title="永久链接至标题">¶</a></h2>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>make<span class="o">[</span><span class="m">2</span><span class="o">]</span>: *** <span class="o">[</span>third_party/mklml/src/extern_mklml-stamp/extern_mklml-download<span class="o">]</span> 错误 <span class="m">4</span>
+make<span class="o">[</span><span class="m">1</span><span class="o">]</span>: *** <span class="o">[</span>CMakeFiles/extern_mklml.dir/all<span class="o">]</span> 错误 <span class="m">2</span>
+make<span class="o">[</span><span class="m">1</span><span class="o">]</span>: *** 正在等待未完成的任务....
+</pre></div>
+</div>
+<p>原因：网速或SSL链接原因，导致MKLML库下载不成功。</p>
+<p>解决办法是：手动下载并安装，具体步骤如下。</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>// <span class="m">1</span>. 进入对应的目录
+<span class="nb">cd</span> build/third_party/mklml/src/extern_mklml
+
+// <span class="m">2</span>. 查看包的大小， 正常情况下是75M，如果小于75M，即下载失败：
+du -sh mklml_lnx_2018.0.1.20171007.tgz
+
+// <span class="m">3</span>. 手动下载且解压缩，并手动生成download成功标签：
+wget --no-check-certificate https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz -c -O mklml_lnx_2018.0.1.20171007.tgz
+tar zxf mklml_lnx_2018.0.1.20171007.tgz
+touch ../extern_mklml-stamp/extern_mklml-download
+
+// <span class="m">4</span>. 接着编译即可
+</pre></div>
+</div>
+</div>
 </div>



--- a/develop/doc_cn/searchindex.js
+++ b/develop/doc_cn/searchindex.js