Skip to content

P
Paddle

Crayon鑫 / Paddle
与 Fork 源项目一致

Fork自 PaddlePaddle / Paddle

通知 1
Star 1
Fork 0
P
Paddle
提交 424055f0 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: 1398854f

上级 c8e3e20e
展开全部 隐藏空白更改
内联 并排
Showing with 99 addition and 96 deletion
develop/doc/api/v2/data/data_reader.html
浏览文件 @ 424055f0
...@@ -730,60 +730,33 @@ And this function contains a buffered decorator. ...@@ -730,60 +730,33 @@ And this function contains a buffered decorator.
:rtype: callable</p> :rtype: callable</p>
</dd></dl> </dd></dl>
<dl class="function"> <dl class="class">
<dt> <dt>
<code class="descclassname">paddle.v2.reader.</code><code class="descname">pipe_reader</code><span class="sig-paren">(</span><em>left_cmd</em>, <em>parser</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em>, <em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt> <em class="property">class </em><code class="descclassname">paddle.v2.reader.</code><code class="descname">PipeReader</code><span class="sig-paren">(</span><em>command</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em><span class="sig-paren">)</span></dt>
<dd><blockquote> <dd><p>PipeReader read data by stream from a command, take it&#8217;s
<div><p>pipe_reader read data by stream from a command, take it&#8217;s
stdout into a pipe buffer and redirect it to the parser to stdout into a pipe buffer and redirect it to the parser to
parse, then yield data as your desired format.</p> parse, then yield data as your desired format.</p>
<p>You can using standard linux command or call another program <p>You can using standard linux command or call another program
to read data, from HDFS, Ceph, URL, AWS S3 etc:</p> to read data, from HDFS, Ceph, URL, AWS S3 etc:</p>
<p>cmd = &#8220;hadoop fs -cat /path/to/some/file&#8221; <p>An example:</p>
cmd = &#8220;cat sample_file.tar.gz&#8221; <div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">example_reader</span><span class="p">():</span>
cmd = &#8220;curl <a class="reference external" href="http://someurl">http://someurl</a>&#8221; <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">myfiles</span><span class="p">:</span>
cmd = &#8220;python print_s3_bucket.py&#8221;</p> <span class="n">pr</span> <span class="o">=</span> <span class="n">PipeReader</span><span class="p">(</span><span class="s2">&quot;cat </span><span class="si">%s</span><span class="s2">&quot;</span><span class="o">%</span><span class="n">f</span><span class="p">)</span>
<p>A sample parser:</p> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">pr</span><span class="o">.</span><span class="n">get_line</span><span class="p">():</span>
<dl class="docutils"> <span class="n">sample</span> <span class="o">=</span> <span class="n">l</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)</span>
<dt>def sample_parser(lines):</dt> <span class="k">yield</span> <span class="n">sample</span>
<dd><p class="first"># parse each line as one sample data, </pre></div>
# return a list of samples as batches. </div>
ret = [] <dl class="method">
for l in lines:</p> <dt>
<blockquote> <code class="descname">get_line</code><span class="sig-paren">(</span><em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt>
<div>ret.append(l.split(&#8221; &#8221;)[1:5])</div></blockquote> <dd><blockquote>
<p class="last">return ret</p> <div><table class="docutils field-list" frame="void" rules="none">
</dd>
</dl>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" /> <col class="field-name" />
<col class="field-body" /> <col class="field-body" />
<tbody valign="top"> <tbody valign="top">
<tr class="field-odd field"><th class="field-name">param left_cmd:</th><td class="field-body">command to excute to get stdout from.</td>
</tr>
<tr class="field-even field"><th class="field-name">type left_cmd:</th><td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param parser:</th><td class="field-body">parser function to parse lines of data.
if cut_lines is True, parser will receive list
of lines.
if cut_lines is False, parser will receive a
raw buffer each time.
parser should return a list of parsed values.</td>
</tr>
<tr class="field-even field"><th class="field-name">type parser:</th><td class="field-body">callable</td>
</tr>
<tr class="field-odd field"><th class="field-name">param bufsize:</th><td class="field-body">the buffer size used for the stdout pipe.</td>
</tr>
<tr class="field-even field"><th class="field-name">type bufsize:</th><td class="field-body">int</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param file_type:</th></tr>
<tr class="field-odd field"><td>&#160;</td><td class="field-body">can be plain/gzip, stream buffer data type.</td>
</tr>
<tr class="field-even field"><th class="field-name">type file_type:</th><td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param cut_lines:</th></tr> <tr class="field-odd field"><th class="field-name" colspan="2">param cut_lines:</th></tr>
<tr class="field-odd field"><td>&#160;</td><td class="field-body">whether to pass lines instead of raw buffer <tr class="field-odd field"><td>&#160;</td><td class="field-body">cut buffer to lines</td>
to the parser</td>
</tr> </tr>
<tr class="field-even field"><th class="field-name">type cut_lines:</th><td class="field-body">bool</td> <tr class="field-even field"><th class="field-name">type cut_lines:</th><td class="field-body">bool</td>
</tr> </tr>
...@@ -802,9 +775,9 @@ to the parser</td> ...@@ -802,9 +775,9 @@ to the parser</td>
<tr class="field-odd field"><th class="field-name" colspan="2">type line_break:</th></tr> <tr class="field-odd field"><th class="field-name" colspan="2">type line_break:</th></tr>
<tr class="field-odd field"><td>&#160;</td><td class="field-body">string</td> <tr class="field-odd field"><td>&#160;</td><td class="field-body">string</td>
</tr> </tr>
<tr class="field-even field"><th class="field-name">return:</th><td class="field-body">the reader generator.</td> <tr class="field-even field"><th class="field-name">return:</th><td class="field-body">one line or a buffer of bytes</td>
</tr> </tr>
<tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">callable</td> <tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">string</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
...@@ -812,6 +785,8 @@ to the parser</td> ...@@ -812,6 +785,8 @@ to the parser</td>
</dl> </dl>
</dd></dl> </dd></dl>
</dd></dl>
</div> </div>
<p>Creator package contains some simple reader creator, which could <p>Creator package contains some simple reader creator, which could
be used in user program.</p> be used in user program.</p>
......
develop/doc/searchindex.js
浏览文件 @ 424055f0
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc_cn/_sources/faq/build_and_install/index_cn.rst.txt
浏览文件 @ 424055f0
...@@ -109,3 +109,31 @@ PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二 ...@@ -109,3 +109,31 @@ PaddlePaddle使用avx SIMD指令提高cpu执行效率，因此错误的使用二
解决办法是： 解决办法是：
* 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。 * 卸载PaddlePaddle包 :code:`pip uninstall paddle`, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 :code:`/python` 目录下的python包。同时，即便设置 :code:`PYTHONPATH` 到 :code:`/python` 也没用，因为python的搜索路径是优先已经安装的python包。
8. 下载MKLML库失败
------------------
.. code-block:: bash
make[2]: *** [third_party/mklml/src/extern_mklml-stamp/extern_mklml-download] 错误 4
make[1]: *** [CMakeFiles/extern_mklml.dir/all] 错误 2
make[1]: *** 正在等待未完成的任务....
原因：网速或SSL链接原因，导致MKLML库下载不成功。
解决办法是：手动下载并安装，具体步骤如下。
.. code-block:: bash
// 1. 进入对应的目录
cd build/third_party/mklml/src/extern_mklml
// 2. 查看包的大小， 正常情况下是75M，如果小于75M，即下载失败：
du -sh mklml_lnx_2018.0.1.20171007.tgz
// 3. 手动下载且解压缩，并手动生成download成功标签：
wget --no-check-certificate https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz -c -O mklml_lnx_2018.0.1.20171007.tgz
tar zxf mklml_lnx_2018.0.1.20171007.tgz
touch ../extern_mklml-stamp/extern_mklml-download
// 4. 接着编译即可
develop/doc_cn/api/v2/data/data_reader.html
浏览文件 @ 424055f0
...@@ -743,60 +743,33 @@ And this function contains a buffered decorator. ...@@ -743,60 +743,33 @@ And this function contains a buffered decorator.
:rtype: callable</p> :rtype: callable</p>
</dd></dl> </dd></dl>
<dl class="function"> <dl class="class">
<dt> <dt>
<code class="descclassname">paddle.v2.reader.</code><code class="descname">pipe_reader</code><span class="sig-paren">(</span><em>left_cmd</em>, <em>parser</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em>, <em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt> <em class="property">class </em><code class="descclassname">paddle.v2.reader.</code><code class="descname">PipeReader</code><span class="sig-paren">(</span><em>command</em>, <em>bufsize=8192</em>, <em>file_type='plain'</em><span class="sig-paren">)</span></dt>
<dd><blockquote> <dd><p>PipeReader read data by stream from a command, take it&#8217;s
<div><p>pipe_reader read data by stream from a command, take it&#8217;s
stdout into a pipe buffer and redirect it to the parser to stdout into a pipe buffer and redirect it to the parser to
parse, then yield data as your desired format.</p> parse, then yield data as your desired format.</p>
<p>You can using standard linux command or call another program <p>You can using standard linux command or call another program
to read data, from HDFS, Ceph, URL, AWS S3 etc:</p> to read data, from HDFS, Ceph, URL, AWS S3 etc:</p>
<p>cmd = &#8220;hadoop fs -cat /path/to/some/file&#8221; <p>An example:</p>
cmd = &#8220;cat sample_file.tar.gz&#8221; <div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">example_reader</span><span class="p">():</span>
cmd = &#8220;curl <a class="reference external" href="http://someurl">http://someurl</a>&#8221; <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">myfiles</span><span class="p">:</span>
cmd = &#8220;python print_s3_bucket.py&#8221;</p> <span class="n">pr</span> <span class="o">=</span> <span class="n">PipeReader</span><span class="p">(</span><span class="s2">&quot;cat </span><span class="si">%s</span><span class="s2">&quot;</span><span class="o">%</span><span class="n">f</span><span class="p">)</span>
<p>A sample parser:</p> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">pr</span><span class="o">.</span><span class="n">get_line</span><span class="p">():</span>
<dl class="docutils"> <span class="n">sample</span> <span class="o">=</span> <span class="n">l</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)</span>
<dt>def sample_parser(lines):</dt> <span class="k">yield</span> <span class="n">sample</span>
<dd><p class="first"># parse each line as one sample data, </pre></div>
# return a list of samples as batches. </div>
ret = [] <dl class="method">
for l in lines:</p> <dt>
<blockquote> <code class="descname">get_line</code><span class="sig-paren">(</span><em>cut_lines=True</em>, <em>line_break='\n'</em><span class="sig-paren">)</span></dt>
<div>ret.append(l.split(&#8221; &#8221;)[1:5])</div></blockquote> <dd><blockquote>
<p class="last">return ret</p> <div><table class="docutils field-list" frame="void" rules="none">
</dd>
</dl>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" /> <col class="field-name" />
<col class="field-body" /> <col class="field-body" />
<tbody valign="top"> <tbody valign="top">
<tr class="field-odd field"><th class="field-name">param left_cmd:</th><td class="field-body">command to excute to get stdout from.</td>
</tr>
<tr class="field-even field"><th class="field-name">type left_cmd:</th><td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name">param parser:</th><td class="field-body">parser function to parse lines of data.
if cut_lines is True, parser will receive list
of lines.
if cut_lines is False, parser will receive a
raw buffer each time.
parser should return a list of parsed values.</td>
</tr>
<tr class="field-even field"><th class="field-name">type parser:</th><td class="field-body">callable</td>
</tr>
<tr class="field-odd field"><th class="field-name">param bufsize:</th><td class="field-body">the buffer size used for the stdout pipe.</td>
</tr>
<tr class="field-even field"><th class="field-name">type bufsize:</th><td class="field-body">int</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param file_type:</th></tr>
<tr class="field-odd field"><td>&#160;</td><td class="field-body">can be plain/gzip, stream buffer data type.</td>
</tr>
<tr class="field-even field"><th class="field-name">type file_type:</th><td class="field-body">string</td>
</tr>
<tr class="field-odd field"><th class="field-name" colspan="2">param cut_lines:</th></tr> <tr class="field-odd field"><th class="field-name" colspan="2">param cut_lines:</th></tr>
<tr class="field-odd field"><td>&#160;</td><td class="field-body">whether to pass lines instead of raw buffer <tr class="field-odd field"><td>&#160;</td><td class="field-body">cut buffer to lines</td>
to the parser</td>
</tr> </tr>
<tr class="field-even field"><th class="field-name">type cut_lines:</th><td class="field-body">bool</td> <tr class="field-even field"><th class="field-name">type cut_lines:</th><td class="field-body">bool</td>
</tr> </tr>
...@@ -815,9 +788,9 @@ to the parser</td> ...@@ -815,9 +788,9 @@ to the parser</td>
<tr class="field-odd field"><th class="field-name" colspan="2">type line_break:</th></tr> <tr class="field-odd field"><th class="field-name" colspan="2">type line_break:</th></tr>
<tr class="field-odd field"><td>&#160;</td><td class="field-body">string</td> <tr class="field-odd field"><td>&#160;</td><td class="field-body">string</td>
</tr> </tr>
<tr class="field-even field"><th class="field-name">return:</th><td class="field-body">the reader generator.</td> <tr class="field-even field"><th class="field-name">return:</th><td class="field-body">one line or a buffer of bytes</td>
</tr> </tr>
<tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">callable</td> <tr class="field-odd field"><th class="field-name">rtype:</th><td class="field-body">string</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
...@@ -825,6 +798,8 @@ to the parser</td> ...@@ -825,6 +798,8 @@ to the parser</td>
</dl> </dl>
</dd></dl> </dd></dl>
</dd></dl>
</div> </div>
<p>Creator package contains some simple reader creator, which could <p>Creator package contains some simple reader creator, which could
be used in user program.</p> be used in user program.</p>
......
develop/doc_cn/faq/build_and_install/index_cn.html
浏览文件 @ 424055f0
...@@ -239,6 +239,7 @@ ...@@ -239,6 +239,7 @@
<li><a class="reference internal" href="#import-paddle-v2-as-paddle-importerror-no-module-named-v2" id="id7">5. 编译安装后执行 import paddle.v2 as paddle 报ImportError: No module named v2</a></li> <li><a class="reference internal" href="#import-paddle-v2-as-paddle-importerror-no-module-named-v2" id="id7">5. 编译安装后执行 import paddle.v2 as paddle 报ImportError: No module named v2</a></li>
<li><a class="reference internal" href="#illegal-instruction" id="id8">6. 遇到“非法指令”或者是“illegal instruction”</a></li> <li><a class="reference internal" href="#illegal-instruction" id="id8">6. 遇到“非法指令”或者是“illegal instruction”</a></li>
<li><a class="reference internal" href="#python" id="id9">7. python相关的单元测试都过不了</a></li> <li><a class="reference internal" href="#python" id="id9">7. python相关的单元测试都过不了</a></li>
<li><a class="reference internal" href="#mklml" id="id10">8. 下载MKLML库失败</a></li>
</ul> </ul>
</li> </li>
</ul> </ul>
...@@ -328,6 +329,30 @@ Please uninstall paddle package before start unittest. Try to <span class="s1">& ...@@ -328,6 +329,30 @@ Please uninstall paddle package before start unittest. Try to <span class="s1">&
<li>卸载PaddlePaddle包 <code class="code docutils literal"><span class="pre">pip</span> <span class="pre">uninstall</span> <span class="pre">paddle</span></code>, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 <code class="code docutils literal"><span class="pre">/python</span></code> 目录下的python包。同时，即便设置 <code class="code docutils literal"><span class="pre">PYTHONPATH</span></code> 到 <code class="code docutils literal"><span class="pre">/python</span></code> 也没用，因为python的搜索路径是优先已经安装的python包。</li> <li>卸载PaddlePaddle包 <code class="code docutils literal"><span class="pre">pip</span> <span class="pre">uninstall</span> <span class="pre">paddle</span></code>, 清理掉老旧的PaddlePaddle安装包，使得单元测试有一个干净的环境。如果PaddlePaddle包已经在python的site-packages里面，单元测试会引用site-packages里面的python包，而不是源码目录里 <code class="code docutils literal"><span class="pre">/python</span></code> 目录下的python包。同时，即便设置 <code class="code docutils literal"><span class="pre">PYTHONPATH</span></code> 到 <code class="code docutils literal"><span class="pre">/python</span></code> 也没用，因为python的搜索路径是优先已经安装的python包。</li>
</ul> </ul>
</div> </div>
<div class="section" id="mklml">
<h2><a class="toc-backref" href="#id10">8. 下载MKLML库失败</a><a class="headerlink" href="#mklml" title="永久链接至标题"></a></h2>
<div class="highlight-bash"><div class="highlight"><pre><span></span>make<span class="o">[</span><span class="m">2</span><span class="o">]</span>: *** <span class="o">[</span>third_party/mklml/src/extern_mklml-stamp/extern_mklml-download<span class="o">]</span> 错误 <span class="m">4</span>
make<span class="o">[</span><span class="m">1</span><span class="o">]</span>: *** <span class="o">[</span>CMakeFiles/extern_mklml.dir/all<span class="o">]</span> 错误 <span class="m">2</span>
make<span class="o">[</span><span class="m">1</span><span class="o">]</span>: *** 正在等待未完成的任务....
</pre></div>
</div>
<p>原因：网速或SSL链接原因，导致MKLML库下载不成功。</p>
<p>解决办法是：手动下载并安装，具体步骤如下。</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>// <span class="m">1</span>. 进入对应的目录
<span class="nb">cd</span> build/third_party/mklml/src/extern_mklml
// <span class="m">2</span>. 查看包的大小， 正常情况下是75M，如果小于75M，即下载失败：
du -sh mklml_lnx_2018.0.1.20171007.tgz
// <span class="m">3</span>. 手动下载且解压缩，并手动生成download成功标签：
wget --no-check-certificate https://github.com/01org/mkl-dnn/releases/download/v0.11/mklml_lnx_2018.0.1.20171007.tgz -c -O mklml_lnx_2018.0.1.20171007.tgz
tar zxf mklml_lnx_2018.0.1.20171007.tgz
touch ../extern_mklml-stamp/extern_mklml-download
// <span class="m">4</span>. 接着编译即可
</pre></div>
</div>
</div>
</div> </div>
......
develop/doc_cn/searchindex.js
浏览文件 @ 424055f0
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或