Skip to content

P
Paddle

BaiXuePrincess / Paddle
与 Fork 源项目一致

Fork自 PaddlePaddle / Paddle

通知 1
Star 1
Fork 0
P
Paddle
切换分支/标签
查找文件 普通视图历史永久链接Permalink
index.html 6.1 KB
Newer Older
Y
Update docs  
Yu Yang 已提交
1 2 3 4 5 6 7 8 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">


<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
9 
    <title>PaddlePaddle的数据提供(DataProvider)介绍 &#8212; PaddlePaddle  documentation</title>
Y
Update docs  
Yu Yang 已提交
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 
    
    <link rel="stylesheet" href="../../_static/classic.css" type="text/css" />
    <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../../',
        VERSION:     '',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="../../_static/jquery.js"></script>
    <script type="text/javascript" src="../../_static/underscore.js"></script>
    <script type="text/javascript" src="../../_static/doctools.js"></script>
    <script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
27 
    <link rel="top" title="PaddlePaddle  documentation" href="../../index.html" />
Y
Update docs  
Yu Yang 已提交
28 29 30 31 32 33 34 35 
  </head>
  <body role="document">
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../../genindex.html" title="General Index"
             accesskey="I">index</a></li>
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
36 
        <li class="nav-item nav-item-0"><a href="../../index.html">PaddlePaddle  documentation</a> &#187;</li>
Y
Update docs  
Yu Yang 已提交
37 38 39 40 41 42 43 44 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
Y
Upload docs  
Yu Yang 已提交
45 46 
  <div class="section" id="paddlepaddle-dataprovider">
<h1>PaddlePaddle的数据提供(DataProvider)介绍<a class="headerlink" href="#paddlepaddle-dataprovider" title="Permalink to this headline"></a></h1>
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
47 48 49 50 51 
<p>数据提供(DataProvider)是PaddlePaddle负责提供数据的模块。其作用是将训练数据传入内存或者显存，让神经网络可以进行训练。简单的使用，用户可以使用Python的 <code class="code docutils literal"><span class="pre">PyDataProvider</span></code> 来自定义传数据的过程。如果有更复杂的使用，或者需要更高的效率，用户也可以在C++端自定义一个 <code class="code docutils literal"><span class="pre">DataProvider</span></code> 。</p>
<p>PaddlePaddle需要用户在网络配置(trainer_config.py)中定义使用哪种DataProvider及其参数，训练文件列表(train.list)和测试文件列表(test.list)。</p>
<p>其中，train.list和test.list均为本地的两个文件(推荐直接放置到训练目录，以相对路径引用)。如果test.list不设置，或者设置为None，那么在训练过程中，不会执行测试操作。否则，会根据命令行参数指定的测试方式，在训练过程中进行测试，从而防止过拟合。</p>
<p>一般情况下，train.list和test.list为纯文本文件，一行对应一个数据文件，数据文件存放在本地磁盘中。将文件的绝对路径或相对路径(相对于PaddlePaddle程序运行时的路径)写在train.list和test.list中。当然，train.list和test.list也可以放置hdfs文件路径，或者数据库连接地址等等。</p>
<p>用户在DataProvider中需要实现如何访问其中每一个文件。DataProvider的具体用法和如何实现一个新的DataProvider，请参考下述文章:</p>
Y
Update docs  
Yu Yang 已提交
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="pydataprovider2.html">PyDataProvider2的使用</a><ul>
<li class="toctree-l2"><a class="reference internal" href="pydataprovider2.html#id1">简单的使用场景</a></li>
<li class="toctree-l2"><a class="reference internal" href="pydataprovider2.html#id3">序列模型数据提供</a></li>
<li class="toctree-l2"><a class="reference internal" href="pydataprovider2.html#reference">参考(Reference)</a><ul>
<li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#provider">&#64;provider</a></li>
<li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#input-types">input_types</a></li>
<li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#init-hook">init_hook</a></li>
<li class="toctree-l3"><a class="reference internal" href="pydataprovider2.html#cache">cache</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="write_new_dataprovider.html">自定义一个DataProvider</a></li>
</ul>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="../../_sources/ui/data_provider/index.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="../../search.html" method="get">
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
87 88 
      <div><input type="text" name="q" /></div>
      <div><input type="submit" value="Go" /></div>
Y
Update docs  
Yu Yang 已提交
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../../genindex.html" title="General Index"
             >index</a></li>
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
104 
        <li class="nav-item nav-item-0"><a href="../../index.html">PaddlePaddle  documentation</a> &#187;</li>
Y
Update docs  
Yu Yang 已提交
105 106 107 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
T
Deploy to GitHub Pages: 703cce35a871c78a5e3f1c33abf7fe788e2bbda5  
Travis CI 已提交
108 109 
        &#169; Copyright 2016, PaddlePaddle developers.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.4.6.
Y
Update docs  
Yu Yang 已提交
110 111 112 
    </div>
  </body>
</html>