Skip to content

P
Paddle

PaddlePaddle / Paddle
大约 1 年 前同步成功

通知 2298
Star 20931
Fork 5422
P
Paddle
切换分支/标签
查找文件 普通视图历史永久链接Permalink
refactorization.html 35.6 KB
Newer Older
T
Deploy to GitHub Pages: 1ff47b284c565d030b198705d5f18b4bd4ce53e5  
Travis CI 已提交
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 


<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Design Doc: Refactorization Overview &mdash; PaddlePaddle  文档</title>
  

  
  

  

  
  
    

  

  
  
    <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  

  
  
        <link rel="index" title="索引"
              href="../genindex.html"/>
        <link rel="search" title="搜索" href="../search.html"/>
    <link rel="top" title="PaddlePaddle  文档" href="../index.html"/> 

  <link rel="stylesheet" href="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/css/perfect-scrollbar.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/override.css" type="text/css" />
  <script>
  var _hmt = _hmt || [];
  (function() {
    var hm = document.createElement("script");
    hm.src = "//hm.baidu.com/hm.js?b9a314ab40d04d805655aab1deee08ba";
    var s = document.getElementsByTagName("script")[0]; 
    s.parentNode.insertBefore(hm, s);
  })();
  </script>

  

  
  <script src="../_static/js/modernizr.min.js"></script>

</head>

<body class="wy-body-for-nav" role="document">

  
  <header class="site-header">
    <div class="site-logo">
      <a href="/"><img src="../_static/images/PP_w.png"></a>
    </div>
    <div class="site-nav-links">
      <div class="site-menu">
        <a class="fork-on-github" href="https://github.com/PaddlePaddle/Paddle" target="_blank"><i class="fa fa-github"></i>Fork me on Github</a>
        <div class="language-switcher dropdown">
          <a type="button" data-toggle="dropdown">
            <span>English</span>
            <i class="fa fa-angle-up"></i>
            <i class="fa fa-angle-down"></i>
          </a>
          <ul class="dropdown-menu">
            <li><a href="/doc_cn">中文</a></li>
            <li><a href="/doc">English</a></li>
          </ul>
        </div>
        <ul class="site-page-links">
          <li><a href="/">Home</a></li>
        </ul>
      </div>
      <div class="doc-module">
        
        <ul>
<li class="toctree-l1"><a class="reference internal" href="../getstarted/index_cn.html">新手入门</a></li>
<li class="toctree-l1"><a class="reference internal" href="../howto/index_cn.html">进阶指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../api/index_cn.html">API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq/index_cn.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../mobile/index_cn.html">MOBILE</a></li>
</ul>

        
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>        
      </div>
    </div>
  </header>
  
  <div class="main-content-wrap">

    
    <nav class="doc-menu-vertical" role="navigation">
        
          
          <ul>
<li class="toctree-l1"><a class="reference internal" href="../getstarted/index_cn.html">新手入门</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../getstarted/build_and_install/index_cn.html">安装与编译</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/pip_install_cn.html">使用pip安装</a></li>
<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/docker_install_cn.html">使用Docker安装运行</a></li>
<li class="toctree-l3"><a class="reference internal" href="../howto/dev/build_cn.html">用Docker编译和测试PaddlePaddle</a></li>
<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/build_from_source_cn.html">从源码编译</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../getstarted/concepts/use_concepts_cn.html">基本使用概念</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../howto/index_cn.html">进阶指南</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../howto/usage/cmd_parameter/index_cn.html">设置命令行参数</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/use_case_cn.html">使用案例</a></li>
<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/arguments_cn.html">参数概述</a></li>
<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/detail_introduction_cn.html">细节描述</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../howto/usage/cluster/cluster_train_cn.html">PaddlePaddle分布式训练</a></li>
<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_basis_cn.html">Kubernetes 简介</a></li>
<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_cn.html">Kubernetes单机训练</a></li>
<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_distributed_cn.html">Kubernetes分布式训练</a></li>
<li class="toctree-l2"><a class="reference internal" href="../howto/dev/contribute_to_paddle_cn.html">如何贡献代码</a></li>
<li class="toctree-l2"><a class="reference internal" href="../howto/dev/write_docs_cn.html">如何贡献/修改文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../howto/deep_model/rnn/index_cn.html">RNN相关模型</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/rnn_config_cn.html">RNN配置</a></li>
<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/recurrent_group_cn.html">Recurrent Group教程</a></li>
<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/hierarchical_layer_cn.html">支持双层序列作为输入的Layer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/hrnn_rnn_api_compare_cn.html">单双层RNN API对比介绍</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../howto/optimization/gpu_profiling_cn.html">GPU性能分析与调优</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../api/index_cn.html">API</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../api/v2/model_configs.html">模型配置</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/activation.html">Activation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/layer.html">Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/evaluators.html">Evaluators</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/optimizer.html">Optimizer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/pooling.html">Pooling</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/networks.html">Networks</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/attr.html">Parameter Attribute</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api/v2/data.html">数据访问</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/data/data_reader.html">Data Reader Interface</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/data/image.html">Image Interface</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api/v2/data/dataset.html">Dataset</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api/v2/run_logic.html">训练与应用</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../faq/index_cn.html">FAQ</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../faq/build_and_install/index_cn.html">编译安装与单元测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../faq/model/index_cn.html">模型配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../faq/parameter/index_cn.html">参数设置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../faq/local/index_cn.html">本地训练与预测</a></li>
<li class="toctree-l2"><a class="reference internal" href="../faq/cluster/index_cn.html">集群训练与预测</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../mobile/index_cn.html">MOBILE</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../mobile/cross_compiling_for_android_cn.html">Android平台编译指南</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mobile/cross_compiling_for_ios_cn.html">iOS平台编译指南</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mobile/cross_compiling_for_raspberry_cn.html">Raspberry Pi平台编译指南</a></li>
</ul>
</li>
</ul>

        
    </nav>
    
    <section class="doc-content-wrap">

      

 







<div role="navigation" aria-label="breadcrumbs navigation">
  <ul class="wy-breadcrumbs">
      
    <li>Design Doc: Refactorization Overview</li>
  </ul>
</div>
      
      <div class="wy-nav-content" id="doc-content">
        <div class="rst-content">
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="design-doc-refactorization-overview">
<span id="design-doc-refactorization-overview"></span><h1>Design Doc: Refactorization Overview<a class="headerlink" href="#design-doc-refactorization-overview" title="永久链接至标题"></a></h1>
<p>The goals of refactoring include:</p>
<ol class="simple">
<li>Making it easy for external contributors to write new elementary computation operations.</li>
<li>Making the codebase clean and readable.</li>
<li>Designing a new computation representation &#8211; a computation graph of operators and variables.</li>
<li>Implementing auto-scalability and auto fault recoverable distributed computing with the help of computation graphs.</li>
</ol>
<div class="section" id="computation-graphs">
<span id="computation-graphs"></span><h2>Computation Graphs<a class="headerlink" href="#computation-graphs" title="永久链接至标题"></a></h2>
<ol class="simple">
<li>PaddlePaddle represents the computation, training and inference of Deep Learning models, by computation graphs.</li>
<li>Please refer to <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/graph.md">computation graphs</a> for a concrete example.</li>
<li>Users write Python programs to describe the graphs and run them (locally or remotely).</li>
<li>A graph is composed of <em>variables</em> and <em>operators</em>.</li>
<li>The description of graphs must be serializable/deserializable, so that:<ol>
<li>It can be sent to the cloud for distributed execution, and</li>
<li>It can be sent to clients for mobile or enterprise deployment.</li>
</ol>
</li>
<li>The Python program does two things<ol>
<li><em>Compilation</em> runs a Python program to generate a protobuf message representation of the graph and send it to<ol>
<li>the C++ library <code class="docutils literal"><span class="pre">libpaddle.so</span></code> for local execution,</li>
<li>the master process of a distributed training job for training, or</li>
<li>the server process of a Kubernetes serving job for distributed serving.</li>
</ol>
</li>
<li><em>Execution</em> executes the graph by constructing instances of class <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24"><code class="docutils literal"><span class="pre">Variable</span></code></a> and <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L70"><code class="docutils literal"><span class="pre">OperatorBase</span></code></a>, according to the protobuf message.</li>
</ol>
</li>
</ol>
</div>
<div class="section" id="description-and-realization-of-computation-graph">
<span id="description-and-realization-of-computation-graph"></span><h2>Description and Realization of Computation Graph<a class="headerlink" href="#description-and-realization-of-computation-graph" title="永久链接至标题"></a></h2>
<p>At compile time, the Python program generates a protobuf message representation of the graph, or a description of the graph.</p>
<p>At runtime, the C++ program realizes the graph and runs it.</p>
<p>| | Representation (protobuf messages) | Realization (C++ class objects) |
|&#8212;|&#8212;|&#8212;|
|Data|<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto#L107">VarDesc</a>|<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/variable.h#L24">Variable</a>|
|Operation|<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto#L35">OpDesc</a>|<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/operator.h#L64">Operator</a>|
|Block|BlockDesc|Block|</p>
<p>The word <em>graph</em> is interchangeable with <em>block</em> in this document.  A graph consists of computation steps and local variables similar to a C++/Java program block, or a pair of parentheses(<code class="docutils literal"><span class="pre">{</span></code> and <code class="docutils literal"><span class="pre">}</span></code>).</p>
</div>
<div class="section" id="compilation-and-execution">
<span id="compilation-and-execution"></span><h2>Compilation and Execution<a class="headerlink" href="#compilation-and-execution" title="永久链接至标题"></a></h2>
<ol class="simple">
<li>Run a Python program to describe the graph.  In particular, the Python application program does the following:<ol>
<li>Create <code class="docutils literal"><span class="pre">VarDesc</span></code> to represent local/intermediate variables,</li>
<li>Create operators and set attributes,</li>
<li>Validate attribute values,</li>
<li>Infer the type and the shape of variables,</li>
<li>Plan memory-reuse for variables,</li>
<li>Generate the backward graph</li>
<li>Add optimization operators to the computation graph.</li>
<li>Optionally, split the graph for distributed training.</li>
</ol>
</li>
<li>The invocation of <code class="docutils literal"><span class="pre">train</span></code> or <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/inference.py#L108"><code class="docutils literal"><span class="pre">infer</span></code></a> methods in the Python program does the following:<ol>
<li>Create a new Scope instance in the <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/scope.md">scope hierarchy</a> for each run of a block,<ol>
<li>realize local variables defined in the BlockDesc message in the new scope,</li>
<li>a scope is similar to the stack frame in programming languages,</li>
</ol>
</li>
<li>Create an instance of class <code class="docutils literal"><span class="pre">Block</span></code>, in which,<ol>
<li>realize operators in the BlockDesc message,</li>
</ol>
</li>
<li>Run the Block by calling<ol>
<li><code class="docutils literal"><span class="pre">Block::Eval(vector&lt;Variable&gt;*</span> <span class="pre">targets)</span></code> for forward and backward computations, or</li>
<li><code class="docutils literal"><span class="pre">Block::Eval(vector&lt;Operator&gt;*</span> <span class="pre">targets)</span></code> for optimization.</li>
</ol>
</li>
</ol>
</li>
</ol>
</div>
<div class="section" id="intermediate-representation-ir">
<span id="intermediate-representation-ir"></span><h2>Intermediate Representation (IR)<a class="headerlink" href="#intermediate-representation-ir" title="永久链接至标题"></a></h2>
<div class="highlight-text"><div class="highlight"><pre><span></span>Compile Time -&gt; IR -&gt; Runtime
</pre></div>
</div>
<div class="section" id="benefits-of-ir">
<span id="benefits-of-ir"></span><h3>Benefits of IR<a class="headerlink" href="#benefits-of-ir" title="永久链接至标题"></a></h3>
<ul>
<li><p class="first">Optimization</p>
<div class="highlight-text"><div class="highlight"><pre><span></span>Compile Time -&gt; IR -&gt; Optimized IR -&gt; Runtime
</pre></div>
</div>
</li>
<li><p class="first">Automatically send partitioned IR to different nodes.</p>
<ul>
<li><p class="first">Automatic Data Parallelism</p>
<div class="highlight-text"><div class="highlight"><pre><span></span>Compile Time
|-&gt; Single GPU IR
    |-&gt; [trainer-IR-0, trainer-IR-1, pserver-IR]
        |-&gt; Node-0 (runs trainer-IR-0)
        |-&gt; Node-1 (runs trainer-IR-1)
        |-&gt; Node-2 (runs pserver-IR)
</pre></div>
</div>
</li>
<li><p class="first">Automatic Model Parallelism (planned for future)</p>
</li>
</ul>
</li>
</ul>
</div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="operator-opwithkernel-opkernel">
<span id="operator-opwithkernel-opkernel"></span><h1>Operator/OpWithKernel/OpKernel<a class="headerlink" href="#operator-opwithkernel-opkernel" title="永久链接至标题"></a></h1>
<p><img alt="class_diagram" src="http://api.paddlepaddle.org/graphviz?dot=https://gist.githubusercontent.com/reyoung/53df507f6749762675dff3e7ce53372f/raw/49caf1fb70820fb4a6c217634317c9306f361f36/op_op_with_kern_class_diagram.dot" /></p>
</div>
<hr class="docutils" />
<div class="section" id="operator">
<span id="operator"></span><h1>Operator<a class="headerlink" href="#operator" title="永久链接至标题"></a></h1>
<p><img alt="class_diagram" src="http://api.paddlepaddle.org/graphviz?dot=https://gist.githubusercontent.com/reyoung/53df507f6749762675dff3e7ce53372f/raw/dd598e8f1976f5759f58af5e5ef94738a6b2e661/op.dot" /></p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">Operator</span></code> is the fundamental building block of the user interface.<ul>
<li>Operator stores input/output variable names and attributes.</li>
<li>The <code class="docutils literal"><span class="pre">InferShape</span></code> interface is used to infer the shape of the output variables based on the shapes of the input variables.</li>
<li>Use <code class="docutils literal"><span class="pre">Run</span></code> to compute the <code class="docutils literal"><span class="pre">output</span></code> variables from the <code class="docutils literal"><span class="pre">input</span></code> variables.</li>
</ul>
</li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="opwithkernel-kernel">
<span id="opwithkernel-kernel"></span><h1>OpWithKernel/Kernel<a class="headerlink" href="#opwithkernel-kernel" title="永久链接至标题"></a></h1>
<p><img alt="class_diagram" src="http://api.paddlepaddle.org/graphviz?dot=https://gist.githubusercontent.com/reyoung/53df507f6749762675dff3e7ce53372f/raw/9d7f4eba185cf41c8e2fbfb40ae21890dbddcd39/op_with_kernel.dot" /></p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">OpWithKernel</span></code> inherits <code class="docutils literal"><span class="pre">Operator</span></code>.</li>
<li><code class="docutils literal"><span class="pre">OpWithKernel</span></code> contains a Kernel map.<ul>
<li><code class="docutils literal"><span class="pre">OpWithKernel::Run</span></code> get device&#8217;s kernel, and invoke <code class="docutils literal"><span class="pre">OpKernel::Compute</span></code>.</li>
<li><code class="docutils literal"><span class="pre">OpKernelKey</span></code> is the map key. Only device place now, but may be data type later.</li>
</ul>
</li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="why-separate-kernel-and-operator">
<span id="why-separate-kernel-and-operator"></span><h1>Why separate Kernel and Operator<a class="headerlink" href="#why-separate-kernel-and-operator" title="永久链接至标题"></a></h1>
<ul class="simple">
<li>Separate GPU and CPU code.<ul>
<li>Make Paddle capable of running without GPU.</li>
</ul>
</li>
<li>Make one operator (which is a user interface) and create many implementations.<ul>
<li>For example, same multiplication op can have different implementations kernels such as FP16 kernel, FP32 kernel, MKL, eigen kernel.</li>
</ul>
</li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="libraries-for-kernel-development">
<span id="libraries-for-kernel-development"></span><h1>Libraries for Kernel development<a class="headerlink" href="#libraries-for-kernel-development" title="永久链接至标题"></a></h1>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">Eigen::Tensor</span></code> contains basic math and element-wise functions.<ul>
<li>Note that <code class="docutils literal"><span class="pre">Eigen::Tensor</span></code> has broadcast implementation.</li>
<li>Limit the number of <code class="docutils literal"><span class="pre">tensor.device(dev)</span> <span class="pre">=</span></code> in your code.</li>
</ul>
</li>
<li><code class="docutils literal"><span class="pre">thrust::transform</span></code> and <code class="docutils literal"><span class="pre">std::transform</span></code>.<ul>
<li><code class="docutils literal"><span class="pre">thrust</span></code> has the same API as C++ standard library. Using <code class="docutils literal"><span class="pre">transform</span></code>, one can quickly implement customized element-wise kernels.</li>
<li><code class="docutils literal"><span class="pre">thrust</span></code>, in addition, supports more complex APIs, like <code class="docutils literal"><span class="pre">scan</span></code>, <code class="docutils literal"><span class="pre">reduce</span></code>, <code class="docutils literal"><span class="pre">reduce_by_key</span></code>.</li>
</ul>
</li>
<li>Hand-writing <code class="docutils literal"><span class="pre">GPUKernel</span></code> and <code class="docutils literal"><span class="pre">CPU</span></code> code<ul>
<li>Do not write in header (<code class="docutils literal"><span class="pre">.h</span></code>) files. CPU Kernel should be in cpp source (<code class="docutils literal"><span class="pre">.cc</span></code>) and GPU kernels should be in cuda (<code class="docutils literal"><span class="pre">.cu</span></code>) files. (GCC cannot compile GPU code.)</li>
</ul>
</li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="operator-registration">
<span id="operator-registration"></span><h1>Operator Registration<a class="headerlink" href="#operator-registration" title="永久链接至标题"></a></h1>
<div class="section" id="why-is-registration-necessary">
<span id="why-is-registration-necessary"></span><h2>Why is registration necessary?<a class="headerlink" href="#why-is-registration-necessary" title="永久链接至标题"></a></h2>
<p>We need a method to build mappings between Op type names and Op classes.</p>
</div>
<div class="section" id="how-is-registration-implemented">
<span id="how-is-registration-implemented"></span><h2>How is registration implemented?<a class="headerlink" href="#how-is-registration-implemented" title="永久链接至标题"></a></h2>
<p>Maintaining a map, whose key is the type name and the value is the corresponding Op constructor.</p>
</div>
</div>
<hr class="docutils" />
<div class="section" id="the-registry-map">
<span id="the-registry-map"></span><h1>The Registry Map<a class="headerlink" href="#the-registry-map" title="永久链接至标题"></a></h1>
<div class="section" id="opinfomap">
<span id="opinfomap"></span><h2><code class="docutils literal"><span class="pre">OpInfoMap</span></code><a class="headerlink" href="#opinfomap" title="永久链接至标题"></a></h2>
<p><code class="docutils literal"><span class="pre">op_type(string)</span></code> -&gt; <code class="docutils literal"><span class="pre">OpInfo</span></code></p>
<p><code class="docutils literal"><span class="pre">OpInfo</span></code>:</p>
<ul class="simple">
<li><strong><code class="docutils literal"><span class="pre">creator</span></code></strong>: The Op constructor.</li>
<li><strong><code class="docutils literal"><span class="pre">grad_op_type</span></code></strong>: The type of the gradient Op.</li>
<li><strong><code class="docutils literal"><span class="pre">proto</span></code></strong>: The Op&#8217;s Protobuf, including inputs, outputs and required attributes.</li>
<li><strong><code class="docutils literal"><span class="pre">checker</span></code></strong>: Used to check attributes.</li>
</ul>
</div>
</div>
<hr class="docutils" />
<div class="section" id="related-concepts">
<span id="related-concepts"></span><h1>Related Concepts<a class="headerlink" href="#related-concepts" title="永久链接至标题"></a></h1>
<div class="section" id="op-maker">
<span id="op-maker"></span><h2>Op_Maker<a class="headerlink" href="#op-maker" title="永久链接至标题"></a></h2>
<p>It&#8217;s constructor takes <code class="docutils literal"><span class="pre">proto</span></code> and <code class="docutils literal"><span class="pre">checker</span></code>. They are completed during Op_Maker&#8217;s construction. (<a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/scale_op.cc#L37">ScaleOpMaker</a>)</p>
</div>
<div class="section" id="register-macros">
<span id="register-macros"></span><h2>Register Macros<a class="headerlink" href="#register-macros" title="永久链接至标题"></a></h2>
<div class="highlight-cpp"><div class="highlight"><pre><span></span><span class="n">REGISTER_OP</span><span class="p">(</span><span class="n">op_type</span><span class="p">,</span> <span class="n">op_class</span><span class="p">,</span> <span class="n">op_maker_class</span><span class="p">,</span> <span class="n">grad_op_type</span><span class="p">,</span> <span class="n">grad_op_class</span><span class="p">)</span>
<span class="n">REGISTER_OP_WITHOUT_GRADIENT</span><span class="p">(</span><span class="n">op_type</span><span class="p">,</span> <span class="n">op_class</span><span class="p">,</span> <span class="n">op_maker_class</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<hr class="docutils" />
<div class="section" id="registration-process">
<span id="registration-process"></span><h1>Registration Process<a class="headerlink" href="#registration-process" title="永久链接至标题"></a></h1>
<ol class="simple">
<li>Write an Op class and its gradient Op class, if required.</li>
<li>Write an Op maker class. In the constructor of this class, describe the inputs, outputs and attributes of the operator.</li>
<li>Invoke the macro <code class="docutils literal"><span class="pre">REGISTER_OP</span></code>. This macro will<ol>
<li>Call maker class to complete <code class="docutils literal"><span class="pre">proto</span></code> and <code class="docutils literal"><span class="pre">checker</span></code></li>
<li>Using the completed <code class="docutils literal"><span class="pre">proto</span></code> and <code class="docutils literal"><span class="pre">checker</span></code>, it will add a new key-value pair to the <code class="docutils literal"><span class="pre">OpInfoMap</span></code></li>
</ol>
</li>
</ol>
</div>
<hr class="docutils" />
<div class="section" id="backward-module-1-2">
<span id="backward-module-1-2"></span><h1>Backward Module (1/2)<a class="headerlink" href="#backward-module-1-2" title="永久链接至标题"></a></h1>
<div class="section" id="create-backward-operator">
<span id="create-backward-operator"></span><h2>Create Backward Operator<a class="headerlink" href="#create-backward-operator" title="永久链接至标题"></a></h2>
<ul class="simple">
<li>Mapping from forward Op to backward Op
<img alt="backward" src="https://gist.githubusercontent.com/dzhwinter/a6fbd4623ee76c459f7f94591fd1abf0/raw/61026ab6e518e66bde66a889bc42557a1fccff33/backward.png" /></li>
</ul>
</div>
</div>
<hr class="docutils" />
<div class="section" id="backward-module-2-2">
<span id="backward-module-2-2"></span><h1>Backward Module (2/2)<a class="headerlink" href="#backward-module-2-2" title="永久链接至标题"></a></h1>
<div class="section" id="build-backward-network">
<span id="build-backward-network"></span><h2>Build Backward Network<a class="headerlink" href="#build-backward-network" title="永久链接至标题"></a></h2>
<ul class="simple">
<li><strong>Input</strong>: a graph of forward operators</li>
<li><strong>Output</strong>: a graph of backward operators</li>
<li><strong>Corner cases in construction</strong><ul>
<li>Shared Variables =&gt; insert an <code class="docutils literal"><span class="pre">Add</span></code> operator to combine gradients</li>
<li>No Gradient =&gt; insert a <code class="docutils literal"><span class="pre">fill_zero_grad</span></code> operator</li>
<li>Recursive NetOp =&gt; call <code class="docutils literal"><span class="pre">Backward</span></code> recursively</li>
<li>RNN Op =&gt; recursively call <code class="docutils literal"><span class="pre">Backward</span></code> on stepnet</li>
<li>RNN Op =&gt; recursively call <code class="docutils literal"><span class="pre">Backward</span></code> on stepnet</li>
</ul>
</li>
</ul>
</div>
</div>
<hr class="docutils" />
<div class="section" id="scope-variable-tensor">
<span id="scope-variable-tensor"></span><h1>Scope, Variable, Tensor<a class="headerlink" href="#scope-variable-tensor" title="永久链接至标题"></a></h1>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">Tensor</span></code> is an n-dimension array with type.<ul>
<li>Only dims and data pointers are stored in <code class="docutils literal"><span class="pre">Tensor</span></code>.</li>
<li>All operations on <code class="docutils literal"><span class="pre">Tensor</span></code> are written in <code class="docutils literal"><span class="pre">Operator</span></code> or global functions.</li>
<li>Variable length Tensor design <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md">LoDTensor</a></li>
</ul>
</li>
<li><code class="docutils literal"><span class="pre">Variable</span></code> instances are the inputs and the outputs of an operator, not just <code class="docutils literal"><span class="pre">Tensor</span></code>.<ul>
<li><code class="docutils literal"><span class="pre">step_scopes</span></code> in RNN is a variable and not a tensor.</li>
</ul>
</li>
<li><code class="docutils literal"><span class="pre">Scope</span></code> is where variables are stored.<ul>
<li>map&lt;string <code class="docutils literal"><span class="pre">var</span> <span class="pre">name</span></code>, Variable&gt;</li>
<li><code class="docutils literal"><span class="pre">Scope</span></code> has a hierarchical structure. The local scope can get variables from its parent scope.</li>
</ul>
</li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="block-in-design">
<span id="block-in-design"></span><h1>Block (in design)<a class="headerlink" href="#block-in-design" title="永久链接至标题"></a></h1>
<div class="section" id="the-difference-between-original-rnnop-and-block">
<span id="the-difference-between-original-rnnop-and-block"></span><h2>the difference between original RNNOp and Block<a class="headerlink" href="#the-difference-between-original-rnnop-and-block" title="永久链接至标题"></a></h2>
<ul class="simple">
<li>As an operator is more intuitive than <code class="docutils literal"><span class="pre">RNNOp</span></code>,</li>
<li>Offers a new interface <code class="docutils literal"><span class="pre">Eval(targets)</span></code> to deduce the minimal block to <code class="docutils literal"><span class="pre">Run</span></code>,</li>
<li>Fits the compile-time/ runtime separation design paradigm.<ul>
<li>During the compilation, <code class="docutils literal"><span class="pre">SymbolTable</span></code> stores <code class="docutils literal"><span class="pre">VarDesc</span></code>s and <code class="docutils literal"><span class="pre">OpDesc</span></code>s and serialize to a <code class="docutils literal"><span class="pre">BlockDesc</span></code></li>
<li>When graph executes, a Block with <code class="docutils literal"><span class="pre">BlockDesc</span></code> is passed. It then creates <code class="docutils literal"><span class="pre">Op</span></code> and <code class="docutils literal"><span class="pre">Var</span></code> instances and then invokes <code class="docutils literal"><span class="pre">Run</span></code>.</li>
</ul>
</li>
</ul>
</div>
</div>
<hr class="docutils" />
<div class="section" id="milestone">
<span id="milestone"></span><h1>Milestone<a class="headerlink" href="#milestone" title="永久链接至标题"></a></h1>
<ul class="simple">
<li>Take Paddle/books as the main line, the requirement of the models motivates framework refactoring,</li>
<li>Model migration<ul>
<li>Framework development gives <strong>priority support</strong> to model migration, for example,<ul>
<li>the MNIST demo needs a Python interface,</li>
<li>the RNN models require the framework to support <code class="docutils literal"><span class="pre">LoDTensor</span></code>.</li>
</ul>
</li>
<li>Determine some timelines,</li>
<li>Frequently used Ops need to be migrated first,</li>
<li>Different models can be migrated in parallel.</li>
</ul>
</li>
<li>Improve the framework at the same time</li>
<li>Accept imperfection, concentrate on solving the specific problem at the right price.</li>
</ul>
</div>
<hr class="docutils" />
<div class="section" id="control-the-migration-quality">
<span id="control-the-migration-quality"></span><h1>Control the migration quality<a class="headerlink" href="#control-the-migration-quality" title="永久链接至标题"></a></h1>
<ul class="simple">
<li>Compare the performance of migrated models with old ones.</li>
<li>Follow the google C++ style guide.</li>
<li>Build the automatic workflow of generating Python/C++ documentations.<ul>
<li>The documentation of layers and ops should be written inside the code.</li>
<li>Take the documentation quality into account when submitting pull requests.</li>
<li>Preview the documentations, read and improve them from a user&#8217;s perspective.</li>
</ul>
</li>
</ul>
</div>


           </div>
          </div>
          <footer>
  

  <hr/>

  <div role="contentinfo">
    <p>
        &copy; Copyright 2016, PaddlePaddle developers.

    </p>
  </div>
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  


  

    <script type="text/javascript">
        var DOCUMENTATION_OPTIONS = {
            URL_ROOT:'../',
            VERSION:'',
            COLLAPSE_INDEX:false,
            FILE_SUFFIX:'.html',
            HAS_SOURCE:  true,
            SOURCELINK_SUFFIX: ".txt",
        };
    </script>
      <script type="text/javascript" src="../_static/jquery.js"></script>
      <script type="text/javascript" src="../_static/underscore.js"></script>
      <script type="text/javascript" src="../_static/doctools.js"></script>
      <script type="text/javascript" src="../_static/translations.js"></script>
      <script type="text/javascript" src="https://cdn.bootcss.com/mathjax/2.7.0/MathJax.js"></script>
       
  

  
  
    <script type="text/javascript" src="../_static/js/theme.js"></script>
  
  
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
  <script src="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/js/perfect-scrollbar.jquery.min.js"></script>
  <script src="../_static/js/paddle_doc_init.js"></script> 

</body>
</html>