[MKL-DNN] Tensors conversion design doc (#1153)

* MKL-DNN Data transform doc design

- Cosmetic

- Changes

- CPU -> Paddle conversion

- Finished data transform draft

- Cosmetic fixes

* - Fixes after internal review

doc/fluid/design/mkldnn/data_transformation/data_transform.md

+# Design Doc: MKL-DNN Data Transformation
+
+When fluid is using MKL-DNN engine to execute program, not all operators are having mkl-dnn kernels and some of operators are executed by CPU. MKL-DNN kernels of operators expect input Tensors to be provided in MKL-DNN layout , while Paddle(CPU) kernels are expecting input Tensors to be of Paddle layout.
+
+We can distinguish following scenarios(presented below on the picture):
+* Paddle(CPU) kernel is followed by MKL-DNN kernel
+* MKL-DNN kernel is followed by Paddle(CPU) kernel
+* MKL-DNN kernel is followed by fetch operator
+
+
+![](images/data_transform.svg)
+
+
+### Paddle(CPU) kernel is followed by MKL-DNN kernel
+In a situation when Paddle(CPU) kernel finished execution, its outcome is one or many Tensors of Paddle layout. Each of those
+Tensors to be feed into MKL-DNN kernel, needs to be transformed to be of MKL-DNN layout. For this scenario conversion of Paddle Tensor to MKL-DNN Tensor is done by just
+changing layout flag to MKL-DNN and picking MKL-DNN format that match Paddle Tensor rank. This is computationally cheap operation as there is no real data rearrangement.
+
+This scenario is drawn on the picture with bold lines. Starting from Paddle(CPU) op on the left side , following arrows drawn in bold and finishing with MKL-DNN op on the right side of picture.
+
+### MKL-DNN kernel is followed by Paddle(CPU) kernel
+In this situation MKL-DNN kernel finished its execution and as a result it produced one or more output Tensors. Each of those Tensors are of MKL-DNN layout and to be fed into Paddle(CPU) kernel,
+they need to be converted into Paddle layout. In a detail MKL-DNN Tensor arrangement (mkl-dnn memory format) is checked if it is compatible with Paddle(CPU) layout and if positive then
+just layout of Tensor is set as Paddle and mkl-dnn format is set to ``undef``. In case when MKL-DNN Tensor data arrangement is not compatible with Paddle layout then actual data arrangement
+is performed. For example MKL-DNN Tensor is 4D and having format ``NCHW16C`` and to convert it into Paddle layout we need to rearrange data to be ``NCHW`` format. To do so
+MKL-DNN Reorder primitive is created that can do data rearrangement. 
+
+This scenario is marked on the picture with outlined, empty inside arrows. Starting from MKL-DNN op on the left side , following empty arrows finishing with Paddle(CPU) op on the right side of picture.
+### MKL-DNN kernel is followed by fetch operator
+This situation is similar conceptually to previous section, but because fetch operator is an operator without kernel then it does not share data transformation code with operators that are having kernel registered.
+Hence execution flow looks a bit different, although conceptually conversion of MKL-DNN Tensor into Paddle(CPU) Tensor is the same as in a described above
+
+This scenario is marked on the picture with regular arrows. Starting from MKL-DNN op on the left side , following regular arrows finishing with fetch op on the right side of picture.
+### GPU and MKL-DNN kernels interoperability.
+Currently Fluid is not supporting execution of programs by using combination of MKL-DNN and GPU kernels

doc/fluid/design/mkldnn/data_transformation/images/data_transform.svg

+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by graphviz version 2.38.0 (20140413.2041)
+ -->
+<!-- Title: Q Pages: 1 -->
+<svg width="1314pt" height="456pt"
+ viewBox="0.00 0.00 1313.58 456.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 452)">
+<title>Q</title>
+<polygon fill="white" stroke="none" points="-4,4 -4,-452 1309.58,-452 1309.58,4 -4,4"/>
+<g id="clust1" class="cluster"><title>cluster_in</title>
+<polygon fill="none" stroke="black" stroke-dasharray="1,5" points="0,-68 0,-399 186.788,-399 186.788,-68 0,-68"/>
+<text text-anchor="start" x="7.89389" y="-384.8" font-family="Times,serif" font-size="14.00">Possible </text>
+<text text-anchor="start" x="56.8939" y="-384.8" font-family="Times,serif" font-weight="bold" font-size="14.00">i</text>
+<text text-anchor="start" x="60.8939" y="-384.8" font-family="Times,serif" font-size="14.00"> &#160;order operator types</text>
+</g>
+<g id="clust2" class="cluster"><title>cluster_out</title>
+<polygon fill="none" stroke="black" stroke-dasharray="1,5" points="1103.79,-8 1103.79,-440 1305.58,-440 1305.58,-8 1103.79,-8"/>
+<text text-anchor="start" x="1111.68" y="-425.8" font-family="Times,serif" font-size="14.00">Possible </text>
+<text text-anchor="start" x="1160.68" y="-425.8" font-family="Times,serif" font-weight="bold" font-size="14.00">i+1</text>
+<text text-anchor="start" x="1179.68" y="-425.8" font-family="Times,serif" font-size="14.00"> &#160;order operator types</text>
+</g>
+<g id="clust3" class="cluster"><title>cluster_Trans</title>
+<polygon fill="none" stroke="black" stroke-dasharray="1,5" points="316.788,-164 316.788,-293 1041.79,-293 1041.79,-164 316.788,-164"/>
+<text text-anchor="middle" x="679.288" y="-277.8" font-family="Times,serif" font-size="14.00">Transformation of Tensors</text>
+</g>
+<!-- TransData -->
+<g id="node1" class="node"><title>TransData</title>
+<polygon fill="none" stroke="black" points="424.788,-226 324.788,-226 324.788,-190 424.788,-190 424.788,-226"/>
+<text text-anchor="middle" x="374.788" y="-204.3" font-family="Times,serif" font-size="14.00">TransformData</text>
+</g>
+<!-- toMKLDNN -->
+<g id="node2" class="node"><title>toMKLDNN</title>
+<polygon fill="none" stroke="black" points="753.288,-262 581.288,-262 581.288,-226 753.288,-226 753.288,-262"/>
+<text text-anchor="middle" x="667.288" y="-240.3" font-family="Times,serif" font-size="14.00">Label Tensor as MKL&#45;DNN</text>
+</g>
+<!-- TransData&#45;&gt;toMKLDNN -->
+<g id="edge7" class="edge"><title>TransData&#45;&gt;toMKLDNN</title>
+<path fill="none" stroke="black" stroke-width="2" d="M425.067,-214.107C464.972,-219.052 522.371,-226.165 571.166,-232.212"/>
+<polygon fill="black" stroke="black" stroke-width="2" points="570.766,-235.689 581.12,-233.446 571.626,-228.742 570.766,-235.689"/>
+</g>
+<!-- fromMKLDNN -->
+<g id="node3" class="node"><title>fromMKLDNN</title>
+<polygon fill="none" stroke="black" points="767.788,-208 566.788,-208 566.788,-172 767.788,-172 767.788,-208"/>
+<text text-anchor="middle" x="667.288" y="-186.3" font-family="Times,serif" font-size="14.00">TransDataLayoutFromMKLDNN</text>
+</g>
+<!-- TransData&#45;&gt;fromMKLDNN -->
+<g id="edge8" class="edge"><title>TransData&#45;&gt;fromMKLDNN</title>
+<path fill="none" stroke="black" d="M424.943,-202.95C460.912,-200.722 511.091,-197.612 556.374,-194.807"/>
+<path fill="none" stroke="none" d="M425.067,-204.947C461.035,-202.718 511.215,-199.609 556.497,-196.803"/>
+<path fill="none" stroke="black" d="M425.191,-206.943C461.159,-204.714 511.339,-201.605 556.621,-198.799"/>
+<polygon fill="black" stroke="black" points="556.915,-200.284 566.679,-196.172 556.482,-193.297 556.915,-200.284"/>
+</g>
+<!-- mkldnnop2 -->
+<g id="node9" class="node"><title>mkldnnop2</title>
+<ellipse fill="none" stroke="black" stroke-width="2" cx="1204.68" cy="-345" rx="63.8893" ry="63.8893"/>
+<text text-anchor="middle" x="1204.68" y="-341.3" font-family="Times,serif" font-size="14.00">MKL&#45;DNN op</text>
+</g>
+<!-- toMKLDNN&#45;&gt;mkldnnop2 -->
+<g id="edge4" class="edge"><title>toMKLDNN&#45;&gt;mkldnnop2</title>
+<path fill="none" stroke="black" stroke-width="2" d="M753.418,-260.06C857.998,-279.788 1033.58,-312.91 1131.73,-331.426"/>
+<polygon fill="black" stroke="black" stroke-width="2" points="1131.38,-334.921 1141.85,-333.336 1132.67,-328.043 1131.38,-334.921"/>
+<text text-anchor="middle" x="919.288" y="-317.8" font-family="Times,serif" font-size="14.00">MKL&#45;DNN Tensor </text>
+</g>
+<!-- innerMKLDNN -->
+<g id="node4" class="node"><title>innerMKLDNN</title>
+<polygon fill="none" stroke="black" points="1033.79,-208 804.788,-208 804.788,-172 1033.79,-172 1033.79,-208"/>
+<text text-anchor="start" x="812.788" y="-193.8" font-family="Times,serif" font-size="14.00">innerTransDataLayoutFromMKLDNN</text>
+<text text-anchor="start" x="850.288" y="-179.8" font-family="Times,serif" font-size="14.00"> {</text>
+<text text-anchor="start" x="861.288" y="-179.8" font-family="Times,serif" font-weight="bold" font-size="14.00">MKL&#45;DNN Reorder</text>
+<text text-anchor="start" x="981.288" y="-179.8" font-family="Times,serif" font-size="14.00">}</text>
+</g>
+<!-- fromMKLDNN&#45;&gt;innerMKLDNN -->
+<g id="edge9" class="edge"><title>fromMKLDNN&#45;&gt;innerMKLDNN</title>
+<path fill="none" stroke="black" d="M767.813,-188C776.618,-188 785.599,-188 794.571,-188"/>
+<path fill="none" stroke="none" d="M767.813,-190C776.618,-190 785.599,-190 794.571,-190"/>
+<path fill="none" stroke="black" d="M767.813,-192C776.618,-192 785.599,-192 794.571,-192"/>
+<polygon fill="black" stroke="black" points="794.688,-193.5 804.688,-190 794.688,-186.5 794.688,-193.5"/>
+</g>
+<!-- cpuop2 -->
+<g id="node7" class="node"><title>cpuop2</title>
+<ellipse fill="none" stroke="black" cx="1204.68" cy="-190" rx="68.7937" ry="68.7937"/>
+<ellipse fill="none" stroke="black" cx="1204.68" cy="-190" rx="72.7879" ry="72.7879"/>
+<text text-anchor="middle" x="1204.68" y="-186.3" font-family="Times,serif" font-size="14.00">Paddle(CPU) op</text>
+</g>
+<!-- innerMKLDNN&#45;&gt;cpuop2 -->
+<g id="edge5" class="edge"><title>innerMKLDNN&#45;&gt;cpuop2</title>
+<path fill="none" stroke="black" d="M1033.92,-188C1063.14,-188 1094.02,-188 1121.32,-188"/>
+<path fill="none" stroke="none" d="M1033.92,-190C1063.14,-190 1094.02,-190 1121.32,-190"/>
+<path fill="none" stroke="black" d="M1033.92,-192C1063.14,-192 1094.02,-192 1121.32,-192"/>
+<polygon fill="black" stroke="black" points="1121.59,-193.5 1131.59,-190 1121.59,-186.5 1121.59,-193.5"/>
+<text text-anchor="middle" x="1072.79" y="-193.8" font-family="Times,serif" font-size="14.00">Tensor </text>
+</g>
+<!-- fetchop -->
+<g id="node8" class="node"><title>fetchop</title>
+<ellipse fill="none" stroke="black" cx="1204.68" cy="-58" rx="41.6928" ry="41.6928"/>
+<text text-anchor="middle" x="1204.68" y="-54.3" font-family="Times,serif" font-size="14.00">Fetch op</text>
+</g>
+<!-- innerMKLDNN&#45;&gt;fetchop -->
+<g id="edge6" class="edge"><title>innerMKLDNN&#45;&gt;fetchop</title>
+<path fill="none" stroke="black" d="M959.098,-171.923C1010.83,-147.827 1102.26,-105.242 1157.48,-79.5189"/>
+<polygon fill="black" stroke="black" points="1159.1,-82.6258 1166.69,-75.2308 1156.15,-76.2804 1159.1,-82.6258"/>
+<text text-anchor="middle" x="1072.79" y="-130.8" font-family="Times,serif" font-size="14.00">Tensor </text>
+</g>
+<!-- cpuop -->
+<g id="node5" class="node"><title>cpuop</title>
+<ellipse fill="none" stroke="black" stroke-width="2" cx="92.8939" cy="-299" rx="68.7879" ry="68.7879"/>
+<text text-anchor="middle" x="92.8939" y="-295.3" font-family="Times,serif" font-size="14.00">Paddle(CPU) op</text>
+</g>
+<!-- cpuop&#45;&gt;TransData -->
+<g id="edge1" class="edge"><title>cpuop&#45;&gt;TransData</title>
+<path fill="none" stroke="black" stroke-width="2" d="M158.811,-277.894C205.902,-262.584 269.086,-242.041 314.756,-227.193"/>
+<polygon fill="black" stroke="black" stroke-width="2" points="316.085,-230.441 324.513,-224.02 313.921,-223.784 316.085,-230.441"/>
+<text text-anchor="middle" x="251.788" y="-267.8" font-family="Times,serif" font-size="14.00">Tensor </text>
+</g>
+<!-- mkldnnop -->
+<g id="node6" class="node"><title>mkldnnop</title>
+<ellipse fill="none" stroke="black" cx="92.8939" cy="-144" rx="63.8777" ry="63.8777"/>
+<ellipse fill="none" stroke="black" cx="92.8939" cy="-144" rx="67.8893" ry="67.8893"/>
+<text text-anchor="middle" x="92.8939" y="-140.3" font-family="Times,serif" font-size="14.00">MKL&#45;DNN op</text>
+</g>
+<!-- mkldnnop&#45;&gt;TransData -->
+<g id="edge2" class="edge"><title>mkldnnop&#45;&gt;TransData</title>
+<path fill="none" stroke="black" d="M159.609,-156.975C206.615,-167.723 269.517,-182.106 315.062,-192.52"/>
+<path fill="none" stroke="none" d="M159.163,-158.924C206.169,-169.673 269.071,-184.056 314.616,-194.47"/>
+<path fill="none" stroke="black" d="M158.718,-160.874C205.723,-171.622 268.625,-186.005 314.17,-196.42"/>
+<polygon fill="black" stroke="black" points="314.109,-197.944 324.638,-196.761 315.67,-191.12 314.109,-197.944"/>
+<text text-anchor="middle" x="251.788" y="-195.8" font-family="Times,serif" font-size="14.00">MKL&#45;DNN Tensor </text>
+</g>
+<!-- mkldnnop&#45;&gt;innerMKLDNN -->
+<g id="edge3" class="edge"><title>mkldnnop&#45;&gt;innerMKLDNN</title>
+<path fill="none" stroke="black" d="M160.586,-142.592C281.825,-140.763 546.292,-140.138 767.788,-163 784.784,-164.754 802.748,-167.333 820.118,-170.211"/>
+<polygon fill="black" stroke="black" points="819.82,-173.711 830.266,-171.937 820.994,-166.81 819.82,-173.711"/>
+<text text-anchor="middle" x="495.788" y="-150.8" font-family="Times,serif" font-size="14.00">MKL&#45;DNN Tensor</text>
+</g>
+</g>
+</svg>

doc/fluid/design/mkldnn/data_transformation/index_en.rst

+MKL-DNN Data Transformation
+--------------------------------------
+
+.. toctree::
+  :maxdepth: 1
+
+  data_transform.md

doc/fluid/design/mkldnn/data_transformation/scripts/data_transform.dot

+
+digraph Q {
+
+  rankdir=LR
+  node[shape=box]
+
+  TransData[label="TransformData"] 
+  toMKLDNN[label="Label Tensor as MKL-DNN"]
+  fromMKLDNN[label="TransDataLayoutFromMKLDNN"]
+  innerMKLDNN[label=<innerTransDataLayoutFromMKLDNN<br/> {<b>MKL-DNN Reorder</b>}>]
+
+
+ node[shape=circle]
+
+ subgraph cluster_in {
+ label=<Possible <b>i</b>  order operator types>
+ style=dotted
+ cpuop[label="Paddle(CPU) op",style=bold]
+ mkldnnop[label="MKL-DNN op",shape=doublecircle]
+ }
+
+ subgraph cluster_out {
+ label=<Possible <b>i+1</b>  order operator types>
+ style=dotted
+ cpuop2[label="Paddle(CPU) op",shape=doublecircle]
+ fetchop[label="Fetch op"]
+ mkldnnop2[label="MKL-DNN op", style=bold]
+ }
+
+
+   cpuop -> TransData[label="Tensor ", style=bold] 
+   mkldnnop -> TransData[label="MKL-DNN Tensor ", color="black:invis:black"]
+   mkldnnop -> innerMKLDNN[label="MKL-DNN Tensor"]
+   toMKLDNN -> mkldnnop2[style=bold, label="MKL-DNN Tensor "] 
+   innerMKLDNN -> cpuop2[label="Tensor ", color="black:invis:black"]
+   innerMKLDNN -> fetchop[label="Tensor "] 
+ subgraph cluster_Trans {
+ label="Transformation of Tensors"
+ style=dotted
+   TransData -> toMKLDNN[style=bold]
+   TransData -> fromMKLDNN[color="black:invis:black"]
+   fromMKLDNN -> innerMKLDNN[color="black:invis:black"]
+    
+ }
+}
+