Deploy to GitHub Pages: c193f823

develop/doc/_sources/design/gan_api.md.txt

+# Design for GAN
+
+GAN (General Adversarial Net [https://arxiv.org/abs/1406.2661]) is an important model for unsupervised learning and widely used in many areas. 
+
+It applies several important concepts in machine learning system design, including building and running subgraphs, dependency tracing, different optimizers in one executor and so forth.
+
+In our GAN design, we wrap it as a user-friendly easily customized python API to design different models. We take the conditional DC-GAN (Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [https://arxiv.org/abs/1511.06434]) as an example due to its good performance on image generation.
+
+<p align="center">
+<img src="./test.dot.png" width = "35%" align="center"/><br/>
+Figure 1. The overall running logic of GAN. The black solid arrows indicate the forward pass; the green dashed arrows indicate the backward pass of generator training; the red dashed arrows indicate the backward pass of the discriminator training. The BP pass of the green (red) arrow should only update the parameters in the green (red) boxes. The diamonds indicate the data providers. d\_loss and g\_loss marked in red and green are the two targets we would like to run.
+</p>
+
+The operators, layers and functions required/optional to build a GAN demo is summarized in https://github.com/PaddlePaddle/Paddle/issues/4563.
+
+<p align="center">
+<img src="./dcgan.png" width = "90%" align="center"/><br/>
+Figure 2. Photo borrowed from the original DC-GAN paper.
+</p>
+
+## The Conditional-GAN might be a class. 
+This design we adopt the popular open source design in https://github.com/carpedm20/DCGAN-tensorflow and https://github.com/rajathkmp/DCGAN. It contains following data structure:
+
+- DCGAN(object): which contains everything required to build a GAN model. It provides following member functions methods as API:
+
+- __init__(...): Initialize hyper-parameters (like conv dimension and so forth), and declare model parameters of discriminator and generator as well.
+
+- generator(z, y=None): Generate a fake image from input noise z. If the label y is provided, the conditional GAN model will be chosen.
+Returns a generated image.
+
+- discriminator(image):
+Given an image, decide if it is from a real source or a fake one. 
+Returns a 0/1 binary label.
+
+- build_model(self):
+build the whole GAN model, define training loss for both generator and discrimator.
+
+## Discussion on Engine Functions required to build GAN
+- Trace the tensor and variable dependency in the engine executor. (Very critical, otherwise GAN can'be be trained correctly)
+- Different optimizers responsible for optimizing different loss.
+
+To be more detailed, we introduce our design of DCGAN as following:
+
+### Class member Function: Initializer
+- Set up hyper-parameters, including condtional dimension, noise dimension, batch size and so forth.
+- Declare and define all the model variables. All the discriminator parameters are included in the list self.theta_D and all the generator parameters are included in the list self.theta_G.
+```python
+class DCGAN(object):
+  def __init__(self, y_dim=None):
+  
+    # hyper parameters  
+    self.y_dim = y_dim # conditional gan or not
+    self.batch_size = 100
+    self.z_dim = z_dim # input noise dimension
+
+    # define parameters of discriminators
+    self.D_W0 = pd.Variable(shape=[3,3, 1, 128], data=pd.gaussian_normal_randomizer())
+    self.D_b0 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.D_W1 = pd.Variable(shape=[784, 128], data=pd.gaussian_normal_randomizer())
+    self.D_b1 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.D_W2 = pd.Varialble(np.random.rand(128, 1))
+    self.D_b2 = pd.Variable(np.zeros(128))
+    self.theta_D = [self.D_W0, self.D_b0, self.D_W1, self.D_b1, self.D_W2, self.D_b2]
+
+    # define parameters of generators
+    self.G_W0 = pd.Variable(shape=[784, 128], data=pd.gaussian_normal_randomizer())
+    self.G_b0 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.G_W1 = pd.Variable(shape=[784, 128], data=pd.gaussian_normal_randomizer())
+    self.G_b1 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.G_W2 = pd.Varialble(np.random.rand(128, 1))
+    self.G_b2 = pd.Variable(np.zeros(128))
+    self.theta_G = [self.G_W0, self.G_b0, self.G_W1, self.G_b1, self.G_W2, self.G_b2]
+```
+
+### Class member Function: Generator
+- Given a noisy input z, returns a fake image.
+- Concatenation, batch-norm, FC operations required;
+- Deconv layer required, which is missing now...
+```python
+class DCGAN(object):
+  def generator(self, z, y = None):
+    # input z: the random noise
+    # input y: input data label (optional)
+    # output G_im: generated fake images
+    
+    if not self.y_dim:
+      z = pd.layer.concat(1, [z, y])
+      
+    G_h0 = pd.layer.fc(z, self.G_w0, self.G_b0)
+    G_h0_bn = pd.layer.batch_norm(G_h0)
+    G_h0_relu = pd.layer.relu(G_h0_bn)
+    
+    G_h1 = pd.layer.deconv(G_h0_relu, self.G_w1, self.G_b1)
+    G_h1_bn = pd.layer.batch_norm(G_h1)
+    G_h1_relu = pd.layer.relu(G_h1_bn)
+    
+    G_h2 = pd.layer.deconv(G_h1_relu, self.G_W2, self.G_b2))
+    G_im = pd.layer.tanh(G_im)
+    return G_im
+```
+
+### Class member function: Discriminator
+- Given a noisy input z, returns a fake image.
+- Concatenation, Convolution, batch-norm, FC, Leaky-ReLU operations required;
+```python
+class DCGAN(object):
+  def discriminator(self, image):
+    # input image: either generated images or real ones
+    # output D_h2: binary logit of the label
+
+    D_h0 = pd.layer.conv2d(image, w=self.D_w0, b=self.D_b0)
+    D_h0_bn = pd.layer.batchnorm(h0)
+    D_h0_relu = pd.layer.lrelu(h0_bn)
+    
+    D_h1 = pd.layer.conv2d(D_h0_relu, w=self.D_w1, b=self.D_b1)
+    D_h1_bn = pd.layer.batchnorm(D_h1)
+    D_h1_relu = pd.layer.lrelu(D_h1_bn)
+    
+    D_h2 = pd.layer.fc(D_h1_relu, w=self.D_w2, b=self.D_b2)
+    return D_h2
+```
+
+### Class member function: Build the model
+- Define data readers as placeholders to hold the data;
+- Build generator and discriminators;
+- Define two training losses for discriminator and generator, respectively. 
+If we have execution dependency engine to back-trace all tensors, the module building our GAN model will be like this:
+```python
+class DCGAN(object):
+  def build_model(self):
+    if self.y_dim:
+        self.y = pd.data(pd.float32, [self.batch_size, self.y_dim])
+    self.images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    self.faked_images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    self.z = pd.data(tf.float32, [None, self.z_size])
+    
+    # step 1: generate images by generator, classify real/fake images with discriminator
+    if self.y_dim: # if conditional GAN, includes label
+        self.G = self.generator(self.z, self.y)
+        self.D_t = self.discriminator(self.images)
+        # generated fake images
+        self.sampled = self.sampler(self.z, self.y)
+        self.D_f = self.discriminator(self.G)
+    else: # original version of GAN
+        self.G = self.generator(self.z)
+        self.D_t = self.discriminator(self.images)
+        # generate fake images
+        self.sampled = self.sampler(self.z)
+        self.D_f = self.discriminator(self.images)
+    
+    # step 2: define the two losses
+    self.d_loss_real = pd.reduce_mean(pd.cross_entropy(self.D_t, np.ones(self.batch_size))
+    self.d_loss_fake = pd.reduce_mean(pd.cross_entropy(self.D_f, np.zeros(self.batch_size))
+    self.d_loss = self.d_loss_real + self.d_loss_fake
+    
+    self.g_loss = pd.reduce_mean(pd.cross_entropy(self.D_f, np.ones(self.batch_szie))
+```
+
+If we do not have dependency engine but blocks, the module building our GAN model will be like this:
+```python
+class DCGAN(object):
+  def build_model(self, default_block):
+    # input data in the default block
+    if self.y_dim:
+        self.y = pd.data(pd.float32, [self.batch_size, self.y_dim])
+    self.images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    # self.faked_images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    self.z = pd.data(tf.float32, [None, self.z_size])
+
+    # step 1: generate images by generator, classify real/fake images with discriminator
+    with pd.default_block().g_block():
+      if self.y_dim: # if conditional GAN, includes label
+        self.G = self.generator(self.z, self.y)
+        self.D_g = self.discriminator(self.G, self.y)
+      else: # original version of GAN
+        self.G = self.generator(self.z)
+        self.D_g = self.discriminator(self.G, self.y)
+      self.g_loss = pd.reduce_mean(pd.cross_entropy(self.D_g, np.ones(self.batch_szie))
+    
+    with pd.default_block().d_block():
+      if self.y_dim: # if conditional GAN, includes label
+        self.D_t = self.discriminator(self.images, self.y)
+        self.D_f = self.discriminator(self.G, self.y)
+      else: # original version of GAN
+        self.D_t = self.discriminator(self.images)
+        self.D_f = self.discriminator(self.G)
+
+      # step 2: define the two losses
+      self.d_loss_real = pd.reduce_mean(pd.cross_entropy(self.D_t, np.ones(self.batch_size))
+      self.d_loss_fake = pd.reduce_mean(pd.cross_entropy(self.D_f, np.zeros(self.batch_size))
+      self.d_loss = self.d_loss_real + self.d_loss_fake
+```
+Some small confusion and problems with this design:
+- D\_g and D\_f are actually the same thing, but has to be written twice; i.e., if we want to run two sub-graphs conceptually, the same codes have to be written twice if they are shared by the graph.
+- Requires ability to create a block anytime, rather than in if-else or rnn only;
+
+## Main function for the demo:
+Generally, the user of GAN just need to the following things:
+- Define an object as DCGAN class;
+- Build the DCGAN model;
+- Specify two optimizers for two different losses with respect to different parameters.
+```python
+# pd for short, should be more concise.
+from paddle.v2 as pd
+import numpy as np
+import logging
+
+if __name__ == "__main__":
+    # dcgan class in the default graph/block
+    # if we use dependency engine as tensorflow
+    # the codes, will be slightly different like:
+    # dcgan = DCGAN()
+    # dcgan.build_model()
+    with pd.block() as def_block:
+      dcgan = DCGAN()
+      dcgan.build_model(def_block)
+
+    # load mnist data
+    data_X, data_y = self.load_mnist()
+    
+    # Two subgraphs required!!!
+    with pd.block().d_block():
+      d_optim = pd.train.Adam(lr = .001, beta= .1)
+      d_step = d_optim.minimize(dcgan.d_loss, dcgan.theta_D)
+    with pd.block.g_block():
+      g_optim = pd.train.Adam(lr = .001, beta= .1)
+      g_step = pd.minimize(dcgan.g_loss, dcgan.theta_G)
+
+    # executor
+    sess = pd.executor()
+    
+    # training
+    for epoch in xrange(10000):
+      for batch_id in range(N / batch_size):
+        idx = ...
+        # sample a batch
+        batch_im, batch_label = data_X[idx:idx+batch_size], data_y[idx:idx+batch_size]
+        # sample z
+        batch_z = np.random.uniform(-1., 1., [batch_size, z_dim])
+
+        if batch_id % 2 == 0:
+          sess.run(d_step, 
+                   feed_dict = {dcgan.images: batch_im,
+                                dcgan.y: batch_label,
+                                dcgan.z: batch_z})
+        else:
+          sess.run(g_step,
+                   feed_dict = {dcgan.z: batch_z})
+```
+
+# More thinking about dependency engine v.s. block design:
+- What if we just want to run an intermediate result? Do we need to run the whole block/graph?
+- Should we call eval() to get the fake images in the first stage? And then train the discriminator in the second stage?

develop/doc/design/gan_api.html

+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>Design for GAN &mdash; PaddlePaddle  documentation</title>
+  
+
+  
+  
+
+  
+
+  
+  
+    
+
+  
+
+  
+  
+    <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+  
+
+  
+  
+        <link rel="index" title="Index"
+              href="../genindex.html"/>
+        <link rel="search" title="Search" href="../search.html"/>
+    <link rel="top" title="PaddlePaddle  documentation" href="../index.html"/> 
+
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/css/perfect-scrollbar.min.css" type="text/css" />
+  <link rel="stylesheet" href="../_static/css/override.css" type="text/css" />
+  <script>
+  var _hmt = _hmt || [];
+  (function() {
+    var hm = document.createElement("script");
+    hm.src = "//hm.baidu.com/hm.js?b9a314ab40d04d805655aab1deee08ba";
+    var s = document.getElementsByTagName("script")[0]; 
+    s.parentNode.insertBefore(hm, s);
+  })();
+  </script>
+
+  
+
+  
+  <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+  
+  <header class="site-header">
+    <div class="site-logo">
+      <a href="/"><img src="../_static/images/PP_w.png"></a>
+    </div>
+    <div class="site-nav-links">
+      <div class="site-menu">
+        <a class="fork-on-github" href="https://github.com/PaddlePaddle/Paddle" target="_blank"><i class="fa fa-github"></i>Fork me on Github</a>
+        <div class="language-switcher dropdown">
+          <a type="button" data-toggle="dropdown">
+            <span>English</span>
+            <i class="fa fa-angle-up"></i>
+            <i class="fa fa-angle-down"></i>
+          </a>
+          <ul class="dropdown-menu">
+            <li><a href="/doc_cn">中文</a></li>
+            <li><a href="/doc">English</a></li>
+          </ul>
+        </div>
+        <ul class="site-page-links">
+          <li><a href="/">Home</a></li>
+        </ul>
+      </div>
+      <div class="doc-module">
+        
+        <ul>
+<li class="toctree-l1"><a class="reference internal" href="../getstarted/index_en.html">GET STARTED</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../howto/index_en.html">HOW TO</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api/index_en.html">API</a></li>
+</ul>
+
+        
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>        
+      </div>
+    </div>
+  </header>
+  
+  <div class="main-content-wrap">
+
+    
+    <nav class="doc-menu-vertical" role="navigation">
+        
+          
+          <ul>
+<li class="toctree-l1"><a class="reference internal" href="../getstarted/index_en.html">GET STARTED</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../getstarted/build_and_install/index_en.html">Install and Build</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/docker_install_en.html">PaddlePaddle in Docker Containers</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/build_from_source_en.html">Installing from Sources</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../howto/index_en.html">HOW TO</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/cmd_parameter/index_en.html">Set Command-line Parameters</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/use_case_en.html">Use Case</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/arguments_en.html">Argument Outline</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/detail_introduction_en.html">Detail Description</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/cluster/cluster_train_en.html">Run Distributed Training</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_en.html">Paddle On Kubernetes</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_aws_en.html">Distributed PaddlePaddle Training on AWS with Kubernetes</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/dev/build_en.html">Build PaddlePaddle from Source Code and Run Unit Test</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/dev/new_layer_en.html">Write New Layers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/dev/contribute_to_paddle_en.html">Contribute Code</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/deep_model/rnn/index_en.html">RNN Models</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/rnn_config_en.html">RNN Configuration</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/optimization/gpu_profiling_en.html">Tune GPU Performance</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../api/index_en.html">API</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../api/v2/model_configs.html">Model Configuration</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/activation.html">Activation</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/layer.html">Layers</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/evaluators.html">Evaluators</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/optimizer.html">Optimizer</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/pooling.html">Pooling</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/networks.html">Networks</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/attr.html">Parameter Attribute</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../api/v2/data.html">Data Reader Interface and DataSets</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../api/v2/run_logic.html">Training and Inference</a></li>
+</ul>
+</li>
+</ul>
+
+        
+    </nav>
+    
+    <section class="doc-content-wrap">
+
+      
+
+ 
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+  <ul class="wy-breadcrumbs">
+      
+    <li>Design for GAN</li>
+  </ul>
+</div>
+      
+      <div class="wy-nav-content" id="doc-content">
+        <div class="rst-content">
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <div class="section" id="design-for-gan">
+<span id="design-for-gan"></span><h1>Design for GAN<a class="headerlink" href="#design-for-gan" title="Permalink to this headline">¶</a></h1>
+<p>GAN (General Adversarial Net [https://arxiv.org/abs/1406.2661]) is an important model for unsupervised learning and widely used in many areas.</p>
+<p>It applies several important concepts in machine learning system design, including building and running subgraphs, dependency tracing, different optimizers in one executor and so forth.</p>
+<p>In our GAN design, we wrap it as a user-friendly easily customized python API to design different models. We take the conditional DC-GAN (Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [https://arxiv.org/abs/1511.06434]) as an example due to its good performance on image generation.</p>
+<p align="center">
+<img src="./test.dot.png" width = "35%" align="center"/><br/>
+Figure 1. The overall running logic of GAN. The black solid arrows indicate the forward pass; the green dashed arrows indicate the backward pass of generator training; the red dashed arrows indicate the backward pass of the discriminator training. The BP pass of the green (red) arrow should only update the parameters in the green (red) boxes. The diamonds indicate the data providers. d\_loss and g\_loss marked in red and green are the two targets we would like to run.
+</p><p>The operators, layers and functions required/optional to build a GAN demo is summarized in https://github.com/PaddlePaddle/Paddle/issues/4563.</p>
+<p align="center">
+<img src="./dcgan.png" width = "90%" align="center"/><br/>
+Figure 2. Photo borrowed from the original DC-GAN paper.
+</p><div class="section" id="the-conditional-gan-might-be-a-class">
+<span id="the-conditional-gan-might-be-a-class"></span><h2>The Conditional-GAN might be a class.<a class="headerlink" href="#the-conditional-gan-might-be-a-class" title="Permalink to this headline">¶</a></h2>
+<p>This design we adopt the popular open source design in https://github.com/carpedm20/DCGAN-tensorflow and https://github.com/rajathkmp/DCGAN. It contains following data structure:</p>
+<ul class="simple">
+<li>DCGAN(object): which contains everything required to build a GAN model. It provides following member functions methods as API:</li>
+<li><strong>init</strong>(...): Initialize hyper-parameters (like conv dimension and so forth), and declare model parameters of discriminator and generator as well.</li>
+<li>generator(z, y=None): Generate a fake image from input noise z. If the label y is provided, the conditional GAN model will be chosen.
+Returns a generated image.</li>
+<li>discriminator(image):
+Given an image, decide if it is from a real source or a fake one.
+Returns a 0/1 binary label.</li>
+<li>build_model(self):
+build the whole GAN model, define training loss for both generator and discrimator.</li>
+</ul>
+</div>
+<div class="section" id="discussion-on-engine-functions-required-to-build-gan">
+<span id="discussion-on-engine-functions-required-to-build-gan"></span><h2>Discussion on Engine Functions required to build GAN<a class="headerlink" href="#discussion-on-engine-functions-required-to-build-gan" title="Permalink to this headline">¶</a></h2>
+<ul class="simple">
+<li>Trace the tensor and variable dependency in the engine executor. (Very critical, otherwise GAN can&#8217;be be trained correctly)</li>
+<li>Different optimizers responsible for optimizing different loss.</li>
+</ul>
+<p>To be more detailed, we introduce our design of DCGAN as following:</p>
+<div class="section" id="class-member-function-initializer">
+<span id="class-member-function-initializer"></span><h3>Class member Function: Initializer<a class="headerlink" href="#class-member-function-initializer" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li>Set up hyper-parameters, including condtional dimension, noise dimension, batch size and so forth.</li>
+<li>Declare and define all the model variables. All the discriminator parameters are included in the list self.theta_D and all the generator parameters are included in the list self.theta_G.</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">y_dim</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+  
+    <span class="c1"># hyper parameters  </span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span> <span class="o">=</span> <span class="n">y_dim</span> <span class="c1"># conditional gan or not</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="mi">100</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">z_dim</span> <span class="o">=</span> <span class="n">z_dim</span> <span class="c1"># input noise dimension</span>
+
+    <span class="c1"># define parameters of discriminators</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_W0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_b0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_W1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_b1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_W2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Varialble</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_b2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">theta_D</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">D_W0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_b0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_W1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_b1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_W2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_b2</span><span class="p">]</span>
+
+    <span class="c1"># define parameters of generators</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_W0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_b0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_W1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_b1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_W2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Varialble</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_b2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">theta_G</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">G_W0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_W1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_W2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b2</span><span class="p">]</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="class-member-function-generator">
+<span id="class-member-function-generator"></span><h3>Class member Function: Generator<a class="headerlink" href="#class-member-function-generator" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li>Given a noisy input z, returns a fake image.</li>
+<li>Concatenation, batch-norm, FC operations required;</li>
+<li>Deconv layer required, which is missing now...</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">generator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">z</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="bp">None</span><span class="p">):</span>
+    <span class="c1"># input z: the random noise</span>
+    <span class="c1"># input y: input data label (optional)</span>
+    <span class="c1"># output G_im: generated fake images</span>
+    
+    <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span>
+      <span class="n">z</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">concat</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="n">z</span><span class="p">,</span> <span class="n">y</span><span class="p">])</span>
+      
+    <span class="n">G_h0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_w0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b0</span><span class="p">)</span>
+    <span class="n">G_h0_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batch_norm</span><span class="p">(</span><span class="n">G_h0</span><span class="p">)</span>
+    <span class="n">G_h0_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">G_h0_bn</span><span class="p">)</span>
+    
+    <span class="n">G_h1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">deconv</span><span class="p">(</span><span class="n">G_h0_relu</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_w1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b1</span><span class="p">)</span>
+    <span class="n">G_h1_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batch_norm</span><span class="p">(</span><span class="n">G_h1</span><span class="p">)</span>
+    <span class="n">G_h1_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">G_h1_bn</span><span class="p">)</span>
+    
+    <span class="n">G_h2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">deconv</span><span class="p">(</span><span class="n">G_h1_relu</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_W2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b2</span><span class="p">))</span>
+    <span class="n">G_im</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">G_im</span><span class="p">)</span>
+    <span class="k">return</span> <span class="n">G_im</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="class-member-function-discriminator">
+<span id="class-member-function-discriminator"></span><h3>Class member function: Discriminator<a class="headerlink" href="#class-member-function-discriminator" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li>Given a noisy input z, returns a fake image.</li>
+<li>Concatenation, Convolution, batch-norm, FC, Leaky-ReLU operations required;</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image</span><span class="p">):</span>
+    <span class="c1"># input image: either generated images or real ones</span>
+    <span class="c1"># output D_h2: binary logit of the label</span>
+
+    <span class="n">D_h0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">conv2d</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_w0</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_b0</span><span class="p">)</span>
+    <span class="n">D_h0_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batchnorm</span><span class="p">(</span><span class="n">h0</span><span class="p">)</span>
+    <span class="n">D_h0_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">lrelu</span><span class="p">(</span><span class="n">h0_bn</span><span class="p">)</span>
+    
+    <span class="n">D_h1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">conv2d</span><span class="p">(</span><span class="n">D_h0_relu</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_w1</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_b1</span><span class="p">)</span>
+    <span class="n">D_h1_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batchnorm</span><span class="p">(</span><span class="n">D_h1</span><span class="p">)</span>
+    <span class="n">D_h1_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">lrelu</span><span class="p">(</span><span class="n">D_h1_bn</span><span class="p">)</span>
+    
+    <span class="n">D_h2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">D_h1_relu</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_w2</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_b2</span><span class="p">)</span>
+    <span class="k">return</span> <span class="n">D_h2</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="class-member-function-build-the-model">
+<span id="class-member-function-build-the-model"></span><h3>Class member function: Build the model<a class="headerlink" href="#class-member-function-build-the-model" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li>Define data readers as placeholders to hold the data;</li>
+<li>Build generator and discriminators;</li>
+<li>Define two training losses for discriminator and generator, respectively.
+If we have execution dependency engine to back-trace all tensors, the module building our GAN model will be like this:</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">build_model</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">images</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">faked_images</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">z_size</span><span class="p">])</span>
+    
+    <span class="c1"># step 1: generate images by generator, classify real/fake images with discriminator</span>
+    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span> <span class="c1"># if conditional GAN, includes label</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+        <span class="c1"># generated fake images</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">sampled</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sampler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">)</span>
+    <span class="k">else</span><span class="p">:</span> <span class="c1"># original version of GAN</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+        <span class="c1"># generate fake images</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">sampled</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sampler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+    
+    <span class="c1"># step 2: define the two losses</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_t</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_f</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">d_loss</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span>
+    
+    <span class="bp">self</span><span class="o">.</span><span class="n">g_loss</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_f</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_szie</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>If we do not have dependency engine but blocks, the module building our GAN model will be like this:</p>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">build_model</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">default_block</span><span class="p">):</span>
+    <span class="c1"># input data in the default block</span>
+    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">images</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">])</span>
+    <span class="c1"># self.faked_images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">z_size</span><span class="p">])</span>
+
+    <span class="c1"># step 1: generate images by generator, classify real/fake images with discriminator</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">default_block</span><span class="p">()</span><span class="o">.</span><span class="n">g_block</span><span class="p">():</span>
+      <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span> <span class="c1"># if conditional GAN, includes label</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_g</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+      <span class="k">else</span><span class="p">:</span> <span class="c1"># original version of GAN</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_g</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">g_loss</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_g</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_szie</span><span class="p">))</span>
+    
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">default_block</span><span class="p">()</span><span class="o">.</span><span class="n">d_block</span><span class="p">():</span>
+      <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span> <span class="c1"># if conditional GAN, includes label</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+      <span class="k">else</span><span class="p">:</span> <span class="c1"># original version of GAN</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">)</span>
+
+      <span class="c1"># step 2: define the two losses</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_t</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_f</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">d_loss</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span>
+</pre></div>
+</div>
+<p>Some small confusion and problems with this design:</p>
+<ul class="simple">
+<li>D_g and D_f are actually the same thing, but has to be written twice; i.e., if we want to run two sub-graphs conceptually, the same codes have to be written twice if they are shared by the graph.</li>
+<li>Requires ability to create a block anytime, rather than in if-else or rnn only;</li>
+</ul>
+</div>
+</div>
+<div class="section" id="main-function-for-the-demo">
+<span id="main-function-for-the-demo"></span><h2>Main function for the demo:<a class="headerlink" href="#main-function-for-the-demo" title="Permalink to this headline">¶</a></h2>
+<p>Generally, the user of GAN just need to the following things:</p>
+<ul class="simple">
+<li>Define an object as DCGAN class;</li>
+<li>Build the DCGAN model;</li>
+<li>Specify two optimizers for two different losses with respect to different parameters.</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># pd for short, should be more concise.</span>
+<span class="kn">from</span> <span class="nn">paddle.v2</span> <span class="nn">as</span> <span class="nn">pd</span>
+<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span>
+    <span class="c1"># dcgan class in the default graph/block</span>
+    <span class="c1"># if we use dependency engine as tensorflow</span>
+    <span class="c1"># the codes, will be slightly different like:</span>
+    <span class="c1"># dcgan = DCGAN()</span>
+    <span class="c1"># dcgan.build_model()</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">block</span><span class="p">()</span> <span class="k">as</span> <span class="n">def_block</span><span class="p">:</span>
+      <span class="n">dcgan</span> <span class="o">=</span> <span class="n">DCGAN</span><span class="p">()</span>
+      <span class="n">dcgan</span><span class="o">.</span><span class="n">build_model</span><span class="p">(</span><span class="n">def_block</span><span class="p">)</span>
+
+    <span class="c1"># load mnist data</span>
+    <span class="n">data_X</span><span class="p">,</span> <span class="n">data_y</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">load_mnist</span><span class="p">()</span>
+    
+    <span class="c1"># Two subgraphs required!!!</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">block</span><span class="p">()</span><span class="o">.</span><span class="n">d_block</span><span class="p">():</span>
+      <span class="n">d_optim</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">lr</span> <span class="o">=</span> <span class="o">.</span><span class="mo">001</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span> <span class="o">.</span><span class="mi">1</span><span class="p">)</span>
+      <span class="n">d_step</span> <span class="o">=</span> <span class="n">d_optim</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">dcgan</span><span class="o">.</span><span class="n">d_loss</span><span class="p">,</span> <span class="n">dcgan</span><span class="o">.</span><span class="n">theta_D</span><span class="p">)</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">block</span><span class="o">.</span><span class="n">g_block</span><span class="p">():</span>
+      <span class="n">g_optim</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">lr</span> <span class="o">=</span> <span class="o">.</span><span class="mo">001</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span> <span class="o">.</span><span class="mi">1</span><span class="p">)</span>
+      <span class="n">g_step</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">dcgan</span><span class="o">.</span><span class="n">g_loss</span><span class="p">,</span> <span class="n">dcgan</span><span class="o">.</span><span class="n">theta_G</span><span class="p">)</span>
+
+    <span class="c1"># executor</span>
+    <span class="n">sess</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">executor</span><span class="p">()</span>
+    
+    <span class="c1"># training</span>
+    <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">10000</span><span class="p">):</span>
+      <span class="k">for</span> <span class="n">batch_id</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span> <span class="o">/</span> <span class="n">batch_size</span><span class="p">):</span>
+        <span class="n">idx</span> <span class="o">=</span> <span class="o">...</span>
+        <span class="c1"># sample a batch</span>
+        <span class="n">batch_im</span><span class="p">,</span> <span class="n">batch_label</span> <span class="o">=</span> <span class="n">data_X</span><span class="p">[</span><span class="n">idx</span><span class="p">:</span><span class="n">idx</span><span class="o">+</span><span class="n">batch_size</span><span class="p">],</span> <span class="n">data_y</span><span class="p">[</span><span class="n">idx</span><span class="p">:</span><span class="n">idx</span><span class="o">+</span><span class="n">batch_size</span><span class="p">]</span>
+        <span class="c1"># sample z</span>
+        <span class="n">batch_z</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="o">-</span><span class="mf">1.</span><span class="p">,</span> <span class="mf">1.</span><span class="p">,</span> <span class="p">[</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">z_dim</span><span class="p">])</span>
+
+        <span class="k">if</span> <span class="n">batch_id</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+          <span class="n">sess</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">d_step</span><span class="p">,</span> 
+                   <span class="n">feed_dict</span> <span class="o">=</span> <span class="p">{</span><span class="n">dcgan</span><span class="o">.</span><span class="n">images</span><span class="p">:</span> <span class="n">batch_im</span><span class="p">,</span>
+                                <span class="n">dcgan</span><span class="o">.</span><span class="n">y</span><span class="p">:</span> <span class="n">batch_label</span><span class="p">,</span>
+                                <span class="n">dcgan</span><span class="o">.</span><span class="n">z</span><span class="p">:</span> <span class="n">batch_z</span><span class="p">})</span>
+        <span class="k">else</span><span class="p">:</span>
+          <span class="n">sess</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">g_step</span><span class="p">,</span>
+                   <span class="n">feed_dict</span> <span class="o">=</span> <span class="p">{</span><span class="n">dcgan</span><span class="o">.</span><span class="n">z</span><span class="p">:</span> <span class="n">batch_z</span><span class="p">})</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="more-thinking-about-dependency-engine-v-s-block-design">
+<span id="more-thinking-about-dependency-engine-v-s-block-design"></span><h1>More thinking about dependency engine v.s. block design:<a class="headerlink" href="#more-thinking-about-dependency-engine-v-s-block-design" title="Permalink to this headline">¶</a></h1>
+<ul class="simple">
+<li>What if we just want to run an intermediate result? Do we need to run the whole block/graph?</li>
+<li>Should we call eval() to get the fake images in the first stage? And then train the discriminator in the second stage?</li>
+</ul>
+</div>
+
+
+           </div>
+          </div>
+          <footer>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2016, PaddlePaddle developers.
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  
+
+    <script type="text/javascript">
+        var DOCUMENTATION_OPTIONS = {
+            URL_ROOT:'../',
+            VERSION:'',
+            COLLAPSE_INDEX:false,
+            FILE_SUFFIX:'.html',
+            HAS_SOURCE:  true,
+            SOURCELINK_SUFFIX: ".txt",
+        };
+    </script>
+      <script type="text/javascript" src="../_static/jquery.js"></script>
+      <script type="text/javascript" src="../_static/underscore.js"></script>
+      <script type="text/javascript" src="../_static/doctools.js"></script>
+      <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
+       
+  
+
+  
+  
+    <script type="text/javascript" src="../_static/js/theme.js"></script>
+  
+  
+  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/js/perfect-scrollbar.jquery.min.js"></script>
+  <script src="../_static/js/paddle_doc_init.js"></script> 
+
+</body>
+</html>
\ No newline at end of file

develop/doc_cn/_sources/design/gan_api.md.txt

+# Design for GAN
+
+GAN (General Adversarial Net [https://arxiv.org/abs/1406.2661]) is an important model for unsupervised learning and widely used in many areas. 
+
+It applies several important concepts in machine learning system design, including building and running subgraphs, dependency tracing, different optimizers in one executor and so forth.
+
+In our GAN design, we wrap it as a user-friendly easily customized python API to design different models. We take the conditional DC-GAN (Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [https://arxiv.org/abs/1511.06434]) as an example due to its good performance on image generation.
+
+<p align="center">
+<img src="./test.dot.png" width = "35%" align="center"/><br/>
+Figure 1. The overall running logic of GAN. The black solid arrows indicate the forward pass; the green dashed arrows indicate the backward pass of generator training; the red dashed arrows indicate the backward pass of the discriminator training. The BP pass of the green (red) arrow should only update the parameters in the green (red) boxes. The diamonds indicate the data providers. d\_loss and g\_loss marked in red and green are the two targets we would like to run.
+</p>
+
+The operators, layers and functions required/optional to build a GAN demo is summarized in https://github.com/PaddlePaddle/Paddle/issues/4563.
+
+<p align="center">
+<img src="./dcgan.png" width = "90%" align="center"/><br/>
+Figure 2. Photo borrowed from the original DC-GAN paper.
+</p>
+
+## The Conditional-GAN might be a class. 
+This design we adopt the popular open source design in https://github.com/carpedm20/DCGAN-tensorflow and https://github.com/rajathkmp/DCGAN. It contains following data structure:
+
+- DCGAN(object): which contains everything required to build a GAN model. It provides following member functions methods as API:
+
+- __init__(...): Initialize hyper-parameters (like conv dimension and so forth), and declare model parameters of discriminator and generator as well.
+
+- generator(z, y=None): Generate a fake image from input noise z. If the label y is provided, the conditional GAN model will be chosen.
+Returns a generated image.
+
+- discriminator(image):
+Given an image, decide if it is from a real source or a fake one. 
+Returns a 0/1 binary label.
+
+- build_model(self):
+build the whole GAN model, define training loss for both generator and discrimator.
+
+## Discussion on Engine Functions required to build GAN
+- Trace the tensor and variable dependency in the engine executor. (Very critical, otherwise GAN can'be be trained correctly)
+- Different optimizers responsible for optimizing different loss.
+
+To be more detailed, we introduce our design of DCGAN as following:
+
+### Class member Function: Initializer
+- Set up hyper-parameters, including condtional dimension, noise dimension, batch size and so forth.
+- Declare and define all the model variables. All the discriminator parameters are included in the list self.theta_D and all the generator parameters are included in the list self.theta_G.
+```python
+class DCGAN(object):
+  def __init__(self, y_dim=None):
+  
+    # hyper parameters  
+    self.y_dim = y_dim # conditional gan or not
+    self.batch_size = 100
+    self.z_dim = z_dim # input noise dimension
+
+    # define parameters of discriminators
+    self.D_W0 = pd.Variable(shape=[3,3, 1, 128], data=pd.gaussian_normal_randomizer())
+    self.D_b0 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.D_W1 = pd.Variable(shape=[784, 128], data=pd.gaussian_normal_randomizer())
+    self.D_b1 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.D_W2 = pd.Varialble(np.random.rand(128, 1))
+    self.D_b2 = pd.Variable(np.zeros(128))
+    self.theta_D = [self.D_W0, self.D_b0, self.D_W1, self.D_b1, self.D_W2, self.D_b2]
+
+    # define parameters of generators
+    self.G_W0 = pd.Variable(shape=[784, 128], data=pd.gaussian_normal_randomizer())
+    self.G_b0 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.G_W1 = pd.Variable(shape=[784, 128], data=pd.gaussian_normal_randomizer())
+    self.G_b1 = pd.Variable(np.zeros(128)) # variable also support initialization using a  numpy data
+    self.G_W2 = pd.Varialble(np.random.rand(128, 1))
+    self.G_b2 = pd.Variable(np.zeros(128))
+    self.theta_G = [self.G_W0, self.G_b0, self.G_W1, self.G_b1, self.G_W2, self.G_b2]
+```
+
+### Class member Function: Generator
+- Given a noisy input z, returns a fake image.
+- Concatenation, batch-norm, FC operations required;
+- Deconv layer required, which is missing now...
+```python
+class DCGAN(object):
+  def generator(self, z, y = None):
+    # input z: the random noise
+    # input y: input data label (optional)
+    # output G_im: generated fake images
+    
+    if not self.y_dim:
+      z = pd.layer.concat(1, [z, y])
+      
+    G_h0 = pd.layer.fc(z, self.G_w0, self.G_b0)
+    G_h0_bn = pd.layer.batch_norm(G_h0)
+    G_h0_relu = pd.layer.relu(G_h0_bn)
+    
+    G_h1 = pd.layer.deconv(G_h0_relu, self.G_w1, self.G_b1)
+    G_h1_bn = pd.layer.batch_norm(G_h1)
+    G_h1_relu = pd.layer.relu(G_h1_bn)
+    
+    G_h2 = pd.layer.deconv(G_h1_relu, self.G_W2, self.G_b2))
+    G_im = pd.layer.tanh(G_im)
+    return G_im
+```
+
+### Class member function: Discriminator
+- Given a noisy input z, returns a fake image.
+- Concatenation, Convolution, batch-norm, FC, Leaky-ReLU operations required;
+```python
+class DCGAN(object):
+  def discriminator(self, image):
+    # input image: either generated images or real ones
+    # output D_h2: binary logit of the label
+
+    D_h0 = pd.layer.conv2d(image, w=self.D_w0, b=self.D_b0)
+    D_h0_bn = pd.layer.batchnorm(h0)
+    D_h0_relu = pd.layer.lrelu(h0_bn)
+    
+    D_h1 = pd.layer.conv2d(D_h0_relu, w=self.D_w1, b=self.D_b1)
+    D_h1_bn = pd.layer.batchnorm(D_h1)
+    D_h1_relu = pd.layer.lrelu(D_h1_bn)
+    
+    D_h2 = pd.layer.fc(D_h1_relu, w=self.D_w2, b=self.D_b2)
+    return D_h2
+```
+
+### Class member function: Build the model
+- Define data readers as placeholders to hold the data;
+- Build generator and discriminators;
+- Define two training losses for discriminator and generator, respectively. 
+If we have execution dependency engine to back-trace all tensors, the module building our GAN model will be like this:
+```python
+class DCGAN(object):
+  def build_model(self):
+    if self.y_dim:
+        self.y = pd.data(pd.float32, [self.batch_size, self.y_dim])
+    self.images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    self.faked_images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    self.z = pd.data(tf.float32, [None, self.z_size])
+    
+    # step 1: generate images by generator, classify real/fake images with discriminator
+    if self.y_dim: # if conditional GAN, includes label
+        self.G = self.generator(self.z, self.y)
+        self.D_t = self.discriminator(self.images)
+        # generated fake images
+        self.sampled = self.sampler(self.z, self.y)
+        self.D_f = self.discriminator(self.G)
+    else: # original version of GAN
+        self.G = self.generator(self.z)
+        self.D_t = self.discriminator(self.images)
+        # generate fake images
+        self.sampled = self.sampler(self.z)
+        self.D_f = self.discriminator(self.images)
+    
+    # step 2: define the two losses
+    self.d_loss_real = pd.reduce_mean(pd.cross_entropy(self.D_t, np.ones(self.batch_size))
+    self.d_loss_fake = pd.reduce_mean(pd.cross_entropy(self.D_f, np.zeros(self.batch_size))
+    self.d_loss = self.d_loss_real + self.d_loss_fake
+    
+    self.g_loss = pd.reduce_mean(pd.cross_entropy(self.D_f, np.ones(self.batch_szie))
+```
+
+If we do not have dependency engine but blocks, the module building our GAN model will be like this:
+```python
+class DCGAN(object):
+  def build_model(self, default_block):
+    # input data in the default block
+    if self.y_dim:
+        self.y = pd.data(pd.float32, [self.batch_size, self.y_dim])
+    self.images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    # self.faked_images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])
+    self.z = pd.data(tf.float32, [None, self.z_size])
+
+    # step 1: generate images by generator, classify real/fake images with discriminator
+    with pd.default_block().g_block():
+      if self.y_dim: # if conditional GAN, includes label
+        self.G = self.generator(self.z, self.y)
+        self.D_g = self.discriminator(self.G, self.y)
+      else: # original version of GAN
+        self.G = self.generator(self.z)
+        self.D_g = self.discriminator(self.G, self.y)
+      self.g_loss = pd.reduce_mean(pd.cross_entropy(self.D_g, np.ones(self.batch_szie))
+    
+    with pd.default_block().d_block():
+      if self.y_dim: # if conditional GAN, includes label
+        self.D_t = self.discriminator(self.images, self.y)
+        self.D_f = self.discriminator(self.G, self.y)
+      else: # original version of GAN
+        self.D_t = self.discriminator(self.images)
+        self.D_f = self.discriminator(self.G)
+
+      # step 2: define the two losses
+      self.d_loss_real = pd.reduce_mean(pd.cross_entropy(self.D_t, np.ones(self.batch_size))
+      self.d_loss_fake = pd.reduce_mean(pd.cross_entropy(self.D_f, np.zeros(self.batch_size))
+      self.d_loss = self.d_loss_real + self.d_loss_fake
+```
+Some small confusion and problems with this design:
+- D\_g and D\_f are actually the same thing, but has to be written twice; i.e., if we want to run two sub-graphs conceptually, the same codes have to be written twice if they are shared by the graph.
+- Requires ability to create a block anytime, rather than in if-else or rnn only;
+
+## Main function for the demo:
+Generally, the user of GAN just need to the following things:
+- Define an object as DCGAN class;
+- Build the DCGAN model;
+- Specify two optimizers for two different losses with respect to different parameters.
+```python
+# pd for short, should be more concise.
+from paddle.v2 as pd
+import numpy as np
+import logging
+
+if __name__ == "__main__":
+    # dcgan class in the default graph/block
+    # if we use dependency engine as tensorflow
+    # the codes, will be slightly different like:
+    # dcgan = DCGAN()
+    # dcgan.build_model()
+    with pd.block() as def_block:
+      dcgan = DCGAN()
+      dcgan.build_model(def_block)
+
+    # load mnist data
+    data_X, data_y = self.load_mnist()
+    
+    # Two subgraphs required!!!
+    with pd.block().d_block():
+      d_optim = pd.train.Adam(lr = .001, beta= .1)
+      d_step = d_optim.minimize(dcgan.d_loss, dcgan.theta_D)
+    with pd.block.g_block():
+      g_optim = pd.train.Adam(lr = .001, beta= .1)
+      g_step = pd.minimize(dcgan.g_loss, dcgan.theta_G)
+
+    # executor
+    sess = pd.executor()
+    
+    # training
+    for epoch in xrange(10000):
+      for batch_id in range(N / batch_size):
+        idx = ...
+        # sample a batch
+        batch_im, batch_label = data_X[idx:idx+batch_size], data_y[idx:idx+batch_size]
+        # sample z
+        batch_z = np.random.uniform(-1., 1., [batch_size, z_dim])
+
+        if batch_id % 2 == 0:
+          sess.run(d_step, 
+                   feed_dict = {dcgan.images: batch_im,
+                                dcgan.y: batch_label,
+                                dcgan.z: batch_z})
+        else:
+          sess.run(g_step,
+                   feed_dict = {dcgan.z: batch_z})
+```
+
+# More thinking about dependency engine v.s. block design:
+- What if we just want to run an intermediate result? Do we need to run the whole block/graph?
+- Should we call eval() to get the fake images in the first stage? And then train the discriminator in the second stage?

develop/doc_cn/design/gan_api.html

+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+  <meta charset="utf-8">
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  
+  <title>Design for GAN &mdash; PaddlePaddle  文档</title>
+  
+
+  
+  
+
+  
+
+  
+  
+    
+
+  
+
+  
+  
+    <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+  
+
+  
+  
+        <link rel="index" title="索引"
+              href="../genindex.html"/>
+        <link rel="search" title="搜索" href="../search.html"/>
+    <link rel="top" title="PaddlePaddle  文档" href="../index.html"/> 
+
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/css/perfect-scrollbar.min.css" type="text/css" />
+  <link rel="stylesheet" href="../_static/css/override.css" type="text/css" />
+  <script>
+  var _hmt = _hmt || [];
+  (function() {
+    var hm = document.createElement("script");
+    hm.src = "//hm.baidu.com/hm.js?b9a314ab40d04d805655aab1deee08ba";
+    var s = document.getElementsByTagName("script")[0]; 
+    s.parentNode.insertBefore(hm, s);
+  })();
+  </script>
+
+  
+
+  
+  <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+  
+  <header class="site-header">
+    <div class="site-logo">
+      <a href="/"><img src="../_static/images/PP_w.png"></a>
+    </div>
+    <div class="site-nav-links">
+      <div class="site-menu">
+        <a class="fork-on-github" href="https://github.com/PaddlePaddle/Paddle" target="_blank"><i class="fa fa-github"></i>Fork me on Github</a>
+        <div class="language-switcher dropdown">
+          <a type="button" data-toggle="dropdown">
+            <span>English</span>
+            <i class="fa fa-angle-up"></i>
+            <i class="fa fa-angle-down"></i>
+          </a>
+          <ul class="dropdown-menu">
+            <li><a href="/doc_cn">中文</a></li>
+            <li><a href="/doc">English</a></li>
+          </ul>
+        </div>
+        <ul class="site-page-links">
+          <li><a href="/">Home</a></li>
+        </ul>
+      </div>
+      <div class="doc-module">
+        
+        <ul>
+<li class="toctree-l1"><a class="reference internal" href="../getstarted/index_cn.html">新手入门</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../howto/index_cn.html">进阶指南</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api/index_cn.html">API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq/index_cn.html">FAQ</a></li>
+</ul>
+
+        
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>        
+      </div>
+    </div>
+  </header>
+  
+  <div class="main-content-wrap">
+
+    
+    <nav class="doc-menu-vertical" role="navigation">
+        
+          
+          <ul>
+<li class="toctree-l1"><a class="reference internal" href="../getstarted/index_cn.html">新手入门</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../getstarted/build_and_install/index_cn.html">安装与编译</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/docker_install_cn.html">PaddlePaddle的Docker容器使用方式</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../getstarted/build_and_install/cmake/build_from_source_cn.html">PaddlePaddle的编译选项</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../getstarted/concepts/use_concepts_cn.html">基本使用概念</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../howto/index_cn.html">进阶指南</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/cmd_parameter/index_cn.html">设置命令行参数</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/use_case_cn.html">使用案例</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/arguments_cn.html">参数概述</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/usage/cmd_parameter/detail_introduction_cn.html">细节描述</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/cluster/cluster_train_cn.html">运行分布式训练</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_basis_cn.html">Kubernetes 简介</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_cn.html">Kubernetes单机训练</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/usage/k8s/k8s_distributed_cn.html">Kubernetes分布式训练</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/dev/build_cn.html">编译PaddlePaddle和运行单元测试</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/dev/write_docs_cn.html">如何贡献/修改文档</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/dev/contribute_to_paddle_cn.html">如何贡献代码</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/deep_model/rnn/index_cn.html">RNN相关模型</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/rnn_config_cn.html">RNN配置</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/recurrent_group_cn.html">Recurrent Group教程</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/hierarchical_layer_cn.html">支持双层序列作为输入的Layer</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../howto/deep_model/rnn/hrnn_rnn_api_compare_cn.html">单双层RNN API对比介绍</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../howto/optimization/gpu_profiling_cn.html">GPU性能分析与调优</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../api/index_cn.html">API</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../api/v2/model_configs.html">模型配置</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/activation.html">Activation</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/layer.html">Layers</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/evaluators.html">Evaluators</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/optimizer.html">Optimizer</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/pooling.html">Pooling</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/networks.html">Networks</a></li>
+<li class="toctree-l3"><a class="reference internal" href="../api/v2/config/attr.html">Parameter Attribute</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="../api/v2/data.html">数据访问</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../api/v2/run_logic.html">训练与应用</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="../faq/index_cn.html">FAQ</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="../faq/build_and_install/index_cn.html">编译安装与单元测试</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../faq/model/index_cn.html">模型配置</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../faq/parameter/index_cn.html">参数设置</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../faq/local/index_cn.html">本地训练与预测</a></li>
+<li class="toctree-l2"><a class="reference internal" href="../faq/cluster/index_cn.html">集群训练与预测</a></li>
+</ul>
+</li>
+</ul>
+
+        
+    </nav>
+    
+    <section class="doc-content-wrap">
+
+      
+
+ 
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+  <ul class="wy-breadcrumbs">
+      
+    <li>Design for GAN</li>
+  </ul>
+</div>
+      
+      <div class="wy-nav-content" id="doc-content">
+        <div class="rst-content">
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <div class="section" id="design-for-gan">
+<span id="design-for-gan"></span><h1>Design for GAN<a class="headerlink" href="#design-for-gan" title="永久链接至标题">¶</a></h1>
+<p>GAN (General Adversarial Net [https://arxiv.org/abs/1406.2661]) is an important model for unsupervised learning and widely used in many areas.</p>
+<p>It applies several important concepts in machine learning system design, including building and running subgraphs, dependency tracing, different optimizers in one executor and so forth.</p>
+<p>In our GAN design, we wrap it as a user-friendly easily customized python API to design different models. We take the conditional DC-GAN (Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [https://arxiv.org/abs/1511.06434]) as an example due to its good performance on image generation.</p>
+<p align="center">
+<img src="./test.dot.png" width = "35%" align="center"/><br/>
+Figure 1. The overall running logic of GAN. The black solid arrows indicate the forward pass; the green dashed arrows indicate the backward pass of generator training; the red dashed arrows indicate the backward pass of the discriminator training. The BP pass of the green (red) arrow should only update the parameters in the green (red) boxes. The diamonds indicate the data providers. d\_loss and g\_loss marked in red and green are the two targets we would like to run.
+</p><p>The operators, layers and functions required/optional to build a GAN demo is summarized in https://github.com/PaddlePaddle/Paddle/issues/4563.</p>
+<p align="center">
+<img src="./dcgan.png" width = "90%" align="center"/><br/>
+Figure 2. Photo borrowed from the original DC-GAN paper.
+</p><div class="section" id="the-conditional-gan-might-be-a-class">
+<span id="the-conditional-gan-might-be-a-class"></span><h2>The Conditional-GAN might be a class.<a class="headerlink" href="#the-conditional-gan-might-be-a-class" title="永久链接至标题">¶</a></h2>
+<p>This design we adopt the popular open source design in https://github.com/carpedm20/DCGAN-tensorflow and https://github.com/rajathkmp/DCGAN. It contains following data structure:</p>
+<ul class="simple">
+<li>DCGAN(object): which contains everything required to build a GAN model. It provides following member functions methods as API:</li>
+<li><strong>init</strong>(...): Initialize hyper-parameters (like conv dimension and so forth), and declare model parameters of discriminator and generator as well.</li>
+<li>generator(z, y=None): Generate a fake image from input noise z. If the label y is provided, the conditional GAN model will be chosen.
+Returns a generated image.</li>
+<li>discriminator(image):
+Given an image, decide if it is from a real source or a fake one.
+Returns a 0/1 binary label.</li>
+<li>build_model(self):
+build the whole GAN model, define training loss for both generator and discrimator.</li>
+</ul>
+</div>
+<div class="section" id="discussion-on-engine-functions-required-to-build-gan">
+<span id="discussion-on-engine-functions-required-to-build-gan"></span><h2>Discussion on Engine Functions required to build GAN<a class="headerlink" href="#discussion-on-engine-functions-required-to-build-gan" title="永久链接至标题">¶</a></h2>
+<ul class="simple">
+<li>Trace the tensor and variable dependency in the engine executor. (Very critical, otherwise GAN can&#8217;be be trained correctly)</li>
+<li>Different optimizers responsible for optimizing different loss.</li>
+</ul>
+<p>To be more detailed, we introduce our design of DCGAN as following:</p>
+<div class="section" id="class-member-function-initializer">
+<span id="class-member-function-initializer"></span><h3>Class member Function: Initializer<a class="headerlink" href="#class-member-function-initializer" title="永久链接至标题">¶</a></h3>
+<ul class="simple">
+<li>Set up hyper-parameters, including condtional dimension, noise dimension, batch size and so forth.</li>
+<li>Declare and define all the model variables. All the discriminator parameters are included in the list self.theta_D and all the generator parameters are included in the list self.theta_G.</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">y_dim</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+  
+    <span class="c1"># hyper parameters  </span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span> <span class="o">=</span> <span class="n">y_dim</span> <span class="c1"># conditional gan or not</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="mi">100</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">z_dim</span> <span class="o">=</span> <span class="n">z_dim</span> <span class="c1"># input noise dimension</span>
+
+    <span class="c1"># define parameters of discriminators</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_W0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_b0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_W1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_b1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_W2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Varialble</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">D_b2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">theta_D</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">D_W0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_b0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_W1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_b1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_W2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">D_b2</span><span class="p">]</span>
+
+    <span class="c1"># define parameters of generators</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_W0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_b0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_W1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="mi">784</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="n">data</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">gaussian_normal_randomizer</span><span class="p">())</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_b1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span> <span class="c1"># variable also support initialization using a  numpy data</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_W2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Varialble</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">G_b2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="mi">128</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">theta_G</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">G_W0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_W1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_W2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b2</span><span class="p">]</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="class-member-function-generator">
+<span id="class-member-function-generator"></span><h3>Class member Function: Generator<a class="headerlink" href="#class-member-function-generator" title="永久链接至标题">¶</a></h3>
+<ul class="simple">
+<li>Given a noisy input z, returns a fake image.</li>
+<li>Concatenation, batch-norm, FC operations required;</li>
+<li>Deconv layer required, which is missing now...</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">generator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">z</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="bp">None</span><span class="p">):</span>
+    <span class="c1"># input z: the random noise</span>
+    <span class="c1"># input y: input data label (optional)</span>
+    <span class="c1"># output G_im: generated fake images</span>
+    
+    <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span>
+      <span class="n">z</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">concat</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="n">z</span><span class="p">,</span> <span class="n">y</span><span class="p">])</span>
+      
+    <span class="n">G_h0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_w0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b0</span><span class="p">)</span>
+    <span class="n">G_h0_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batch_norm</span><span class="p">(</span><span class="n">G_h0</span><span class="p">)</span>
+    <span class="n">G_h0_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">G_h0_bn</span><span class="p">)</span>
+    
+    <span class="n">G_h1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">deconv</span><span class="p">(</span><span class="n">G_h0_relu</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_w1</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b1</span><span class="p">)</span>
+    <span class="n">G_h1_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batch_norm</span><span class="p">(</span><span class="n">G_h1</span><span class="p">)</span>
+    <span class="n">G_h1_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="n">G_h1_bn</span><span class="p">)</span>
+    
+    <span class="n">G_h2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">deconv</span><span class="p">(</span><span class="n">G_h1_relu</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_W2</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">G_b2</span><span class="p">))</span>
+    <span class="n">G_im</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">G_im</span><span class="p">)</span>
+    <span class="k">return</span> <span class="n">G_im</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="class-member-function-discriminator">
+<span id="class-member-function-discriminator"></span><h3>Class member function: Discriminator<a class="headerlink" href="#class-member-function-discriminator" title="永久链接至标题">¶</a></h3>
+<ul class="simple">
+<li>Given a noisy input z, returns a fake image.</li>
+<li>Concatenation, Convolution, batch-norm, FC, Leaky-ReLU operations required;</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image</span><span class="p">):</span>
+    <span class="c1"># input image: either generated images or real ones</span>
+    <span class="c1"># output D_h2: binary logit of the label</span>
+
+    <span class="n">D_h0</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">conv2d</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_w0</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_b0</span><span class="p">)</span>
+    <span class="n">D_h0_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batchnorm</span><span class="p">(</span><span class="n">h0</span><span class="p">)</span>
+    <span class="n">D_h0_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">lrelu</span><span class="p">(</span><span class="n">h0_bn</span><span class="p">)</span>
+    
+    <span class="n">D_h1</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">conv2d</span><span class="p">(</span><span class="n">D_h0_relu</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_w1</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_b1</span><span class="p">)</span>
+    <span class="n">D_h1_bn</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">batchnorm</span><span class="p">(</span><span class="n">D_h1</span><span class="p">)</span>
+    <span class="n">D_h1_relu</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">lrelu</span><span class="p">(</span><span class="n">D_h1_bn</span><span class="p">)</span>
+    
+    <span class="n">D_h2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">layer</span><span class="o">.</span><span class="n">fc</span><span class="p">(</span><span class="n">D_h1_relu</span><span class="p">,</span> <span class="n">w</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_w2</span><span class="p">,</span> <span class="n">b</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">D_b2</span><span class="p">)</span>
+    <span class="k">return</span> <span class="n">D_h2</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="class-member-function-build-the-model">
+<span id="class-member-function-build-the-model"></span><h3>Class member function: Build the model<a class="headerlink" href="#class-member-function-build-the-model" title="永久链接至标题">¶</a></h3>
+<ul class="simple">
+<li>Define data readers as placeholders to hold the data;</li>
+<li>Build generator and discriminators;</li>
+<li>Define two training losses for discriminator and generator, respectively.
+If we have execution dependency engine to back-trace all tensors, the module building our GAN model will be like this:</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">build_model</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">images</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">faked_images</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">z_size</span><span class="p">])</span>
+    
+    <span class="c1"># step 1: generate images by generator, classify real/fake images with discriminator</span>
+    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span> <span class="c1"># if conditional GAN, includes label</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+        <span class="c1"># generated fake images</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">sampled</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sampler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">)</span>
+    <span class="k">else</span><span class="p">:</span> <span class="c1"># original version of GAN</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+        <span class="c1"># generate fake images</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">sampled</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sampler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+    
+    <span class="c1"># step 2: define the two losses</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_t</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_f</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">d_loss</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span>
+    
+    <span class="bp">self</span><span class="o">.</span><span class="n">g_loss</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_f</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_szie</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>If we do not have dependency engine but blocks, the module building our GAN model will be like this:</p>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DCGAN</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+  <span class="k">def</span> <span class="nf">build_model</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">default_block</span><span class="p">):</span>
+    <span class="c1"># input data in the default block</span>
+    <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">images</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">im_size</span><span class="p">])</span>
+    <span class="c1"># self.faked_images = pd.data(pd.float32, [self.batch_size, self.im_size, self.im_size])</span>
+    <span class="bp">self</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">data</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">z_size</span><span class="p">])</span>
+
+    <span class="c1"># step 1: generate images by generator, classify real/fake images with discriminator</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">default_block</span><span class="p">()</span><span class="o">.</span><span class="n">g_block</span><span class="p">():</span>
+      <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span> <span class="c1"># if conditional GAN, includes label</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_g</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+      <span class="k">else</span><span class="p">:</span> <span class="c1"># original version of GAN</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">G</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">generator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">z</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_g</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">g_loss</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_g</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_szie</span><span class="p">))</span>
+    
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">default_block</span><span class="p">()</span><span class="o">.</span><span class="n">d_block</span><span class="p">():</span>
+      <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">y_dim</span><span class="p">:</span> <span class="c1"># if conditional GAN, includes label</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">y</span><span class="p">)</span>
+      <span class="k">else</span><span class="p">:</span> <span class="c1"># original version of GAN</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">images</span><span class="p">)</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">D_f</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">discriminator</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">G</span><span class="p">)</span>
+
+      <span class="c1"># step 2: define the two losses</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_t</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">reduce_mean</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">cross_entropy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">D_f</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">))</span>
+      <span class="bp">self</span><span class="o">.</span><span class="n">d_loss</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_real</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_loss_fake</span>
+</pre></div>
+</div>
+<p>Some small confusion and problems with this design:</p>
+<ul class="simple">
+<li>D_g and D_f are actually the same thing, but has to be written twice; i.e., if we want to run two sub-graphs conceptually, the same codes have to be written twice if they are shared by the graph.</li>
+<li>Requires ability to create a block anytime, rather than in if-else or rnn only;</li>
+</ul>
+</div>
+</div>
+<div class="section" id="main-function-for-the-demo">
+<span id="main-function-for-the-demo"></span><h2>Main function for the demo:<a class="headerlink" href="#main-function-for-the-demo" title="永久链接至标题">¶</a></h2>
+<p>Generally, the user of GAN just need to the following things:</p>
+<ul class="simple">
+<li>Define an object as DCGAN class;</li>
+<li>Build the DCGAN model;</li>
+<li>Specify two optimizers for two different losses with respect to different parameters.</li>
+</ul>
+<div class="highlight-python"><div class="highlight"><pre><span></span><span class="c1"># pd for short, should be more concise.</span>
+<span class="kn">from</span> <span class="nn">paddle.v2</span> <span class="nn">as</span> <span class="nn">pd</span>
+<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span>
+    <span class="c1"># dcgan class in the default graph/block</span>
+    <span class="c1"># if we use dependency engine as tensorflow</span>
+    <span class="c1"># the codes, will be slightly different like:</span>
+    <span class="c1"># dcgan = DCGAN()</span>
+    <span class="c1"># dcgan.build_model()</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">block</span><span class="p">()</span> <span class="k">as</span> <span class="n">def_block</span><span class="p">:</span>
+      <span class="n">dcgan</span> <span class="o">=</span> <span class="n">DCGAN</span><span class="p">()</span>
+      <span class="n">dcgan</span><span class="o">.</span><span class="n">build_model</span><span class="p">(</span><span class="n">def_block</span><span class="p">)</span>
+
+    <span class="c1"># load mnist data</span>
+    <span class="n">data_X</span><span class="p">,</span> <span class="n">data_y</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">load_mnist</span><span class="p">()</span>
+    
+    <span class="c1"># Two subgraphs required!!!</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">block</span><span class="p">()</span><span class="o">.</span><span class="n">d_block</span><span class="p">():</span>
+      <span class="n">d_optim</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">lr</span> <span class="o">=</span> <span class="o">.</span><span class="mo">001</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span> <span class="o">.</span><span class="mi">1</span><span class="p">)</span>
+      <span class="n">d_step</span> <span class="o">=</span> <span class="n">d_optim</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">dcgan</span><span class="o">.</span><span class="n">d_loss</span><span class="p">,</span> <span class="n">dcgan</span><span class="o">.</span><span class="n">theta_D</span><span class="p">)</span>
+    <span class="k">with</span> <span class="n">pd</span><span class="o">.</span><span class="n">block</span><span class="o">.</span><span class="n">g_block</span><span class="p">():</span>
+      <span class="n">g_optim</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Adam</span><span class="p">(</span><span class="n">lr</span> <span class="o">=</span> <span class="o">.</span><span class="mo">001</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span> <span class="o">.</span><span class="mi">1</span><span class="p">)</span>
+      <span class="n">g_step</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">minimize</span><span class="p">(</span><span class="n">dcgan</span><span class="o">.</span><span class="n">g_loss</span><span class="p">,</span> <span class="n">dcgan</span><span class="o">.</span><span class="n">theta_G</span><span class="p">)</span>
+
+    <span class="c1"># executor</span>
+    <span class="n">sess</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">executor</span><span class="p">()</span>
+    
+    <span class="c1"># training</span>
+    <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">10000</span><span class="p">):</span>
+      <span class="k">for</span> <span class="n">batch_id</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">N</span> <span class="o">/</span> <span class="n">batch_size</span><span class="p">):</span>
+        <span class="n">idx</span> <span class="o">=</span> <span class="o">...</span>
+        <span class="c1"># sample a batch</span>
+        <span class="n">batch_im</span><span class="p">,</span> <span class="n">batch_label</span> <span class="o">=</span> <span class="n">data_X</span><span class="p">[</span><span class="n">idx</span><span class="p">:</span><span class="n">idx</span><span class="o">+</span><span class="n">batch_size</span><span class="p">],</span> <span class="n">data_y</span><span class="p">[</span><span class="n">idx</span><span class="p">:</span><span class="n">idx</span><span class="o">+</span><span class="n">batch_size</span><span class="p">]</span>
+        <span class="c1"># sample z</span>
+        <span class="n">batch_z</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="o">-</span><span class="mf">1.</span><span class="p">,</span> <span class="mf">1.</span><span class="p">,</span> <span class="p">[</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">z_dim</span><span class="p">])</span>
+
+        <span class="k">if</span> <span class="n">batch_id</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+          <span class="n">sess</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">d_step</span><span class="p">,</span> 
+                   <span class="n">feed_dict</span> <span class="o">=</span> <span class="p">{</span><span class="n">dcgan</span><span class="o">.</span><span class="n">images</span><span class="p">:</span> <span class="n">batch_im</span><span class="p">,</span>
+                                <span class="n">dcgan</span><span class="o">.</span><span class="n">y</span><span class="p">:</span> <span class="n">batch_label</span><span class="p">,</span>
+                                <span class="n">dcgan</span><span class="o">.</span><span class="n">z</span><span class="p">:</span> <span class="n">batch_z</span><span class="p">})</span>
+        <span class="k">else</span><span class="p">:</span>
+          <span class="n">sess</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">g_step</span><span class="p">,</span>
+                   <span class="n">feed_dict</span> <span class="o">=</span> <span class="p">{</span><span class="n">dcgan</span><span class="o">.</span><span class="n">z</span><span class="p">:</span> <span class="n">batch_z</span><span class="p">})</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="more-thinking-about-dependency-engine-v-s-block-design">
+<span id="more-thinking-about-dependency-engine-v-s-block-design"></span><h1>More thinking about dependency engine v.s. block design:<a class="headerlink" href="#more-thinking-about-dependency-engine-v-s-block-design" title="永久链接至标题">¶</a></h1>
+<ul class="simple">
+<li>What if we just want to run an intermediate result? Do we need to run the whole block/graph?</li>
+<li>Should we call eval() to get the fake images in the first stage? And then train the discriminator in the second stage?</li>
+</ul>
+</div>
+
+
+           </div>
+          </div>
+          <footer>
+  
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &copy; Copyright 2016, PaddlePaddle developers.
+
+    </p>
+  </div>
+  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+
+  
+
+    <script type="text/javascript">
+        var DOCUMENTATION_OPTIONS = {
+            URL_ROOT:'../',
+            VERSION:'',
+            COLLAPSE_INDEX:false,
+            FILE_SUFFIX:'.html',
+            HAS_SOURCE:  true,
+            SOURCELINK_SUFFIX: ".txt",
+        };
+    </script>
+      <script type="text/javascript" src="../_static/jquery.js"></script>
+      <script type="text/javascript" src="../_static/underscore.js"></script>
+      <script type="text/javascript" src="../_static/doctools.js"></script>
+      <script type="text/javascript" src="../_static/translations.js"></script>
+      <script type="text/javascript" src="https://cdn.bootcss.com/mathjax/2.7.0/MathJax.js"></script>
+       
+  
+
+  
+  
+    <script type="text/javascript" src="../_static/js/theme.js"></script>
+  
+  
+  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/perfect-scrollbar/0.6.14/js/perfect-scrollbar.jquery.min.js"></script>
+  <script src="../_static/js/paddle_doc_init.js"></script> 
+
+</body>
+</html>
\ No newline at end of file