Add old releases documentation

release_doc/0.9.0/.buildinfo

+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 2e7327c43832ec6188cc63b41afbbc75
+tags: 645f666f9bcd5a90fca523b33c5a78b7

release_doc/0.9.0/doc/.buildinfo

+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: abb235454c522821afda02c2aa921d6f
+tags: 645f666f9bcd5a90fca523b33c5a78b7

release_doc/0.9.0/doc/_sources/algorithm/rnn/rnn.txt

+Recurrent Neural Network Configuration
+======================================
+
+This tutorial will guide you how to configure recurrent neural network in PaddlePaddle. PaddlePaddle supports highly flexible and efficient recurrent neural network configuration. In this tutorial, you will learn how to:
+
+- prepare sequence data for learning recurrent neural networks.
+- configure recurrent neural network architecture.
+- generate sequence with learned recurrent neural network models.
+
+We will use vanilla recurrent neural network, and sequence to sequence model to guide you through these steps. The code of sequence to sequence model can be found at :code:`demo/seqToseq`.
+
+=====================
+Prepare Sequence Data
+=====================
+
+PaddlePaddle does not need any preprocessing to sequence data, such as padding. The only thing that needs to be done is to set the type of the corresponding type to input. For example, the following code snippets defines three input. All of them are sequences, and the size of them are :code:`src_dict`, :code:`trg_dict`, and :code:`trg_dict`:
+
+.. code-block:: python
+
+    settings.slots = [
+      integer_value_sequence(len(settings.src_dict)),
+      integer_value_sequence(len(settings.trg_dict)),
+      integer_value_sequence(len(settings.trg_dict))]
+
+
+Then at the :code:`process` function, each :code:`yield` function will return three integer lists. Each integer list is treated as a sequence of integers:
+
+.. code-block:: python
+
+    yield src_ids, trg_ids, trg_ids_next
+
+
+For more details description of how to write a data provider, please refer to `PyDataProvider2 <../../ui/data_provider/index.html>`_. The full data provider file is located at :code:`demo/seqToseq/dataprovider.py`.
+
+===============================================
+Configure Recurrent Neural Network Architecture
+===============================================
+
+-------------------------------------
+Simple Gated Recurrent Neural Network
+-------------------------------------
+
+Recurrent neural network process a sequence at each time step sequentially. An example of the architecture of LSTM is listed below.
+
+.. image:: ./bi_lstm.jpg
+	 :align: center
+
+Generally speaking, a recurrent network perform the following operations from :math:`t=1` to :math:`t=T`, or reversely from :math:`t=T` to :math:`t=1`.
+
+.. math::
+
+    x_{t+1} = f_x(x_t), y_t = f_y(x_t)
+
+
+where :math:`f_x(.)` is called **step function**, and :math:`f_y(.)` is called **output function**. In vanilla recurrent neural network, both of the step function and output function are very simple. However, PaddlePaddle supports the configuration of very complex architectures by modifying these two functions. We will use the sequence to sequence model with attention as an example to demonstrate how you can configure complex recurrent neural network models. In this section, we will use a simple vanilla recurrent neural network as an example of configuring simple recurrent neural network using :code:`recurrent_group`. Notice that if you only need to use simple RNN, GRU, or LSTM, then :code:`grumemory` and :code:`lstmemory` is recommended because they are more computationally efficient than :code:`recurrent_group`.
+
+For vanilla RNN, at each time step, the **step function** is:
+
+.. math::
+
+    x_{t+1} = W_x x_t + W_i I_t + b
+
+where :math:`x_t` is the RNN state, and :math:`I_t` is the input, :math:`W_x` and :math:`W_i` are transformation matrices for RNN states and inputs, respectively. :math:`b` is the bias.
+Its **output function** simply takes :math:`x_t` as the output.
+
+:code:`recurrent_group` is the most important tools for constructing recurrent neural networks. It defines the **step function**, **output function** and the inputs of the recurrent neural network. Notice that the :code:`step` argument of this function implements both the :code:`step function` and the :code:`output function`:
+
+.. code-block:: python
+
+    def simple_rnn(input,
+                   size=None,
+                   name=None,
+                   reverse=False,
+                   rnn_bias_attr=None,
+                   act=None,
+                   rnn_layer_attr=None):
+        def __rnn_step__(ipt):
+           out_mem = memory(name=name, size=size)
+           rnn_out = mixed_layer(input = [full_matrix_projection(ipt),
+                                          full_matrix_projection(out_mem)],
+                                 name = name,
+                                 bias_attr = rnn_bias_attr,
+                                 act = act,
+                                 layer_attr = rnn_layer_attr,
+                                 size = size)
+           return rnn_out
+        return recurrent_group(name='%s_recurrent_group' % name,
+                               step=__rnn_step__,
+                               reverse=reverse,
+                               input=input)
+
+
+PaddlePaddle uses memory to construct step function. **Memory** is the most important concept when constructing recurrent neural networks in PaddlePaddle. A memory is a state that is used recurrently in step functions, such as :math:`x_{t+1} = f_x(x_t)`. One memory contains an **output** and a **input**. The output of memory at the current time step is utilized as the input of the memory at the next time step. A memory can also has a **boot layer**, whose output is utilized as the initial value of the memory. In our case, the output of the gated recurrent unit is employed as the output memory. Notice that the name of the layer :code:`rnn_out` is the same as the name of :code:`out_mem`. This means the output of the layer :code:`rnn_out` (:math:`x_{t+1}`) is utilized as the **output** of :code:`out_mem` memory.
+
+A memory can also be a sequence. In this case, at each time step, we have a sequence as the state of the recurrent neural network. This can be useful when constructing very complex recurrent neural network. Other advanced functions include defining multiple memories, and defining hierarchical recurrent neural network architecture using sub-sequence.
+
+We return :code:`rnn_out` at the end of the function. It means that the output of the layer :code:`rnn_out` is utilized as the **output** function of the gated recurrent neural network.
+
+-----------------------------------------
+Sequence to Sequence Model with Attention
+-----------------------------------------
+We will use the sequence to sequence model with attention as an example to demonstrate how you can configure complex recurrent neural network models. An illustration of the sequence to sequence model with attention is shown in the following figure.
+
+.. image:: ./encoder-decoder-attention-model.png
+ 	 :align: center
+
+In this model, the source sequence :math:`S = \{s_1, \dots, s_T\}` is encoded with a bidirectional gated recurrent neural networks. The hidden states of the bidirectional gated recurrent neural network :math:`H_S = \{H_1, \dots, H_T\}` is called *encoder vector* The decoder is a gated recurrent neural network. When decoding each token :math:`y_t`, the gated recurrent neural network generates a set of weights :math:`W_S^t = \{W_1^t, \dots, W_T^t\}`, which are used to compute a weighted sum of the encoder vector. The weighted sum of the encoder vector is utilized to condition the generation of the token :math:`y_t`.
+
+The encoder part of the model is listed below. It calls :code:`grumemory` to represent gated recurrent neural network. It is the recommended way of using recurrent neural network if the network architecture is simple, because it is faster than :code:`recurrent_group`. We have implemented most of the commonly used recurrent neural network architectures, you can refer to `Layers <../../ui/api/trainer_config_helpers/layers_index.html>`_  for more details.
+
+We also project the encoder vector to :code:`decoder_size` dimensional space, get the first instance of the backward recurrent network, and project it to :code:`decoder_size` dimensional space:
+
+.. code-block:: python
+
+    # Define the data layer of the source sentence.
+    src_word_id = data_layer(name='source_language_word', size=source_dict_dim)
+    # Calculate the word embedding of each word.
+    src_embedding = embedding_layer(
+        input=src_word_id,
+        size=word_vector_dim,
+        param_attr=ParamAttr(name='_source_language_embedding'))
+    # Apply forward recurrent neural network.
+    src_forward = grumemory(input=src_embedding, size=encoder_size)
+    # Apply backward recurrent neural network. reverse=True means backward recurrent neural network.
+    src_backward = grumemory(input=src_embedding,
+                              size=encoder_size,
+                              reverse=True)
+    # Mix the forward and backward parts of the recurrent neural network together.
+    encoded_vector = concat_layer(input=[src_forward, src_backward])
+
+    # Project encoding vector to decoder_size.
+    encoder_proj = mixed_layer(input = [full_matrix_projection(encoded_vector)],
+                               size = decoder_size)
+
+    # Compute the first instance of the backward RNN.
+    backward_first = first_seq(input=src_backward)
+
+    # Project the first instance of backward RNN to decoder size.
+    decoder_boot = mixed_layer(input=[full_matrix_projection(backward_first)], size=decoder_size, act=TanhActivation())
+
+
+The decoder uses :code:`recurrent_group` to define the recurrent neural network. The step and output functions are defined in :code:`gru_decoder_with_attention`:
+
+.. code-block:: python
+
+    group_inputs=[StaticInput(input=encoded_vector,is_seq=True),
+                  StaticInput(input=encoded_proj,is_seq=True)]
+    trg_embedding = embedding_layer(
+        input=data_layer(name='target_language_word',
+                         size=target_dict_dim),
+        size=word_vector_dim,
+        param_attr=ParamAttr(name='_target_language_embedding'))
+    group_inputs.append(trg_embedding)
+
+    # For decoder equipped with attention mechanism, in training,
+    # target embedding (the groudtruth) is the data input,
+    # while encoded source sequence is accessed to as an unbounded memory.
+    # StaticInput means the same value is utilized at different time steps.
+    # Otherwise, it is a sequence input. Inputs at different time steps are different.
+    # All sequence inputs should have the same length.
+    decoder = recurrent_group(name=decoder_group_name,
+                              step=gru_decoder_with_attention,
+                              input=group_inputs)
+
+
+The implementation of the step function is listed as below. First, it defines the **memory** of the decoder network. Then it defines attention, gated recurrent unit step function, and the output function:
+
+.. code-block:: python
+
+    def gru_decoder_with_attention(enc_vec, enc_proj, current_word):
+        # Defines the memory of the decoder.
+        # The output of this memory is defined in gru_step.
+        # Notice that the name of gru_step should be the same as the name of this memory.
+        decoder_mem = memory(name='gru_decoder',
+                             size=decoder_size,
+                             boot_layer=decoder_boot)
+        # Compute attention weighted encoder vector.
+        context = simple_attention(encoded_sequence=enc_vec,
+                                   encoded_proj=enc_proj,
+                                   decoder_state=decoder_mem)
+        # Mix the current word embedding and the attention weighted encoder vector.
+        decoder_inputs = mixed_layer(inputs = [full_matrix_projection(context),
+                                               full_matrix_projection(current_word)],
+                                     size = decoder_size * 3)
+        # Define Gated recurrent unit recurrent neural network step function.
+        gru_step = gru_step_layer(name='gru_decoder',
+                                  input=decoder_inputs,
+                                  output_mem=decoder_mem,
+                                  size=decoder_size)
+        # Defines the output function.
+        out = mixed_layer(input=[full_matrix_projection(input=gru_step)],
+                          size=target_dict_dim,
+                          bias_attr=True,
+                          act=SoftmaxActivation())
+        return out
+
+
+=================
+Generate Sequence
+=================
+After training the model, we can use it to generate sequences. A common practice is to use **beam search** to generate sequences. The following code snippets defines a beam search algorithm. Notice that :code:`beam_search` function assumes the output function of the :code:`step` returns a softmax normalized probability vector of the next token. We made the following changes to the model.
+
+* use :code:`GeneratedInput` for trg_embedding. :code:`GeneratedInput` computes the embedding of the generated token at the last time step for the input at the current time step.
+* use :code:`beam_search` function. This function needs to set:
+
+  - :code:`bos_id`: the start token. Every sentence starts with the start token.
+  - :code:`eos_id`: the end token. Every sentence ends with the end token.
+  - :code:`beam_size`: the beam size used in beam search.
+  - :code:`max_length`: the maximum length of the generated sentences.
+
+* use :code:`seqtext_printer_evaluator` to print text according to index matrix and dictionary. This function needs to set:
+
+  - :code:`id_input`: the integer ID of the data, used to identify the corresponding output in the generated files.
+  - :code:`dict_file`: the dictionary file for converting word id to word.
+  - :code:`result_file`: the path of the generation result file.
+    
+The code is listed below:
+
+.. code-block:: python
+
+    group_inputs=[StaticInput(input=encoded_vector,is_seq=True),
+                  StaticInput(input=encoded_proj,is_seq=True)]
+    # In generation, decoder predicts a next target word based on
+    # the encoded source sequence and the last generated target word.
+    # The encoded source sequence (encoder's output) must be specified by
+    # StaticInput which is a read-only memory.
+    # Here, GeneratedInputs automatically fetchs the last generated word,
+    # which is initialized by a start mark, such as <s>.
+    trg_embedding = GeneratedInput(
+        size=target_dict_dim,
+        embedding_name='_target_language_embedding',
+        embedding_size=word_vector_dim)
+    group_inputs.append(trg_embedding)
+    beam_gen = beam_search(name=decoder_group_name,
+                           step=gru_decoder_with_attention,
+                           input=group_inputs,
+                           bos_id=0, # Beginnning token.
+                           eos_id=1, # End of sentence token.
+                           beam_size=beam_size,
+                           max_length=max_length)
+
+    seqtext_printer_evaluator(input=beam_gen,
+                              id_input=data_layer(name="sent_id", size=1),
+                              dict_file=trg_dict_path,
+                              result_file=gen_trans_file)
+    outputs(beam_gen)
+
+
+Notice that this generation technique is only useful for decoder like generation process. If you are working on sequence tagging tasks, please refer to `Semantic Role Labeling Demo <../../demo/semantic_role_labeling/index.html>`_ for more details.
+
+The full configuration file is located at :code:`demo/seqToseq/seqToseq_net.py`.

release_doc/0.9.0/doc/_sources/build/build_from_source.txt

+Installing from Sources
+==========================
+
+* [1. Download and Setup](#download)
+* [2. Requirements](#requirements)
+* [3. Build on Ubuntu](#ubuntu)
+
+## <span id="download">Download and Setup</span> 
+You can download PaddlePaddle from the [github source](https://github.com/gangliao/Paddle).
+
+```bash
+git clone https://github.com/baidu/Paddle paddle
+cd paddle
+```
+
+## <span id="requirements">Requirements</span>
+
+To compile the source code, your computer must be equipped with GCC >=4.6 or Clang compiler.
+### Dependencies
+
+- **CMake**: version >= 2.8
+- **BLAS**: MKL, OpenBlas or ATLAS
+- **protobuf**: version >= 2.4, **Note: 3.x is not supported**
+- **python**: only python 2.7 is supported currently
+
+### Options
+
+PaddlePaddle supports some build options. To enable it, first you need to install the related libraries. 
+
+<html>
+<table> 
+<thead>
+<tr>
+<th scope="col" class="left">Optional</th>
+<th scope="col" class="left">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr><td class="left">WITH_GPU</td><td class="left">Compile with GPU mode.</td></tr>
+<tr><td class="left">WITH_DOUBLE</td><td class="left">Compile with double precision floating-point, default: single precision.</td></tr>
+<tr><td class="left">WITH_GLOG</td><td class="left">Compile with glog. If not found, default: an internal log implementation.</td></tr>
+<tr><td class="left">WITH_GFLAGS</td><td class="left">Compile with gflags. If not found, default: an internal flag implementation.</td></tr>
+<tr><td class="left">WITH_TESTING</td><td class="left">Compile with gtest for PaddlePaddle's unit testing.</td></tr>
+<tr><td class="left">WITH_DOC</td><td class="left">	Compile to generate PaddlePaddle's docs, default: disabled (OFF).</td></tr>
+<tr><td class="left">WITH_SWIG_PY</td><td class="left">Compile with python predict API, default: disabled (OFF).</td></tr>
+<tr><td class="left">WITH_STYLE_CHECK</td><td class="left">Compile with code style check, default: enabled (ON).</td></tr>
+</tbody>
+</table>
+</html>
+
+**Note:**
+  - The GPU version works best with Cuda Toolkit 7.5 and cuDNN v5.
+  - Other versions like Cuda Toolkit 6.5, 7.0, 8.0 and cuDNN v2, v3, v4 are also supported.
+  - **To utilize cuDNN v5, Cuda Toolkit 7.5 is prerequisite and vice versa.**
+
+As a simple example, consider the following:  
+
+1. **Python Dependencies(optional)**
+  
+    To compile PaddlePaddle with python predict API, make sure swig installed and set `-DWITH_SWIG_PY=ON` as follows:
+
+    ```bash
+    # install swig on ubuntu
+    sudo apt-get install swig
+    # install swig on Mac OS X
+    brew install swig
+
+    # active swig in cmake
+    cmake .. -DWITH_SWIG_PY=ON
+    ```
+
+2. **Doc Dependencies(optional)**
+
+    To generate PaddlePaddle's documentation, install dependencies and set `-DWITH_DOC=ON` as follows:
+
+    ```bash
+    pip install 'sphinx>=1.4.0'
+    pip install sphinx_rtd_theme breathe recommonmark
+
+    # install doxygen on Ubuntu
+    sudo apt-get install doxygen 
+    # install doxygen on Mac OS X
+    brew install doxygen
+
+    # active docs in cmake
+    cmake .. -DWITH_DOC=ON`
+    ```
+
+## <span id="ubuntu">Build on Ubuntu 14.04</span>
+
+### Install Dependencies
+
+- **CPU Dependencies**
+
+    ```bash
+    # necessary
+    sudo apt-get update
+    sudo apt-get install -y g++ make cmake build-essential libatlas-base-dev python python-pip libpython-dev m4 libprotobuf-dev protobuf-compiler python-protobuf python-numpy git
+    # optional
+    sudo apt-get install libgoogle-glog-dev
+    sudo apt-get install libgflags-dev
+    sudo apt-get install libgtest-dev
+    sudo pip install wheel
+    pushd /usr/src/gtest
+    cmake .
+    make
+    sudo cp *.a /usr/lib
+    popd
+    ```
+  
+- **GPU Dependencies (optional)**
+
+    To build GPU version, you will need the following installed:
+
+        1. a CUDA-capable GPU
+        2. A supported version of Linux with a gcc compiler and toolchain
+        3. NVIDIA CUDA Toolkit (available at http://developer.nvidia.com/cuda-downloads)
+        4. NVIDIA cuDNN Library (availabel at https://developer.nvidia.com/cudnn)
+
+    The CUDA development environment relies on tight integration with the host development environment,
+    including the host compiler and C runtime libraries, and is therefore only supported on
+    distribution versions that have been qualified for this CUDA Toolkit release.
+        
+    After downloading cuDNN library, issue the following commands:
+
+    ```bash
+    sudo tar -xzf cudnn-7.5-linux-x64-v5.1.tgz -C /usr/local
+    sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
+    ```
+    Then you need to set LD\_LIBRARY\_PATH, PATH environment variables in ~/.bashrc.
+
+    ```bash
+    export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
+    export PATH=/usr/local/cuda/bin:$PATH
+    ```
+
+### Build and Install
+
+As usual, the best option is to create build folder under paddle project directory.
+
+```bash
+mkdir build && cd build
+cmake ..
+```
+
+CMake first check PaddlePaddle's dependencies in system default path. After installing some optional
+libraries, corresponding build option will be set automatically (for instance, glog, gtest and gflags).
+If still not found, you can manually set it based on CMake error information from your screen.
+
+As a simple example, consider the following:
+
+- **Only CPU**
+
+  ```bash
+  cmake  .. -DWITH_GPU=OFF
+  ```
+- **GPU**
+
+  ```bash
+  cmake .. -DWITH_GPU=ON
+  ```
+
+- **GPU with doc and swig**
+
+  ```bash
+  cmake .. -DWITH_GPU=ON -DWITH_DOC=ON -DWITH_SWIG_PY=ON
+  ``` 
+
+Finally, you can build PaddlePaddle:
+
+```bash
+# you can add build option here, such as:    
+cmake .. -DWITH_GPU=ON -DCMAKE_INSTALL_PREFIX=<path to install>
+# please use sudo make install, if you want to install PaddlePaddle into the system
+make -j `nproc` && make install
+# set PaddlePaddle installation path in ~/.bashrc
+export PATH=<path to install>/bin:$PATH
+```
+
+**Note:**
+
+If you set `WITH_SWIG_PY=ON`, related python dependencies also need to be installed.
+Otherwise, PaddlePaddle will automatically install python dependencies
+at first time when user run paddle commands, such as `paddle version`, `paddle train`.
+It may require sudo privileges:
+
+```bash
+# you can run
+sudo pip install <path to install>/opt/paddle/share/wheels/*.whl
+# or just run 
+sudo paddle version
+```

release_doc/0.9.0/doc/_sources/build/contribute_to_paddle.txt

+# Contribute to PaddlePaddle
+
+We sincerely appreciate your contributions. You can use fork and pull request
+workflow to merge your code. 
+ 
+## Code Requirements
+- Your code must be fully documented by
+  [doxygen](http://www.stack.nl/~dimitri/doxygen/) style.
+- Make sure the compiler option WITH\_STYLE\_CHECK is on and the compiler
+  passes the code style check.
+- All code must have unit test.
+- Pass all unit tests.
+
+The following tutorial guides you into submitting your contibution.
+ 
+## [Creating a Fork](https://help.github.com/articles/fork-a-repo/)
+ 
+Just head over to the GitHub page and click the "Fork" button.
+It's just that simple. 
+
+## Clone
+
+Paddle is currently using [git-flow branching model](http://nvie.com/posts/a-successful-git-branching-model/).
+The **develop** is the main branch, and other user's branches are feature branches.
+
+Once you've created a fork, you can use your favorite git client to clone your
+repo or just head straight to the command line:
+ 
+```shell
+# Clone your fork to your local machine
+git clone --branch develop https://github.com/USERNAME/Paddle.git
+```
+If your repository doesn't contain **develop** branch, just create it by your own.
+
+```shell
+git clone https://github.com/USERNAME/Paddle.git Paddle
+cd Paddle
+git checkout -b develop  # create develop branch.
+git remote add upstream https://github.com/baidu/Paddle.git  # add upstream to baidu/Paddle
+git pull upstream develop  # update to upstream
+```
+
+Then you can start to develop by making a local developement branch
+
+```shell
+git checkout -b MY_COOL_STUFF_BRANCH
+```
+
+## Commit
+
+Commit your changes by following command lines:
+
+```shell
+# show the working tree status
+git status
+# add modified files
+git add xx
+env EDITOR=vim git commit  # You can write your comments by vim/nano/emacs.
+```
+The first line of commit infomation is the title. The second and later lines
+are the details if any.
+
+## Keeping Fork Up to Date
+
+Before pull your request, you should sync your code from the latest PaddlePaddle.
+To do this, you'll need to add a remote at first:
+
+```shell
+# see the current configured remote repository
+git remote -v
+# add upstream repository
+git remote add upstream https://github.com/baidu/Paddle.git
+# verify the new upstream
+git remote -v
+```
+
+Update your fork with the latest upstream changes:
+
+```shell
+git pull --rebase upstream develop
+```
+
+If there are no unique commits locally, git will simply perform a fast-forward.
+However, if you have been making changes (in the vast majority of cases you
+probably shouldn't be), you may have to deal with conflicts. 
+
+Now, your local master branch is up-to-date with everything modified upstream.
+
+## Push to GitHub
+
+```shell
+# push to your repository in Github
+git push -u origin MY_COOL_STUFF_BRANCH  # create remote branch MY_COOL_STUFF_BRANCH to origin.
+```
+
+## Pull Request
+
+Go to the page for your fork on GitHub, select your development branch,
+and click the **pull request button**.
+
+## Update your pull request with the lastest version
+
+During the code review, your pull request may become stale because new commits in
+baidu/Paddle. GitHub allows autmotic update if there is no conflict. You can do this
+by clicking the "Update Branch" button in your pull request page. However, in the case
+of conflict, you need to do the update manually. You need to do the following on
+your local repository:
+```shell
+git checkout MY_COOL_STUFF_BRANCH
+git pull upstream develop
+# You may need to resolve the conflict according to the git prompt.
+# Make and test your code.
+git push origin MY_COOL_STUFF_BRANCH
+```
+Now your Pull Request is updated with the latest version.
+
+## Revise your pull request
+
+When you revise your pull request according to reviewer's comments, please use 'git commit' instead of 'git commit --amend' to commit your changes so that the reviewers can see the difference between the new pull requrest and the old pull request.
+
+The possible commands are
+
+```shell
+git checkout MY_COOL_STUFF_BRANCH
+git pull upstream develop   # update local to newest code base.
+# May be some conflicts will occured.
+# And develop your cool stuff
+env EDITOR=vim git commit  # add your revise log
+git push origin MY_COOL_STUFF_BRANCH
+```

release_doc/0.9.0/doc/_sources/build/docker_install.txt

+Docker installation guide
+==========================
+
+PaddlePaddle provide the `Docker <https://www.docker.com/>`_ image. `Docker`_ is a lightweight container utilities. The performance of PaddlePaddle in `Docker`_ container is basically as same as run it in a normal linux. The `Docker`_ is a very convenient way to deliver the binary release for linux programs.
+
+..  note::
+
+    The `Docker`_ image is the recommended way to run PaddlePaddle 
+
+PaddlePaddle Docker images
+--------------------------
+
+There are 12 `images <https://hub.docker.com/r/paddledev/paddle/tags/>`_ for PaddlePaddle, and the name is :code:`paddle-dev/paddle`,  tags are\: 
+
+
++-----------------+------------------+------------------------+-----------------------+
+|                 |   normal         |           devel        |          demo         |
++=================+==================+========================+=======================+
+|       CPU       | cpu-latest       | cpu-devel-latest       | cpu-demo-latest       |
++-----------------+------------------+------------------------+-----------------------+
+|       GPU       | gpu-latest       | gpu-devel-latest       | gpu-demo-latest       |
++-----------------+------------------+------------------------+-----------------------+
+| CPU WITHOUT AVX | cpu-noavx-latest | cpu-devel-noavx-latest | cpu-demo-noavx-latest |
++-----------------+------------------+------------------------+-----------------------+
+| GPU WITHOUT AVX | gpu-noavx-latest | gpu-devel-noavx-latest | gpu-demo-noavx-latest |
++-----------------+------------------+------------------------+-----------------------+
+
+And the three columns are:
+
+* normal\: The docker image only contains binary of PaddlePaddle.
+* devel\: The docker image contains PaddlePaddle binary, source code and essential build environment.
+* demo\: The docker image contains the dependencies to run PaddlePaddle demo.
+
+And the four rows are:
+
+* CPU\: CPU Version. Support CPU which has :code:`AVX` instructions.
+* GPU\: GPU Version. Support GPU, and cpu has :code:`AVX` instructions.
+* CPU WITHOUT AVX\: CPU Version, which support most CPU even doesn't have :code:`AVX` instructions.
+* GPU WITHOUT AVX\: GPU Version, which support most CPU even doesn't have :code:`AVX` instructions.
+
+User can choose any version depends on machine. The following script can help you to detect your CPU support :code:`AVX` or not.
+
+..  code-block:: bash
+    
+    if cat /proc/cpuinfo | grep -q avx ; then echo "Support AVX"; else echo "Not support AVX"; fi
+
+If the output is :code:`Support AVX`, then you can choose the AVX version of PaddlePaddle, otherwise, you need select :code:`noavx` version of PaddlePaddle. For example, the CPU develop version of PaddlePaddle is :code:`paddle-dev/paddle:cpu-devel-latest`.
+
+The PaddlePaddle images don't contain any entry command. You need to write your entry command to use this image. See :code:`Remote Access` part or just use following command to run a :code:`bash`
+
+..  code-block:: bash
+
+    docker run -it paddledev/paddle:cpu-latest /bin/bash
+
+
+Download and Run Docker images
+------------------------------
+
+You have to install Docker in your machine which has linux kernel version 3.10+ first. You can refer to the official guide https://docs.docker.com/engine/installation/ for further information.
+
+You can use :code:`docker pull ` to download images first, or just launch a container with :code:`docker run` \:
+
+..  code-block:: bash
+
+    docker run -it paddledev/paddle:cpu-latest
+
+
+If you want to launch container with GPU support, you need to set some environment variables at the same time:
+
+..  code-block:: bash
+
+    export CUDA_SO="$(\ls /usr/lib64/libcuda* | xargs -I{} echo '-v {}:{}') $(\ls /usr/lib64/libnvidia* | xargs -I{} echo '-v {}:{}')"
+    export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}')
+    docker run ${CUDA_SO} ${DEVICES} -it paddledev/paddle:gpu-latest
+
+
+Some notes for docker
+---------------------
+
+Performance
++++++++++++
+
+Since Docker is based on the lightweight virtual containers, the CPU computing performance maintains well. And GPU driver and equipments are all mapped to the container, so the GPU computing performance would not be seriously affected.
+
+If you use high performance nic, such as RDMA(RoCE 40GbE or IB 56GbE), Ethernet(10GbE), it is recommended to use config "-net = host".
+
+
+
+
+Remote access
++++++++++++++
+
+
+If you want to enable ssh access background, you need to build an image by yourself. Please refer to official guide https://docs.docker.com/engine/reference/builder/ for further information.
+
+Following is a simple Dockerfile with ssh:
+
+..  literalinclude:: ../../doc_cn/build_and_install/install/paddle_ssh.Dockerfile
+
+Then you can build an image with Dockerfile and launch a container:
+
+..  code-block:: bash
+
+    # cd into Dockerfile directory
+    docker build . -t paddle_ssh
+    # run container, and map host machine port 8022 to container port 22
+    docker run -d -p 8022:22 --name paddle_ssh_machine paddle_ssh
+
+Now, you can ssh on port 8022 to access the container, username is root, password is also root:
+
+..  code-block:: bash
+
+    ssh -p 8022 root@YOUR_HOST_MACHINE
+
+You can stop and delete the container as following:
+
+..  code-block:: bash
+
+    # stop
+    docker stop paddle_ssh_machine
+    # delete
+    docker rm paddle_ssh_machine

release_doc/0.9.0/doc/_sources/build/index.txt

+Build And Install PaddlePaddle
+================================
+
+Install PaddlePaddle
+----------------------
+
+..  toctree::
+    :maxdepth: 1
+    :glob:
+
+    install_*
+    internal/install_from_jumbo.md
+    docker_install.rst
+    ubuntu_install.rst
+
+Build from Source
+-----------------
+
+..  warning::
+
+    Please use :code:`deb` package or :code:`docker` image to install paddle. The building guide is used for hacking or contributing to PaddlePaddle.
+    
+
+If you want to hack and contribute PaddlePaddle source code, following guides can help you\:
+
+
+..  toctree::
+    :maxdepth: 1
+    :glob:
+
+    build_from_source.md
+    contribute_to_paddle.md
+

release_doc/0.9.0/doc/_sources/build/ubuntu_install.txt

+Debian Package installation guide
+=================================
+
+PaddlePaddle supports :code:`deb` pacakge. The installation of this :code:`deb` package is tested in ubuntu 14.04, but it should be support other debian based linux, too.
+
+There are four versions of debian package, :code:`cpu`, :code:`gpu`, :code:`cpu-noavx`, :code:`gpu-noavx`. And :code:`noavx` version is used to support CPU which does not contain :code:`AVX` instructions. The download url of :code:`deb` package is \: https://github.com/baidu/Paddle/releases/
+
+
+After downloading PaddlePaddle deb packages, you can use :code:`gdebi` install.
+
+..	code-block:: bash
+
+	gdebi paddle-*.deb
+
+If :code:`gdebi` is not installed, you can use :code:`sudo apt-get install gdebi` to install it.
+
+Or you can use following commands to install PaddlePaddle.
+
+..	code-block:: bash
+
+	dpkg -i paddle-*.deb
+	apt-get install -f
+
+And if you use GPU version deb package, you need to install CUDA toolkit and cuDNN, and set related environment variables(such as LD_LIBRARY_PATH) first. It is normal when `dpkg -i` get errors. `apt-get install -f` will continue install paddle, and install dependences. 
+

release_doc/0.9.0/doc/_sources/cluster/index.txt

+Cluster Train
+====================
+
+.. toctree::
+  :glob:
+
+  opensource/cluster_train.md
+  internal/index.md

release_doc/0.9.0/doc/_sources/cluster/opensource/cluster_train.txt

+# Distributed Training
+
+In this article, we explain how to run distributed Paddle training jobs on clusters.  We will create the distributed version of the single-process training example, [recommendation](https://github.com/baidu/Paddle/tree/develop/demo/recommendation).
+
+[Scripts](https://github.com/baidu/Paddle/tree/develop/paddle/scripts/cluster_train) used in this article launch distributed jobs via SSH.  They also work as a reference for users running more sophisticated cluster management systems like MPI and Kubernetes.
+
+## Prerequisite
+
+1. Aforementioned scripts use a Python library [fabric](http://www.fabfile.org/) to run SSH commands.  We can use `pip` to install fabric:
+
+   ```bash
+pip install fabric
+   ```
+
+1. We need to install PaddlePaddle on all nodes in the cluster.  To enable GPUs, we need to install CUDA in `/usr/local/cuda`; otherwise Paddle would report errors at runtime.
+
+1. Set the `ROOT_DIR` variable in [`cluster_train/conf.py`] on all nodes.  For convenience, we often create a Unix user `paddle` on all nodes and set `ROOT_DIR=/home/paddle`.  In this way, we can write public SSH keys into `/home/paddle/.ssh/authorized_keys` so that user `paddle` can SSH to all nodes without password.
+
+## Prepare Job Workspace
+
+We refer to the directory where we put dependent libraries, config files, etc., as *workspace*.
+
+These ```train/test``` data should be prepared before launching cluster job. To  satisfy the requirement that train/test data are placed in different directory from workspace, PADDLE refers train/test data according to index file named as ```train.list/test.list``` which are used in model config file. So the train/test data also contains train.list/test.list two list file. All local training demo already provides scripts to help you create these two files,  and all nodes in cluster job will handle files with same logical code in normal condition.
+
+Generally, you can use same model file from local training for cluster training. What you should have in mind that, the ```batch_size``` set in ```setting``` function in model file means batch size in ```each``` node of cluster job instead of total batch size if synchronization SGD was used.
+
+Following steps are based on demo/recommendation demo in demo directory.
+
+You just go through demo/recommendation tutorial doc until ```Train``` section, and at last you will get train/test data and model configuration file. Finaly, just use demo/recommendation as workspace for cluster training.
+
+At last your workspace should look like as follow:
+```
+.
+|-- common_utils.py
+|-- data
+|   |-- config.json
+|   |-- config_generator.py
+|   |-- meta.bin
+|   |-- meta_config.json
+|   |-- meta_generator.py
+|   |-- ml-1m
+|   |-- ml_data.sh
+|   |-- ratings.dat.test
+|   |-- ratings.dat.train
+|   |-- split.py
+|   |-- test.list
+|   `-- train.list
+|-- dataprovider.py
+|-- evaluate.sh
+|-- prediction.py
+|-- preprocess.sh
+|-- requirements.txt
+|-- run.sh
+`-- trainer_config.py
+```
+Not all of these files are needed for cluster training, but it's not necessary to remove useless files.
+
+```trainer_config.py```
+Indicates the model config file.
+
+```train.list``` and ```test.list```
+File index. It stores all relative or absolute file paths of all train/test data at current node.
+
+```dataprovider.py```
+used to read train/test samples. It's same as local training.
+
+```data```
+all files in data directory are refered by train.list/test.list which are refered by data provider.
+
+
+## Prepare Cluster Job Configuration
+
+The options below must be carefully set in cluster_train/conf.py
+
+```HOSTS```  all nodes hostname or ip that will run cluster job. You can also append user and ssh port with hostname, such as root@192.168.100.17:9090.
+
+```ROOT_DIR``` workspace ROOT directory for placing JOB workspace directory
+
+```PADDLE_NIC``` the NIC(Network Interface Card) interface name for cluster communication channel, such as eth0 for ethternet, ib0 for infiniband.
+
+```PADDLE_PORT``` port number for cluster commnunication channel
+
+```PADDLE_PORTS_NUM``` the number of port used for cluster communication channle. if the number of cluster nodes is small(less than 5~6nodes), recommend you set it to larger, such as 2 ~ 8, for better network performance.
+
+```PADDLE_PORTS_NUM_FOR_SPARSE``` the number of port used for sparse updater cluster commnunication channel. if sparse remote update is used, set it like ```PADDLE_PORTS_NUM```
+
+```LD_LIBRARY_PATH``` set addtional LD_LIBRARY_PATH for cluster job. You can use it to set CUDA libraries path.
+
+Default Configuration as follow:
+
+```python
+HOSTS = [
+        "root@192.168.100.17",
+        "root@192.168.100.18",
+        ]
+
+'''
+workspace configuration
+'''
+
+#root dir for workspace
+ROOT_DIR = "/home/paddle"
+
+'''
+network configuration
+'''
+#pserver nics
+PADDLE_NIC = "eth0"
+#pserver port
+PADDLE_PORT = 7164
+#pserver ports num
+PADDLE_PORTS_NUM = 2
+#pserver sparse ports num
+PADDLE_PORTS_NUM_FOR_SPARSE = 2
+
+#environments setting for all processes in cluster job
+LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/lib64"
+```
+
+### Launching Cluster Job
+```paddle.py``` provides automatical scripts to start all PaddlePaddle cluster processes in different nodes. By default, all command line options can set as ```paddle.py``` command options and ```paddle.py``` will transparently and automatically set these options to PaddlePaddle lower level processes.
+
+```paddle.py```provides two distinguished command option for easy job launching.
+
+```job_dispatch_package```  set it with local ```workspace```directory, it will be dispatched to all nodes set in conf.py. It could be helpful for frequent hacking workspace files, otherwise frequent mulit-nodes workspace deployment could make your crazy.
+```job_workspace```  set it with already deployed workspace directory, ```paddle.py``` will skip dispatch stage to directly launch cluster job with all nodes. It could help to reduce heavy
+dispatch latency.
+
+```cluster_train/run.sh``` provides command line sample to run ```demo/recommendation``` cluster job, just modify ```job_dispatch_package``` and ```job_workspace``` with your defined directory, then:
+```
+sh run.sh
+```
+
+The cluster Job will start in several seconds.
+
+### Kill Cluster Job
+```paddle.py``` can capture ```Ctrl + C``` SIGINT signal to automatically kill all processes launched by it. So just stop ```paddle.py``` to kill cluster job. You should mannally kill job if program crashed.
+
+### Check Cluster Training Result
+Check log in $workspace/log for details, each node owns same log structure.
+
+```paddle_trainer.INFO```
+It provides almost all interal output log for training,  same as local training. Check runtime model convergence here.
+
+```paddle_pserver2.INFO```
+It provides pserver running log, which could help to diagnose distributed error.
+
+```server.log```
+It provides stderr and stdout of pserver process. Check error log if training crashs.
+
+```train.log```
+It provides stderr and stdout of trainer process. Check error log if training crashs.
+
+### Check Model Output
+After one pass finished, model files will be writed in ```output``` directory in node 0.
+```nodefile``` in workspace indicates the node id of current cluster job.

release_doc/0.9.0/doc/_sources/demo/embedding_model/index.txt

+# Chinese Word Embedding Model Tutorial #
+----------
+This tutorial is to guide you through the process of using a Pretrained Chinese Word Embedding Model in the PaddlePaddle standard format.
+
+We thank @lipeng for the pull request that defined the model schemas and pretrained the models.
+
+## Introduction ###
+### Chinese Word Dictionary ###
+Our Chinese-word dictionary is created on Baidu ZhiDao and Baidu Baike by using in-house word segmentor. For example, the participle of "《红楼梦》" is "《"，"红楼梦"，"》"，and "《红楼梦》". Our dictionary (using UTF-8 format) has has two columns: word and its frequency. The total word count is 3206325, including 3 special token:
+  - `<s>`: the start of a sequence
+  - `<e>`: the end of a sequence
+  - `<unk>`: a word not included in dictionary
+
+### Pretrained Chinese Word Embedding Model ###
+Inspired by paper [A Neural Probabilistic Language Model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf), our model architecture (**Embedding joint of six words->FullyConnect->SoftMax**) is as following graph. And for our dictionary, we pretrain four models with different word vector dimenstions, i.e 32, 64, 128, 256.
+<center>![](./neural-n-gram-model.png)</center>
+<center>Figure 1. neural-n-gram-model</center>
+
+### Download and Extract ###
+To download and extract our dictionary and pretrained model, run the following commands.
+
+    cd $PADDLE_ROOT/demo/model_zoo/embedding
+    ./pre_DictAndModel.sh
+
+## Chinese Paraphrasing Example ##
+We provide a paraphrasing task to show the usage of pretrained Chinese Word Dictionary and Embedding Model.
+
+### Data Preparation and Preprocess ###
+
+First, run the following commands to download and extract the in-house dataset. The dataset (using UTF-8 format) has 20 training samples, 5 testing samples and 2 generating samples.
+
+    cd $PADDLE_ROOT/demo/seqToseq/data
+    ./paraphrase_data.sh
+
+Second, preprocess data and build dictionary on train data by running the following commands, and the preprocessed dataset is stored in `$PADDLE_SOURCE_ROOT/demo/seqToseq/data/pre-paraphrase`:
+
+    cd $PADDLE_ROOT/demo/seqToseq/
+    python preprocess.py -i data/paraphrase [--mergeDict]
+
+- `--mergeDict`: if using this option, the source and target dictionary are merged, i.e, two dictionaries have the same context. Here, as source and target data are all chinese words, this option can be used.
+
+### User Specified Embedding Model ###
+The general command of extracting desired parameters from the pretrained embedding model based on user dictionary is:
+
+    cd $PADDLE_ROOT/demo/model_zoo/embedding
+    python extract_para.py --preModel PREMODEL --preDict PREDICT --usrModel USRMODEL--usrDict USRDICT -d DIM
+
+- `--preModel PREMODEL`: the name of pretrained embedding model
+- `--preDict PREDICT`: the name of pretrained dictionary
+- `--usrModel USRMODEL`: the name of extracted embedding model
+- `--usrDict USRDICT`: the name of user specified dictionary
+- `-d DIM`: dimension of parameter
+
+Here, you can simply run the command:
+
+    cd $PADDLE_ROOT/demo/seqToseq/data/
+    ./paraphrase_model.sh
+
+And you will see following embedding model structure:
+
+    paraphrase_model
+    |--- _source_language_embedding
+    |--- _target_language_embedding
+
+### Training Model in PaddlePaddle ###
+First, create a model config file, see example `demo/seqToseq/paraphrase/train.conf`:
+
+    from seqToseq_net import *
+    is_generating = False
+
+    ################## Data Definition #####################
+    train_conf = seq_to_seq_data(data_dir = "./data/pre-paraphrase",
+                                 job_mode = job_mode)
+
+    ############## Algorithm Configuration ##################
+    settings(
+          learning_method = AdamOptimizer(),
+          batch_size = 50,
+          learning_rate = 5e-4)
+
+    ################# Network configure #####################
+    gru_encoder_decoder(train_conf, is_generating, word_vector_dim = 32)
+
+This config is almost the same as `demo/seqToseq/translation/train.conf`.
+
+Then, train the model by running the command:
+
+    cd $PADDLE_SOURCE_ROOT/demo/seqToseq/paraphrase
+    ./train.sh
+
+where `train.sh` is almost the same as `demo/seqToseq/translation/train.sh`, the only difference is following two command arguments:
+
+- `--init_model_path`: path of the initialization model, here is `data/paraphrase_model`
+- `--load_missing_parameter_strategy`: operations when model file is missing, here use a normal distibution to initialize the other parameters except for the embedding layer
+
+For users who want to understand the dataset format, model architecture and training procedure in detail, please refer to [Text generation Tutorial](../text_generation/text_generation.md).
+
+## Optional Function ##
+###  Embedding Parameters Observation
+For users who want to observe the embedding parameters, this function can convert a PaddlePaddle binary embedding model to a text model by running the command:
+
+    cd $PADDLE_ROOT/demo/model_zoo/embedding
+    python paraconvert.py --b2t -i INPUT -o OUTPUT -d DIM
+
+- `-i INPUT`: the name of input binary embedding model
+- `-o OUTPUT`: the name of output text embedding model
+- `-d DIM`: the dimension of parameter
+
+You will see parameters like this in output text model:
+
+    0,4,32156096
+    -0.7845433,1.1937413,-0.1704215,0.4154715,0.9566584,-0.5558153,-0.2503305, ......
+    0.0000909,0.0009465,-0.0008813,-0.0008428,0.0007879,0.0000183,0.0001984, ......
+    ......
+
+- 1st line is **PaddlePaddle format file head**, it has 3 attributes:
+  - version of PaddlePaddle, here is 0
+  - sizeof(float), here is 4
+  - total number of parameter, here is 32156096
+- Other lines print the paramters (assume `<dim>` = 32)
+  - each line print 32 paramters splitted by ','
+  - there is 32156096/32 = 1004877 lines, meaning there is 1004877 embedding words
+
+### Embedding Parameters Revision
+For users who want to revise the embedding parameters, this function can convert a revised text embedding model to a PaddlePaddle binary model by running the command:
+
+    cd $PADDLE_ROOT/demo/model_zoo/embedding
+    python paraconvert.py --t2b -i INPUT -o OUTPUT
+
+- `-i INPUT`: the name of input text embedding model.
+- `-o OUTPUT`: the name of output binary embedding model
+
+Note that the format of input text model is as follows:
+
+    -0.7845433,1.1937413,-0.1704215,0.4154715,0.9566584,-0.5558153,-0.2503305, ......
+    0.0000909,0.0009465,-0.0008813,-0.0008428,0.0007879,0.0000183,0.0001984, ......
+    ......
+- there is no file header in 1st line
+- each line stores parameters for one word, the separator is commas ','

release_doc/0.9.0/doc/_sources/demo/image_classification/image_classification.txt

+Image Classification Tutorial
+==============================
+
+This tutorial will guide you through training a convolutional neural network to classify objects using the CIFAR-10 image classification dataset.
+As shown in the following figure, the convolutional neural network can recognize the main object in images, and output the classification result.
+
+<center>![Image Classification](./image_classification.png)</center>
+
+## Data Preparation
+First, download CIFAR-10 dataset. CIFAR-10 dataset can be downloaded from its official website.
+
+<https://www.cs.toronto.edu/~kriz/cifar.html>
+
+We have prepared a script to download and process CIFAR-10 dataset. The script will download CIFAR-10 dataset from the official dataset.
+It will convert it to jpeg images and organize them into a directory with the required structure for the tutorial. Make sure that you have installed pillow and its dependents.
+Consider the following commands:
+
+1. install pillow dependents
+
+```bash
+sudo apt-get install libjpeg-dev
+pip install pillow
+```
+
+2. download data and preparation
+
+```bash
+cd demo/image_classification/data/
+sh download_cifar.sh
+```
+
+The CIFAR-10 dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
+
+Here are the classes in the dataset, as well as 10 random images from each:
+<center>![Image Classification](./cifar.png)</center>
+
+
+After downloading and converting, we should find a directory (cifar-out) containing the dataset in the following format:
+
+```
+train
+---airplane
+---automobile
+---bird
+---cat
+---deer
+---dog
+---frog
+---horse
+---ship
+---truck
+test
+---airplane
+---automobile
+---bird
+---cat
+---deer
+---dog
+---frog
+---horse
+---ship
+---truck
+```
+
+It has two directories:`train` and `test`. These two directories contain training data and testing data of CIFAR-10, respectively. Each of these two folders contains 10 sub-folders, ranging from `airplane` to `truck`. Each sub-folder contains images with the corresponding label. After the images are organized into this structure, we are ready to train an image classification model.
+
+## Preprocess
+After the data has been downloaded, it needs to be pre-processed into the Paddle format. We can run the following command for preprocessing.
+
+```
+cd demo/image_classification/
+sh preprocess.sh
+```
+
+`preprocess.sh` calls `./demo/image_classification/preprocess.py` to preprocess image data.
+```sh
+export PYTHONPATH=$PYTHONPATH:../../
+data_dir=./data/cifar-out
+python preprocess.py -i $data_dir -s 32 -c 1
+```
+
+`./demo/image_classification/preprocess.py` has the following arguments
+
+- `-i` or `--input` specifes  the input data directory.
+- `-s` or `--size` specifies the processed size of images.
+- `-c` or `--color` specifes whether images are color images or gray images.
+
+
+## Model Training
+We need to create a model config file before training the model. An example of the config file (vgg_16_cifar.py) is listed below. **Note**, it is slightly different from the `vgg_16_cifar.py` which also applies to the prediction.
+
+```python
+from paddle.trainer_config_helpers import *
+data_dir='data/cifar-out/batches/'
+meta_path=data_dir+'batches.meta'
+args = {'meta':meta_path, 'mean_img_size': 32,
+        'img_size': 32, 'num_classes': 10,
+        'use_jpeg': 1, 'color': "color"}
+define_py_data_sources2(train_list=data_dir+"train.list",
+                        test_list=data_dir+'test.list',
+                        module='image_provider',
+                        obj='processData',
+                        args=args)
+settings(
+    batch_size = 128,
+    learning_rate = 0.1 / 128.0,
+    learning_method = MomentumOptimizer(0.9),
+    regularization = L2Regularization(0.0005 * 128))
+
+img = data_layer(name='image', size=3*32*32)
+lbl = data_layer(name="label", size=10)
+# small_vgg is predined in trainer_config_helpers.network
+predict = small_vgg(input_image=img, num_channels=3)
+outputs(classification_cost(input=predict, label=lbl))
+```
+
+The first line imports python functions for defining networks.
+```python
+from paddle.trainer_config_helpers import *
+```
+
+Then define an `define_py_data_sources2` which use python data provider
+interface. The arguments in `args` are used in `image_provider.py` which
+yeilds image data and transform them to Paddle.
+ - `meta`: the mean value of training set.
+ - `mean_img_size`: the size of mean feature map.
+ - `img_size`: the height and width of input image.
+ - `num_classes`: the number of classes.
+ - `use_jpeg`: the data storage type when preprocessing.
+ - `color`: specify color image.
+
+`settings` specifies the training algorithm. In the following example,
+it specifies learning rate as 0.1, but divided by batch size, and the weight decay
+is 0.0005 and multiplied by batch size.
+```python
+settings(
+    batch_size = 128,
+    learning_rate = 0.1 / 128.0,
+    learning_method = MomentumOptimizer(0.9),
+    regularization = L2Regularization(0.0005 * 128)
+)
+```
+
+The `small_vgg` specifies the network. We use a small version of VGG convolutional network as our network
+for classification. A description of VGG network can be found here [http://www.robots.ox.ac.uk/~vgg/research/very_deep/](http://www.robots.ox.ac.uk/~vgg/research/very_deep/).
+```python
+# small_vgg is predined in trainer_config_helpers.network
+predict = small_vgg(input_image=img, num_channels=3)
+```
+After writing the config, we can train the model by running the script train.sh. Notice that the following script assumes the you run the script in the `./demo/image_classification` folder. If you run the script in a different folder, you need to change the paths of the scripts and the configuration files accordingly.
+
+```bash
+config=vgg_16_cifar.py
+output=./cifar_vgg_model
+log=train.log
+
+paddle train \
+--config=$config \
+--dot_period=10 \
+--log_period=100 \
+--test_all_data_in_one_period=1 \
+--use_gpu=1 \
+--save_dir=$output \
+2>&1 | tee $log
+
+python -m paddle.utils.plotcurve -i $log > plot.png
+```
+
+- Here we use GPU mode to train. If you have no gpu environment, just set `use_gpu=0`.
+
+- `./demo/image_classification/vgg_16_cifar.py` is the network and data configuration file. The meaning of the other flags can be found in the documentation of the command line flags.
+
+- The script `plotcurve.py` requires the python module of `matplotlib`, so if it fails, maybe you need to install `matplotlib`.
+
+
+After training finishes, the training and testing error curves will be saved to `plot.png` using `plotcurve.py` script. An example of the plot is shown below:
+
+<center>![Training and testing curves.](./plot.png)</center>
+
+
+## Prediction
+After we train the model, the model file as well as the model parameters are stored in path `./cifar_vgg_model/pass-%05d`. For example, the model of the 300-th pass is stored at `./cifar_vgg_model/pass-00299`.
+
+To make a prediction for an image, one can run `predict.sh` as follows. The script will output the label of the classfiication.
+
+```
+sh predict.sh
+```
+
+predict.sh:
+```
+model=cifar_vgg_model/pass-00299/
+image=data/cifar-out/test/airplane/seaplane_s_000978.png
+use_gpu=1
+python prediction.py $model $image $use_gpu
+```
+
+## Exercise
+Train a image classification of birds using VGG model and CUB-200 dataset. The birds dataset can be downloaded here. It contains an image dataset with photos of 200 bird species (mostly North American).
+
+<http://www.vision.caltech.edu/visipedia/CUB-200.html>
+
+
+
+
+## Delve into Details
+### Convolutional Neural Network
+A Convolutional Neural Network is a feedforward neural network that uses convolution layers. It is very suitable for building neural networks that process and understand images. A standard convolutional neural network is shown below:
+
+![Convolutional Neural Network](./lenet.png)
+
+Convolutional Neural Network contains the following layers:
+
+- Convolutional layer: It uses convolution operation to extract features from an image or a feature map.
+- Pooling layer: It uses max-pooling to downsample feature maps.
+- Fully Connected layer: It uses fully connected connections to transform features.
+
+Convolutional Neural Network achieves amazing performance for image classification because it exploits two important characteristics of images: *local correlation* and *spatial invariance*. By iteratively applying convolution and max-pooing operations, convolutional neural network can well represent these two characteristics of images.
+
+
+For more details of how to define layers and their connections, please refer to the documentation of layers.

release_doc/0.9.0/doc/_sources/demo/image_classification/index.txt

+Image Classification Tutorial
+=============================
+
+.. toctree::
+    :maxdepth: 3
+    :glob:
+
+    Training Locally <image_classification.md>
+    cluster_train/internal/cluster_train.md
+    cluster_train/opensource/cluster_train.md

release_doc/0.9.0/doc/_sources/demo/imagenet_model/resnet_model.txt

+# Model Zoo - ImageNet #
+
+[ImageNet](http://www.image-net.org/) is a popular dataset for generic object classification. This tutorial provides convolutional neural network(CNN) models for ImageNet.
+
+## ResNet Introduction
+
+ResNets from paper [Deep Residual Learning for Image Recognition](http://arxiv.org/abs/1512.03385) won the 1st place on the ILSVRC 2015 classification task. They present residual learning framework to ease the training of networks that are substantially deeper than those used previously. The residual connections are shown in following figure. The left building block is used in network of 34 layers and the right bottleneck building block is used in network of 50, 101, 152 layers .
+
+<center>![resnet_block](./resnet_block.jpg)</center>
+<center>Figure 1. ResNet Block</center>
+
+We present three ResNet models, which are converted from the models provided by the authors <https://github.com/KaimingHe/deep-residual-networks>.  The classfication errors tested in PaddlePaddle on 50,000 ILSVRC validation set with input images channel order of **BGR** by single scale with the shorter side of 256 and single crop as following table.
+<center>
+<table border="2" cellspacing="0" cellpadding="6" rules="all" frame="border">
+<colgroup>
+<col  class="left" />
+<col  class="left" />
+<col  class="left" />
+</colgroup>
+<thead>
+<tr>
+<th scope="col" class="left">ResNet</th>
+<th scope="col" class="left">Top-1</th>
+<th scope="col" class="left">Model Size</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td class="left">ResNet-50</td>
+<td class="left">24.9%</td>
+<td class="left">99M</td>
+</tr>
+<tr>
+<td class="left">ResNet-101</td>
+<td class="left">23.7%</td>
+<td class="left">173M</td>
+</tr>
+<tr>
+<td class="left">ResNet-152</td>
+<td class="left">23.2%</td>
+<td class="left">234M</td>
+</tr>
+</tbody>
+
+</table></center>
+<br>
+
+## ResNet Model
+
+See ```demo/model_zoo/resnet/resnet.py```. This config contains network of 50, 101 and 152 layers. You can specify layer number by adding argument like ```--config_args=layer_num=50``` in command line arguments.
+
+### Network Visualization
+
+You can get a diagram of ResNet network by running the following commands. The script generates dot file and then converts dot file to PNG file, which uses installed draw_dot tool in our server. If you can not access the server, just install graphviz to convert dot file.
+
+```
+cd demo/model_zoo/resnet
+./net_diagram.sh
+```
+
+### Model Download
+
+```
+cd demo/model_zoo/resnet
+./get_model.sh
+```
+You can run above command to download all models and mean file and save them in ```demo/model_zoo/resnet/model``` if downloading successfully.
+
+```
+mean_meta_224  resnet_101  resnet_152  resnet_50
+```
+   * resnet_50: model of 50 layers.
+   * resnet_101: model of 101 layers.
+   * resnet_152: model of 152 layers.
+   * mean\_meta\_224: mean file with 3 x 224 x 224 size in **BGR** order. You also can use three mean values: 103.939, 116.779, 123.68.
+
+### Parameter Info
+
+* **Convolution Layer Weight**
+
+  As batch normalization layer is connected after each convolution layer, there is no parameter of bias and only one weight in this layer.
+  shape: `(Co, ky, kx, Ci)`
+   * Co: channle number of output feature map.
+   * ky: filter size in vertical direction.
+   * kx: filter size in horizontal direction.
+   * Ci: channle number of input feature map.
+
+  2-Dim matrix: (Co * ky * kx, Ci), saved in row-major order.
+
+* **Fully connected Layer Weight**
+
+  2-Dim matrix: (input layer size, this layer size), saved in row-major order.
+
+* **[Batch Normalization](<http://arxiv.org/abs/1502.03167>) Layer Weight**
+
+There are four parameters in this layer. In fact, only .w0 and .wbias are the learned parameters. The other two are therunning mean and variance respectively. They will be loaded in testing. Following table shows parameters of a batch normzalization layer.
+<center>
+<table border="2" cellspacing="0" cellpadding="6" rules="all" frame="border">
+<colgroup>
+<col  class="left" />
+<col  class="left" />
+<col  class="left" />
+</colgroup>
+<thead>
+<tr>
+<th scope="col" class="left">Parameter Name</th>
+<th scope="col" class="left">Number</th>
+<th scope="col" class="left">Meaning</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td class="left">_res2_1_branch1_bn.w0</td>
+<td class="left">256</td>
+<td class="left">gamma, scale parameter</td>
+</tr>
+<tr>
+<td class="left">_res2_1_branch1_bn.w1</td>
+<td class="left">256</td>
+<td class="left">mean value of feature map</td>
+</tr>
+<tr>
+<td class="left">_res2_1_branch1_bn.w2</td>
+<td class="left">256</td>
+<td class="left">variance of feature map</td>
+</tr>
+<tr>
+<td class="left">_res2_1_branch1_bn.wbias</td>
+<td class="left">256</td>
+<td class="left">beta, shift parameter</td>
+</tr>
+</tbody>
+
+</table></center>
+<br>
+
+### Parameter Observation
+
+Users who want to observe the parameters can use python to read:
+
+```
+import sys
+import numpy as np
+
+def load(file_name):
+    with open(file_name, 'rb') as f:
+        f.read(16) # skip header for float type.
+        return np.fromfile(f, dtype=np.float32)
+
+if __name__=='__main__':
+    weight = load(sys.argv[1])
+```
+
+or simply use following shell command:
+
+```
+od -j 16 -f _res2_1_branch1_bn.w0
+```
+
+## Feature Extraction
+
+We provide both C++ and Python interfaces to extract features. The following examples use data in `demo/model_zoo/resnet/example` to show the extracting process in detail.
+
+### C++ Interface
+
+First, specify image data list in `define_py_data_sources2` in the config, see example `demo/model_zoo/resnet/resnet.py`.
+
+```
+    train_list = 'train.list' if not is_test else None
+    # mean.meta is mean file of ImageNet dataset.
+    # mean.meta size : 3 x 224 x 224.
+    # If you use three mean value, set like:
+    # "mean_value:103.939,116.779,123.68;"
+    args={
+        'mean_meta': "model/mean_meta_224/mean.meta",
+        'image_size': 224, 'crop_size': 224,
+        'color': True,'swap_channel:': [2, 1, 0]}
+    define_py_data_sources2(train_list,
+                           'example/test.list',
+                           module="example.image_list_provider",
+                           obj="processData",
+                           args=args)
+```
+
+Second, specify layers to extract features in `Outputs()` of `resnet.py`. For example,
+
+```
+Outputs("res5_3_branch2c_conv", "res5_3_branch2c_bn")
+```
+
+Third, specify model path and output directory in `extract_fea_c++.sh`, and then run the following commands.
+
+```
+cd demo/model_zoo/resnet
+./extract_fea_c++.sh
+```
+
+If successful, features are saved in `fea_output/rank-00000` as follows. And you can use `load_feature_c` interface in `load_feature.py ` to load such a file.
+
+```
+-0.115318 -0.108358 ... -0.087884;-1.27664 ... -1.11516 -2.59123;
+-0.126383 -0.116248 ... -0.00534909;-1.42593 ... -1.04501 -1.40769;
+```
+
+* Each line stores features of a sample. Here, the first line stores features of `example/dog.jpg` and second line stores features of `example/cat.jpg`.
+* Features of different layers are splitted by `;`, and their order is consistent with the layer order in `Outputs()`. Here, the left features are `res5_3_branch2c_conv` layer and right features are `res5_3_branch2c_bn` layer.
+
+### Python Interface
+
+`demo/model_zoo/resnet/classify.py` is an example to show how to use python to extract features. Following example still uses data of `./example/test.list`. Command is as follows:
+
+```
+cd demo/model_zoo/resnet
+./extract_fea_py.sh
+```
+
+extract_fea_py.sh:
+
+```
+python classify.py \
+     --job=extract \
+     --conf=resnet.py\
+     --use_gpu=1 \
+     --mean=model/mean_meta_224/mean.meta \
+     --model=model/resnet_50 \
+     --data=./example/test.list \
+     --output_layer="res5_3_branch2c_conv,res5_3_branch2c_bn" \
+     --output_dir=features
+
+```
+* \--job=extract:              specify job mode to extract feature.
+* \--conf=resnet.py:           network configure.
+* \--use_gpu=1:             speficy GPU mode.
+* \--model=model/resnet_5:     model path.
+* \--data=./example/test.list: data list.
+* \--output_layer="xxx,xxx":   specify layers to extract features.
+* \--output_dir=features:      output diretcoty.
+
+Note, since the convolution layer in these ResNet models is suitable for the cudnn implementation which only support GPU. It not support CPU mode because of compatibility issue and we will fix later.
+
+If run successfully, you will see features saved in `features/batch_0`, this file is produced with cPickle. You can use `load_feature_py` interface in `load_feature.py` to open the file, and it returns a dictionary as follows:
+
+```
+{
+'cat.jpg': {'res5_3_branch2c_conv': array([[-0.12638293, -0.116248  , -0.11883899, ..., -0.00895038, 0.01994277, -0.00534909]], dtype=float32), 'res5_3_branch2c_bn': array([[-1.42593431, -1.28918779, -1.32414699, ..., -1.45933616, -1.04501402, -1.40769434]], dtype=float32)},
+'dog.jpg': {'res5_3_branch2c_conv': array([[-0.11531784, -0.10835785, -0.08809858, ...,0.0055237, 0.01505112, -0.08788397]], dtype=float32), 'res5_3_branch2c_bn': array([[-1.27663755, -1.18272924, -0.90937918, ..., -1.25178063, -1.11515927, -2.59122872]], dtype=float32)}
+}
+```
+
+Observed carefully, these feature values are consistent with the above results extracted by C++ interface.
+
+## Prediction
+
+`classify.py` also can be used to predict. We provide an example script `predict.sh` to predict data in `example/test.list` using a ResNet model with 50 layers.
+
+```
+cd demo/model_zoo/resnet
+./predict.sh
+```
+
+predict.sh calls the `classify.py`:
+
+```
+python classify.py \
+     --job=predict \
+     --conf=resnet.py\
+     --multi_crop \
+     --model=model/resnet_50 \
+     --use_gpu=1 \
+     --data=./example/test.list
+```
+* \--job=extract:              speficy job mode to predict.
+* \--conf=resnet.py:           network configure.
+* \--multi_crop:               use 10 crops and average predicting probability.
+* \--use_gpu=1:             speficy GPU mode.
+* \--model=model/resnet_50:    model path.
+* \--data=./example/test.list: data list.
+
+If run successfully, you will see following results, where 156 and 285 are labels of the images.
+
+```
+Label of example/dog.jpg is: 156
+Label of example/cat.jpg is: 282
+```

release_doc/0.9.0/doc/_sources/demo/index.txt

+# Examples and demos
+There are serveral examples and demos here.
+
+## Image
+
+* [Image Classification](image_classification/index.rst)
+
+## NLP
+
+* [Sentiment Analysis](sentiment_analysis/index.rst)
+* [Text Generation](text_generation/index.rst)
+* [Semantic Role Labeling](semantic_role_labeling/index.rst)
+
+## Recommendation
+
+* [MovieLens Dataset](rec/ml_dataset.md)
+* [MovieLens Regression](rec/ml_regression.rst)
+
+## Model Zoo
+* [ImageNet: ResNet](imagenet_model/resnet_model.md)
+* [Embedding: Chinese Word](embedding_model/index.md)

release_doc/0.9.0/doc/_sources/demo/rec/ml_dataset.txt

+# MovieLens Dataset
+
+The [MovieLens Dataset](http://grouplens.org/datasets/movielens/) was collected by GroupLens Research.
+The data set contains some user information, movie information, and many movie ratings from \[1-5\].
+The data sets have many version depending on the size of set.
+We use [MovieLens 1M Dataset](http://files.grouplens.org/datasets/movielens/ml-1m.zip) as a demo dataset, which contains
+1 million ratings from 6000 users on 4000 movies. Released 2/2003.
+
+## Dataset Features
+
+In [ml-1m Dataset](http://files.grouplens.org/datasets/movielens/ml-1m.zip), there are many features in these dataset.
+The data files (which have ".dat" extension) in [ml-1m Dataset](http://files.grouplens.org/datasets/movielens/ml-1m.zip)
+is basically CSV file that delimiter is "::". The description in README we quote here.
+
+### RATINGS FILE DESCRIPTION(ratings.dat)
+
+
+All ratings are contained in the file "ratings.dat" and are in the
+following format:
+
+UserID::MovieID::Rating::Timestamp
+
+- UserIDs range between 1 and 6040
+- MovieIDs range between 1 and 3952
+- Ratings are made on a 5-star scale (whole-star ratings only)
+- Timestamp is represented in seconds since the epoch as returned by time(2)
+- Each user has at least 20 ratings
+
+### USERS FILE DESCRIPTION(users.dat)
+
+User information is in the file "users.dat" and is in the following
+format:
+
+UserID::Gender::Age::Occupation::Zip-code
+
+All demographic information is provided voluntarily by the users and is
+not checked for accuracy.  Only users who have provided some demographic
+information are included in this data set.
+
+- Gender is denoted by a "M" for male and "F" for female
+- Age is chosen from the following ranges:
+
+	*  1:  "Under 18"
+	* 18:  "18-24"
+	* 25:  "25-34"
+	* 35:  "35-44"
+	* 45:  "45-49"
+	* 50:  "50-55"
+	* 56:  "56+"
+
+- Occupation is chosen from the following choices:
+
+	*  0:  "other" or not specified
+	*  1:  "academic/educator"
+	*  2:  "artist"
+	*  3:  "clerical/admin"
+	*  4:  "college/grad student"
+	*  5:  "customer service"
+	*  6:  "doctor/health care"
+	*  7:  "executive/managerial"
+	*  8:  "farmer"
+	*  9:  "homemaker"
+	* 10:  "K-12 student"
+	* 11:  "lawyer"
+	* 12:  "programmer"
+	* 13:  "retired"
+	* 14:  "sales/marketing"
+	* 15:  "scientist"
+	* 16:  "self-employed"
+	* 17:  "technician/engineer"
+	* 18:  "tradesman/craftsman"
+	* 19:  "unemployed"
+	* 20:  "writer"
+
+### MOVIES FILE DESCRIPTION(movies.dat)
+
+Movie information is in the file "movies.dat" and is in the following
+format:
+
+MovieID::Title::Genres
+
+- Titles are identical to titles provided by the IMDB (including
+year of release)
+- Genres are pipe-separated and are selected from the following genres:
+
+	* Action
+	* Adventure
+	* Animation
+	* Children's
+	* Comedy
+	* Crime
+	* Documentary
+	* Drama
+	* Fantasy
+	* Film-Noir
+	* Horror
+	* Musical
+	* Mystery
+	* Romance
+	* Sci-Fi
+	* Thriller
+	* War
+	* Western
+
+- Some MovieIDs do not correspond to a movie due to accidental duplicate
+entries and/or test entries
+- Movies are mostly entered by hand, so errors and inconsistencies may exist

release_doc/0.9.0/doc/_sources/demo/rec/ml_regression.txt

+Regression MovieLens Ratting
+============================
+
+Here we demonstrate a **Cosine Similarity Regression** job in movie lens dataset.
+This demo will show how paddle does (word) embedding job,
+handles the similarity regression,
+the character-level convolutional networks for text, and how does paddle handle
+multiple types of inputs.
+Note that the model structure is not fine-tuned and just a demo to show how paddle works.
+
+
+YOU ARE WELCOME TO BUILD A BETTER DEMO
+BY USING PADDLEPADDLE, AND LET US KNOW TO MAKE THIS DEMO BETTER.
+
+Data Preparation
+````````````````
+Download and extract dataset
+''''''''''''''''''''''''''''
+We use `movielens 1m dataset <ml_dataset.html>`_ here. 
+To download and unzip the dataset, simply run the following commands.
+
+..  code-block:: bash
+
+    cd demo/recommendation/data 
+    ./ml_data.sh
+
+And the directory structure of :code:`demo/recommendation/data/ml-1m` is:
+
+..  code-block:: text
+
+    +--ml-1m
+         +--- movies.dat    # movie features
+         +--- ratings.dat   # ratings
+         +--- users.dat     # user features
+         +--- README        # dataset description
+
+Field config file
+'''''''''''''''''
+**Field config file** is used to specific the fields dataset and file format,
+i.e, specific **WHAT** type it is in each feature file.
+
+The field config file of ml-1m shows in :code:`demo/recommendation/data/config.json`.
+It specifics the field types and file names: 1) there are four types of field for user file\: id, gender, age and occupation;
+2) the filename is "users.dat", and the delimiter of file is "::".
+
+..  include:: ../../../demo/recommendation/data/config.json
+    :code: json
+    :literal:
+
+Preprocess Data
+```````````````
+You need to install python 3rd party libraries.
+IT IS HIGHLY RECOMMEND TO USE VIRTUALENV MAKE A CLEAN PYTHON ENVIRONMENT.
+
+..  code-block:: bash
+
+    pip install -r requirements.txt
+
+The general command for preprocessing the dataset is:
+
+..  code-block:: bash
+
+    cd demo/recommendation
+    ./preprocess.sh
+    
+And the detail steps are introduced as follows.
+
+Extract Movie/User features to python object
+'''''''''''''''''''''''''''''''''''''''''''''
+
+There are many features in movie or user in movielens 1m dataset.
+Each line of rating file just provides a Movie/User id to refer each movie or user.
+We process the movie/user feature file first, and pickle the feature (**Meta**) object as a file.
+
+Meta config file
+................
+
+**Meta config file** is used to specific **HOW** to parse each field in dataset.
+It could be translated from field config file, or written by hand.
+Its file format could be either json or yaml syntax file. Parser will automatically choose the file format by extension name.
+
+To convert Field config file to meta config file, just run:
+
+..  code-block:: bash
+
+    cd demo/recommendation/data
+    python config_generator.py config.json > meta_config.json
+
+The meta config file shows below:
+
+..  include:: ../../../demo/recommendation/data/meta_config.json
+    :code: json
+    :literal:
+
+There are two kinds of features in meta\: movie and user.
+
+* in movie file, whose name is movies.dat
+   * we just split each line by "::"
+   * pos 0 is id.
+   * pos 1 feature:
+      * name is title.
+      * it uses regex to parse this feature.
+      * it is a char based word embedding feature.
+      * it is a sequence.
+   * pos 2 feature:
+      * name is genres.
+      * type is one hot dense vector.
+      * dictionary is auto generated by parsing, each key is split by '|'
+* in user file, whose name is users.dat
+   * we just split each line by "::"
+   * pos 0 is id.
+   * pos 1 feature:
+       * name is gender
+       * just simple char based embedding.
+   * pos 2 feature:
+       * name is age
+       * just whole word embedding.
+       * embedding id will be sort by word.
+   * pos 3 feature:
+       * name is occupation.
+       * just simple whole word embedding.
+
+
+Meta file
+'''''''''
+
+After having meta config file, we can generate **Meta file**, a python pickle object which stores movie/user information.
+The following commands could be run to generate it.
+
+..  code-block:: bash
+
+    python meta_generator.py ml-1m meta.bin --config=meta_config.json
+
+And the structure of the meta file :code:`meta.bin` is:
+
+..  code-block:: text
+
+    +--+ movie
+    |      +--+ __meta__
+    |      |       +--+ raw_meta  # each feature meta config. list
+    |      |       |       +
+    |      |       |       |     # ID Field, we use id as key
+    |      |       |       +--+ {'count': 3883, 'max': 3952, 'is_key': True, 'type': 'id', 'min': 1}
+    |      |       |       |
+    |      |       |       |     # Titile field, the dictionary list of embedding.
+    |      |       |       +--+ {'dict': [ ... ], 'type': 'embedding', 'name': 'title', 'seq': 'sequence'}
+    |      |       |       |
+    |      |       |       |     # Genres field, the genres dictionary
+    |      |       |       +--+ {'dict': [ ... ], 'type': 'one_hot_dense', 'name': 'genres'}
+    |      |       |
+    |      |       +--+ feature_map [1, 2] # a list for raw_meta index for feature field.
+    |      |                               # it means there are 2 features for each key.
+    |      |                               #    * 0 offset of feature is raw_meta[1], Title.
+    |      |                               #    * 1 offset of feature is raw_meta[2], Genres.
+    |      |
+    |      +--+ 1 # movie 1 features
+    |      |    +
+    |      |    +---+ [[...], [...]] # title ids, genres dense vector
+    |      |
+    |      +--+ 2
+    |      |
+    |      +--+ ...
+    |
+    +--- user
+           +--+ __meta__
+           |       +
+           |       +--+ raw_meta
+           |       |       +
+           |       |       +--+ id field as user
+           |       |       |
+           |       |       +--+ {'dict': ['F', 'M'], 'type': 'embedding', 'name': 'gender', 'seq': 'no_sequence'}
+           |       |       |
+           |       |       +--+ {'dict': ['1', '18', '25', '35', '45', '50', '56'], 'type': 'embedding', 'name': 'age', 'seq': 'no_sequence'}
+           |       |       |
+           |       |       +--+ {'dict': [...], 'type': 'embedding', 'name': 'occupation', 'seq': 'no_sequence'}
+           |       |
+           |       +--+ feature_map [1, 2, 3]
+           |
+           +--+ 1 # user 1 features
+           |
+           +--+ 2
+           +--+ ...
+
+
+Split Training/Testing files
+''''''''''''''''''''''''''''
+
+We split :code:`ml-1m/ratings.dat` into a training and testing file. The way to split file is for each user, we split the
+rating by two parts. So each user in testing file will have some rating information in training file.
+
+Use separate.py to separate the training and testing file.
+
+..  code-block:: bash
+
+    python split.py ml-1m/ratings.dat --delimiter="::" --test_ratio=0.1
+
+Then two files will be generated\: :code:`ml-1m/ratings.dat.train` and :code:`ml-1m/rating.data.test`.
+Move them to workspace :code:`data`, shuffle the train file, and prepare the file list for paddle train.
+
+..  code-block:: bash
+
+    shuf ml-1m/ratings.dat.train > ratings.dat.train
+    cp ml-1m/ratings.dat.test .
+    echo "./data/ratings.dat.train" > train.list
+    echo "./data/ratings.dat.test" > test.list
+
+
+Neural Network Configuration
+````````````````````````````
+
+Trainer Config File
+'''''''''''''''''''
+
+The network structure shows below.
+
+..  image:: rec_regression_network.png
+    :align: center
+    :alt: rec_regression_network
+
+The demo's neural network config file "trainer_config.py" show as below.
+
+..  literalinclude:: ../../../demo/recommendation/trainer_config.py
+    :language: python
+    :lines: 15-
+
+In this :code:`trainer_config.py`, we just map each feature type to
+a feature vector, following shows how to map each feature to a vector shows below.
+
+* :code:`id`\: Just simple embedding, and then add to fully connected layer.
+* :code:`embedding`\:
+    - if is_sequence, get the embedding and do a text convolutional operation,
+      get the average pooling result.
+    - if not sequence, get the embedding and add to fully connected layer.
+* :code:`one_host_dense`\:
+    - just two fully connected layer.
+
+Then we combine each features of movie into one movie feature by a
+:code:`fc_layer` with multiple inputs, and do the same thing to user features,
+get one user feature. Then we calculate the cosine similarity of these two
+features.
+
+In these network, we use several api in `trainer_config_helpers
+<../../ui/api/trainer_config_helpers/index.html>`_. There are
+
+*  Data Layer, `data_layer 
+   <../../ui/api/trainer_config_helpers/layers.html#id1>`_
+*  Fully Connected Layer, `fc_layer
+   <../../ui/api/trainer_config_helpers/layers.html#fc-layer>`_
+*  Embedding Layer, `embedding_layer
+   <../../ui/api/trainer_config_helpers/layers.html#embedding-layer>`_
+*  Context Projection Layer, `context_projection
+   <../../ui/api/trainer_config_helpers/layers.html#context-projection>`_
+*  Pooling Layer, `pooling_layer
+   <../../ui/api/trainer_config_helpers/layers.html#pooling-layer>`_
+*  Cosine Similarity Layer, `cos_sim
+   <../../ui/api/trainer_config_helpers/layers.html#cos-sim>`_
+*  Text Convolution Pooling Layer, `text_conv_pool
+   <../../ui/api/trainer_config_helpers/networks.html
+   #trainer_config_helpers.networks.text_conv_pool>`_
+*  Declare Python Data Sources, `define_py_data_sources2
+   <../../ui/api/trainer_config_helpers/data_sources.html>`_
+
+Data Provider
+'''''''''''''
+
+..  literalinclude:: ../../../demo/recommendation/dataprovider.py
+    :language: python
+    :lines: 15-
+
+The data provider just read the meta.bin and rating file, yield each sample for training.
+In this :code:`dataprovider.py`, we should set\:
+
+* obj.slots\: The feature types and dimension.
+* use_seq\: Whether this :code:`dataprovider.py` in sequence mode or not.
+* process\: Return each sample of data to :code:`paddle`.
+
+The data provider details document see `there <../../ui/data_provider/pydataprovider2.html>`_.
+
+Train
+`````
+
+After prepare data, config network, writting data provider, now we can run paddle training.
+
+The run.sh is shown as follow:
+
+..  literalinclude:: ../../../demo/recommendation/run.sh
+    :language: bash
+    :lines: 16-
+
+It just start a paddle training process, write the log to `log.txt`,
+then print it on screen.
+
+Each command line argument in :code:`run.sh`, please refer to the `command line
+arguments <../../ui/index.html#command-line-argument>`_ page. The short description of these arguments is shown as follow.
+
+*  config\: Tell paddle which file is neural network configuration.
+*  save_dir\: Tell paddle save model into './output'
+*  use_gpu\: Use gpu or not. Default is false.
+*  trainer_count\: The compute thread in one machine.
+*  test_all_data_in_one_period\: Test All Data during one test period. Otherwise,
+   will test a :code:`batch_size` data in one test period.
+*  log_period\: Print log after train :code:`log_period` batches.
+*  dot_period\: Print a :code:`.` after train :code:`dot_period` batches.
+*  num_passes\: Train at most :code:`num_passes`.
+
+If training process starts successfully, the output likes follow:
+
+..  code-block:: text
+
+    I0601 08:07:22.832059 10549 TrainerInternal.cpp:157]  Batch=100 samples=160000 AvgCost=4.13494 CurrentCost=4.13494 Eval:  CurrentEval:
+
+    I0601 08:07:50.672627 10549 TrainerInternal.cpp:157]  Batch=200 samples=320000 AvgCost=3.80957 CurrentCost=3.48421 Eval:  CurrentEval:
+
+    I0601 08:08:18.877369 10549 TrainerInternal.cpp:157]  Batch=300 samples=480000 AvgCost=3.68145 CurrentCost=3.42519 Eval:  CurrentEval:
+
+    I0601 08:08:46.863963 10549 TrainerInternal.cpp:157]  Batch=400 samples=640000 AvgCost=3.6007 CurrentCost=3.35847 Eval:  CurrentEval:
+
+    I0601 08:09:15.413025 10549 TrainerInternal.cpp:157]  Batch=500 samples=800000 AvgCost=3.54811 CurrentCost=3.33773 Eval:  CurrentEval:
+    I0601 08:09:36.058670 10549 TrainerInternal.cpp:181]  Pass=0 Batch=565 samples=902826 AvgCost=3.52368 Eval:
+    I0601 08:09:46.215489 10549 Tester.cpp:101]  Test samples=97383 cost=3.32155 Eval:
+    I0601 08:09:46.215966 10549 GradientMachine.cpp:132] Saving parameters to ./output/model/pass-00000
+    I0601 08:09:46.233397 10549 ParamUtil.cpp:99] save dir ./output/model/pass-00000
+    I0601 08:09:46.233438 10549 Util.cpp:209] copy trainer_config.py to ./output/model/pass-00000
+    I0601 08:09:46.233541 10549 ParamUtil.cpp:147] fileName trainer_config.py
+
+The model is saved in :code:`output/` directory. You can use :code:`Ctrl-C` to stop training whenever you want.
+
+Evaluate and Predict
+````````````````````
+
+After training several passes, you can evaluate them and get the best pass. Just run
+
+.. code-block:: bash
+
+    ./evaluate.sh 
+
+You will see messages like this:
+
+.. code-block:: text
+
+    Best pass is 00009,  error is 3.06949, which means predict get error as 0.875998002281
+    evaluating from pass output/pass-00009
+
+Then, you can predict what any user will rate a movie. Just run
+
+..  code-block:: bash
+
+    python prediction.py 'output/pass-00009/'
+
+Predictor will read user input, and predict scores. It has a command-line user interface as follows:
+
+..  code-block:: text
+
+    Input movie_id: 9
+    Input user_id: 4
+    Prediction Score is 2.56
+    Input movie_id: 8
+    Input user_id: 2
+    Prediction Score is 3.13

release_doc/0.9.0/doc/_sources/demo/semantic_role_labeling/index.txt

+Semantic Role Labeling Tutorial
+===============================
+
+.. toctree::
+    :maxdepth: 3
+
+    semantic_role_labeling.md

release_doc/0.9.0/doc/_sources/demo/semantic_role_labeling/semantic_role_labeling.txt

+# Semantic Role labeling Tutorial #
+
+Semantic role labeling (SRL) is a form of shallow semantic parsing whose goal is to discover the predicate-argument structure of each predicate in a given input sentence. SRL is useful as an intermediate step in a wide range of natural language processing tasks, such as information extraction. automatic document categorization and question answering.  An instance is as following [1]:
+
+ [ <sub>A0</sub> He ] [ <sub>AM-MOD</sub> would ][ <sub>AM-NEG</sub> n’t ] [ <sub>V</sub> accept] [ <sub>A1</sub> anything of value ] from [<sub>A2</sub> those he was writing about ]. 
+
+- V: verb
+- A0: acceptor
+- A1: thing accepted
+- A2: accepted-from
+- A3: Attribute
+- AM-MOD: modal 
+- AM-NEG: negation
+
+Given the verb "accept", the chunks in sentence would play certain semantic roles. Here, the label scheme is from Penn Proposition Bank. 
+
+To this date, most of the successful SRL systems are built on top of some form of parsing results where pre-defined feature templates over the syntactic structure are used. This tutorial will present an end-to-end system using deep bidirectional long short-term memory (DB-LSTM)[2] for solving the SRL task, which largely outperforms the previous state-of-the-art systems. The system regards SRL task as the sequence labelling problem. 
+
+## Data Description
+The relevant paper[2] takes the data set in CoNLL-2005&2012 Shared Task for training and testing. Accordingto data license,  the demo adopts the test data set of CoNLL-2005, which can be reached on website.
+
+To download and process the original data, user just need to execute the following command:
+
+```bash
+cd data
+./get_data.sh
+```
+Several new files appear in the `data `directory as follows.
+```bash
+conll05st-release：the test data set of CoNll-2005 shared task 
+test.wsj.words：the Wall Street Journal data sentences
+test.wsj.props:  the propositional arguments
+src.dict：the dictionary of words in sentences
+tgt.dict：the labels dictionary
+feature: the extracted features from data set
+```
+
+## Training
+### DB-LSTM
+Please refer to the Sentiment Analysis demo to learn more about the long short-term memory unit. 
+
+Unlike Bidirectional-LSTM that used in Sentiment Analysis demo,  the DB-LSTM adopts another way to stack LSTM layer. First a standard LSTM processes the sequence in forward direction. The input and output of this LSTM layer are taken by the next LSTM layer as input, processed in reversed direction. These two standard LSTM layers compose a pair of LSTM. Then we stack LSTM layers pair after pair to obtain the deep LSTM model. 
+
+The following figure shows a temporal expanded 2-layer DB-LSTM network.
+<center>
+![pic](./network_arch.png)
+</center>
+
+### Features
+Two input features play an essential role in this pipeline: predicate (pred) and argument (argu). Two other features: predicate context (ctx-p) and region mark (mr) are also adopted. Because a single predicate word can not exactly describe the predicate information, especially when the same words appear more than one times in a sentence. With the predicate context, the ambiguity can be largely eliminated. Similarly, we use region mark m<sub>r</sub> = 1 to denote the argument position if it locates in the predicate context region, or m<sub>r</sub> = 0 if does not. These four simple features are all we need for our SRL system. Features of one sample with context size set to 1 is showed as following[2]:
+<center>
+![pic](./feature.jpg)
+</center>
+
+In this sample, the coresponding labelled sentence is:
+
+[ <sub>A1</sub> A record date ] has [ <sub>AM-NEG</sub> n't ] been [ <sub>V</sub> set ] . 
+
+In the demo, we adopt the feature template as above, consists of :  `argument`, `predicate`, `ctx-p (p=-1,0,1)`, `mark` and use `B/I/O` scheme to label each argument. These features and labels are stored in `feature` file, and separated by `\t`.
+
+### Data Provider
+
+`dataprovider.py` is the python file to wrap data. `hook()` function is to define the data slots for network. The  Six features and label are all IndexSlots.
+```
+def hook(settings, word_dict, label_dict, **kwargs):
+    settings.word_dict = word_dict
+    settings.label_dict = label_dict
+    #all inputs are integral and sequential type
+    settings.slots = [
+        integer_value_sequence(len(word_dict)),
+        integer_value_sequence(len(word_dict)),
+        integer_value_sequence(len(word_dict)),
+        integer_value_sequence(len(word_dict)),
+        integer_value_sequence(len(word_dict)),
+        integer_value_sequence(2),
+        integer_value_sequence(len(label_dict))]
+```
+The corresponding data iterator is as following:
+```
+@provider(use_seq=True, init_hook=hook)
+def process(obj, file_name):
+    with open(file_name, 'r') as fdata:
+        for line in fdata:
+            sentence, predicate, ctx_n1, ctx_0, ctx_p1, mark, label = line.strip().split('\t')
+            words = sentence.split()
+            sen_len = len(words)
+            word_slot = [obj.word_dict.get(w, UNK_IDX) for w in words]
+
+            predicate_slot = [obj.word_dict.get(predicate, UNK_IDX)] * sen_len
+            ctx_n1_slot = [obj.word_dict.get(ctx_n1, UNK_IDX) ] * sen_len
+            ctx_0_slot = [obj.word_dict.get(ctx_0, UNK_IDX) ] * sen_len
+            ctx_p1_slot = [obj.word_dict.get(ctx_p1, UNK_IDX) ] * sen_len
+
+            marks = mark.split()
+            mark_slot = [int(w) for w in marks]
+
+            label_list = label.split()
+            label_slot = [obj.label_dict.get(w) for w in label_list]
+
+            yield word_slot, predicate_slot, ctx_n1_slot, ctx_0_slot, ctx_p1_slot, mark_slot, label_slot
+```
+The `process`function yield 7 lists which are six features and labels.
+ 
+### Neural Network Config
+`db_lstm.py` is the neural network config file to load the dictionaries and define the  data provider module and network architecture during the training procedure. 
+
+Seven `data_layer` load instances from data provider. Six features are transformed into embedddings respectively, and mixed by `mixed_layer` .  Deep bidirectional LSTM layers extract features for the softmax layer. The objective function is cross entropy of labels.
+
+### Run Training 
+The script for training is `train.sh`, user just need to execute:
+```bash
+  ./train.sh
+```
+The content in `train.sh`:
+```
+paddle train \
+  --config=./db_lstm.py \
+  --save_dir=./output \
+  --trainer_count=4 \
+  --log_period=10 \
+  --num_passes=500 \
+  --use_gpu=false \
+  --show_parameter_stats_period=10 \
+  --test_all_data_in_one_period=1 \
+2>&1 | tee 'train.log'
+```
+
+-  \--config=./db_lstm.py : network config file.
+-  \--save_di=./output: output path to save models.
+-  \--trainer_count=4 : set thread number (or GPU count).
+-  \--log_period=10 : print log every 20 batches.
+-  \--num_passes=500: set pass number, one pass in PaddlePaddle means training all samples in dataset one time.
+-  \--use_gpu=false: use CPU to train, set true, if you install GPU version of PaddlePaddle and want to use GPU to train.
+-  \--show_parameter_stats_period=10: show parameter statistic every 100 batches.
+-  \--test_all_data_in_one_period=1: test all data in every testing.
+
+
+After training, the models  will be saved in directory `output`.
+
+### Run testing
+The script for testing is `test.sh`, user just need to execute:
+```bash
+  ./test.sh
+```
+The main part in `tesh.sh`
+```
+paddle train \
+  --config=./db_lstm.py \
+  --model_list=$model_list \
+  --job=test \
+  --config_args=is_test=1 \
+```
+
+  - \--config=./db_lstm.py: network config file
+  - \--model_list=$model_list.list: model list file
+  - \--job=test: indicate the test job
+  - \--config_args=is_test=1: flag to indicate test
+  
+
+### Run prediction
+The script for prediction is `predict.sh`, user just need to execute:
+```bash
+  ./predict.sh
+  
+```
+In `predict.sh`, user should offer the network config file, model path, label file, word dictionary file, feature file
+```
+python predict.py 
+     -c $config_file 
+     -w $model_path 
+     -l $label_file 
+     -d $dict_file 
+     -i $input_file
+```
+
+`predict.py` is the main executable python script, which includes functions: load model, load data, data prediction. The network model will output the probability distribution of labels. In the demo, we take the label with maximum probability as result. User can also implement the beam search or viterbi decoding upon the probability distribution matrix.
+
+After prediction,  the result is saved in `predict.res`.
+
+## Reference
+[1] Martha Palmer, Dan Gildea, and Paul Kingsbury. The Proposition Bank: An Annotated Corpus of Semantic Roles , Computational Linguistics, 31(1), 2005. 
+
+[2] Zhou, Jie, and Wei Xu. "End-to-end learning of semantic role labeling using recurrent neural networks." Proceedings of the Annual Meeting of the Association for Computational Linguistics. 2015.

release_doc/0.9.0/doc/_sources/demo/sentiment_analysis/index.txt

+Sentiment Analasis Tutorial
+===========================
+
+.. toctree::
+    :maxdepth: 3
+    :glob:
+
+    Training Locally <sentiment_analysis.md>
+    internal/cluster_train.md

release_doc/0.9.0/doc/_sources/demo/sentiment_analysis/sentiment_analysis.txt

+# Sentiment Analysis Tutorial
+
+Sentiment analysis has many applications. A basic task in sentiment analysis is classifying the polarity of a given text at the document, sentence or feature/aspect level. One simple example is to classify the customer reviews in a shopping website, a tourism website, and group buying websites like Amazon, TaoBao, Tmall etc.
+
+Sentiment analysis is also used to monitor social media based on large amount of reviews or blogs. For example, the researchers analyzed several surveys on consumer confidence and political opinion, found they correlate to sentiment word frequencies in contemporaneous Twitter messages [1]. Another example is to forecast stock movements through analyzing the text content of a daily Twitter blog [2].
+
+On the other hand, grabbing the user comments of products and analyzing their sentiment are useful to understand user preferences for companies, products, even competing products.
+
+This tutorial will guide you through the process of training a Long Short Term Memory (LSTM) Network to classify the sentiment of sentences from [Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/), sometimes known as the [Internet Movie Database (IMDB)](http://ai.stanford.edu/~amaas/papers/wvSent_acl2011.pdf). This dataset contains movie reviews along with their associated binary sentiment polarity labels, namely positive and negative. So randomly guessing yields 50% accuracy.
+
+## Data Preparation
+
+### IMDB Data Introduction
+
+Before training models, we need to preprocess the data and build a dictionary. First, you can use following script to download IMDB dataset and [Moses](http://www.statmt.org/moses/) tool, which is a statistical machine translation system. We provide a data preprocessing script, which is capable of handling not only IMDB data, but also other user-defined data. In order to use the pre-written script, it needs to move labeled train and test samples to another path, which has been done in `get_imdb.sh`.
+
+```
+cd demo/sentiment/data
+./get_imdb.sh
+```
+If the data is obtained successfuly, you will see the following files at ```./demo/sentiment/data```:
+
+```
+aclImdb  get_imdb.sh  imdb  mosesdecoder-master
+```
+
+* aclImdb: raw dataset downloaded from website.
+* imdb: only contains train and test data.
+* mosesdecoder-master: Moses tool.
+
+IMDB dataset contains 25,000 highly polar movie reviews for training, and 25,000 for testing. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. After running `./get_imdb.sh`, we can find the dataset has the following structure in `aclImdb`.
+
+```
+imdbEr.txt  imdb.vocab  README  test  train
+```
+* train: train sets.
+* test : test sets.
+* imdb.vocab: dictionary.
+* imdbEr.txt: expected rating for each token in imdb.vocab.
+* README: data documentation.
+
+Both train and test set directory contains:
+
+```
+labeledBow.feat  neg  pos  unsup  unsupBow.feat  urls_neg.txt  urls_pos.txt  urls_unsup.txt
+```
+
+* pos: positive samples, contains 12,500 txt files, each file is one movie review.
+* neg: negative samples, contains 12,500 txt files, each file is one movie review.
+* unsup: unlabeled samples, contains 50,000 txt files.
+* urls_xx.txt: urls of each reviews.
+* xxBow.feat: already-tokenized bag of words (BoW) features.
+
+### IMDB Data Preparation
+
+In this demo, we only use labled train and test set and not use imdb.vocab as dictionary. By default, dictionary is builded on train set. Train set is shuffled and test set is not. `tokenizer.perl` in Moses tool is used to tokenize the words and punctuation. Simply execute the following command to preprcess data.
+
+```
+cd demo/sentiment/
+./preprocess.sh
+```
+preprocess.sh:
+
+```
+data_dir="./data/imdb"
+python preprocess.py -i data_dir
+```
+
+* data_dir: input data directory.
+* preprocess.py: preprocess script.
+
+If running successfully, you will see `demo/sentiment/data/pre-imdb` directory as follows:
+
+```
+dict.txt  labels.list  test.list  test_part_000  train.list  train_part_000
+```
+* test\_part\_000 and train\_part\_000: all labeled test and train sets. Train sets have be shuffled.
+* train.list and test.list: train and test file lists.
+* dict.txt: dictionary generated on train sets by default.
+* labels.txt: neg  0, pos 1, means label 0 is negative review, label 1 is positive review.
+
+### User-defined Data Preparation
+
+If you perform other sentiment classifcation task, you can prepare data as follows. We have provided the scripts to build dictionary and preprocess data. So just organize data as follows.
+
+```
+dataset
+|----train
+|    |----class1
+|    |    |----text_files
+|    |----class2
+|    |    |----text_files
+|    |    ...
+|----test
+|    |----class1
+|    |    |----text_files
+|    |----class2
+|    |    |----text_files
+|    |    ...
+```
+* dataset: 1st directory.
+* train, test: 2nd directory.
+* class1,class2,...: 3rd directory.
+* text_files: samples with text file format.
+
+All samples with text files format under the same folder are same category. Each text file contains one or more samples and each line is one sample. In order to shuffle fully, the preprocessing is a little different for data with multiple lines in one text file, which needs to set `-m True` in `preprocess.sh`. And tokenizer.perl is used by default. If you don't need it, only set `-t False` in `preprocess.sh'.
+
+## Training
+
+In this task, we use Recurrent Neural Network (RNN) of LSTM architecure to train sentiment analysis model. LSTM model was introduced primarily in order to overcome the problem of vanishing gradients. LSTM network resembles a standard recurrent neural network with a hidden layer, but each ordinary node in the hidden layer is replaced by a memory cell. Each memory cell contains four main elements: an input gate, a neuron with a self-recurrent connection, a forget gate and an output gate. More details can be found in the literature [4]. The biggest advantage of the LSTM architecture is that it learns to memorize information over long time intervals without the loss of short time memory. At each time step with a new coming word, historical information stored in the memory block is updated to iteratively learn the sequence representation.
+
+<center>![LSTM](./lstm.png)</center>
+<center>Figure 1. LSTM [3]</center>
+
+Sentiment analysis is among the most typical problems in natural language understanding. It aims at predicting the attitude expressed in a sequence. Usually, only some key words, like adjectives and adverbs words, play a major role in predicting the sentiment of sequences or paragraphs. However, some review or comment contexts are very long, such as IMDB dataset. We use LSTM to perform this task for its improved design with the gate mechanism. First, it is able to summarize the representation from word level to context level with variable context length which is adapted by the gate values. Second, it can utilize the expanded context at the sentence level, while most methods are good at utilizing n-gram level knowledge. Third, it learns the paragraph representation directly rather than combining the context level information. This results in this end-to-end framework.
+
+In this demo we provide two network, namely bidirectional-LSTM and three layers of stacked-LSTM.
+
+#### Bidirectional-LSTM
+
+One is a bidirectional LSTM network, connected by fully connected layer and softmax, as shown in Figure 2.
+
+<center>![BiLSTM](./bi_lstm.jpg)</center>
+<center>Figure 2. Bidirectional-LSTM </center>
+
+#### Stacked-LSTM
+Another is three-layer LSTM structure in Figure 3. The bottom of the figure is word embedding. Next, three LSTM-Hidden layers are connected and the second LSTM is reversed. Then extract the maximum hidden vectors of all time step of hidden and LSTM layer as the representation for the entire sequence. Finally, a fully connected feed forward layer with softmax activation is used to perform the classification task. This network is refered to paper [5].
+
+<center>![StackedLSTM](./stacked_lstm.jpg)</center>
+<center>Figure 3. Stacked-LSTM for sentiment analysis </center>
+
+**Config**
+
+Switch into `demo/sentiment` directory, `trainer_config.py` file is an example of the config, containing algorithm and newtork configure. The first line imports predefined networks from `sentiment_net.py`.
+
+trainer_config.py:
+
+```python
+from sentiment_net import *
+
+data_dir  = "./data/pre-imdb"
+# whether this config is used for test
+is_test = get_config_arg('is_test', bool, False)
+# whether this config is used for prediction
+is_predict = get_config_arg('is_predict', bool, False)
+dict_dim, class_dim = sentiment_data(data_dir, is_test, is_predict)
+
+################## Algorithm Config #####################
+
+settings(
+  batch_size=128,
+  learning_rate=2e-3,
+  learning_method=AdamOptimizer(),
+  regularization=L2Regularization(8e-4),
+  gradient_clipping_threshold=25
+)
+
+#################### Network Config ######################
+stacked_lstm_net(dict_dim, class_dim=class_dim,
+                 stacked_num=3, is_predict=is_predict)
+#bidirectional_lstm_net(dict_dim, class_dim=class_dim, is_predict=is_predict)
+```
+
+* **Data Definition**:
+   * get\_config\_arg(): get arguments setted by `--config_args=xx` in commandline argument.
+   * Define TrainData and TestData provider, here using Python interface (PyDataProviderWrapper) of PaddlePaddle to load data. For details, you can refer to the document of PyDataProvider.
+
+* **Algorithm Configuration**:
+   * use sgd algorithm.
+   * use adam optimization.
+   * set batch size of 128.
+   * set average sgd window.
+   * set global learning rate.
+* **Network Configuration**:
+   * dict_dim: get dictionary dimension.
+   * class_dim: set category number, IMDB has two label, namely positive and negative label.
+   * `stacked_lstm_net`: predefined network as shown in Figure 3, use this network by default.
+   * `bidirectional_lstm_net`: predefined network as shown in Figure 2.
+
+**Training**
+
+Install PaddlePaddle first if necessary. Then you can use script `train.sh` as follows to launch local training.
+
+```
+cd demo/sentiment/
+./train.sh
+```
+
+train.sh:
+
+```
+config=trainer_config.py
+output=./model_output
+paddle train --config=$config \
+             --save_dir=$output \
+             --job=train \
+             --use_gpu=false \
+             --trainer_count=4 \
+             --num_passes=10 \
+             --log_period=20 \
+             --dot_period=20 \
+             --show_parameter_stats_period=100 \
+             --test_all_data_in_one_period=1 \
+             2>&1 | tee 'train.log'
+```
+
+* \--config=$config: set network config.
+* \--save\_dir=$output: set output path to save models.
+* \--job=train: set job mode to train.
+* \--use\_gpu=false: use CPU to train, set true, if you install GPU version of PaddlePaddle and want to use GPU to train.
+* \--trainer\_count=4: set thread number (or GPU count).
+* \--num\_passes=15: set pass number, one pass in PaddlePaddle means training all samples in dataset one time.
+* \--log\_period=20: print log every 20 batches.
+* \--show\_parameter\_stats\_period=100: show parameter statistic every 100 batches.
+* \--test\_all_data\_in\_one\_period=1: test all data every testing.
+
+If the run succeeds, the output log is saved in path of `demo/sentiment/train.log` and model is saved in path of `demo/sentiment/model_output/`. The output log is explained as follows.
+
+```
+Batch=20 samples=2560 AvgCost=0.681644 CurrentCost=0.681644 Eval: classification_error_evaluator=0.36875  CurrentEval: classification_error_evaluator=0.36875
+...
+Pass=0 Batch=196 samples=25000 AvgCost=0.418964 Eval: classification_error_evaluator=0.1922
+Test samples=24999 cost=0.39297 Eval: classification_error_evaluator=0.149406
+```
+- Batch=xx: means passing xx batches.
+- samples=xx: means passing xx samples.
+- AvgCost=xx: averaged cost from 0-th batch to current batch.
+- CurrentCost=xx: current cost of latest log_period batches.
+- Eval: classification\_error\_evaluator=xx: means classfication error from 0-th batch ro current batch.
+- CurrentEval: classification\_error\_evaluator: current classfication error of the lates log_period batches.
+- Pass=0: Going through all training set one time is called one pass. 0 means going through training set first time.
+
+By default, we use the `stacked_lstm_net` network, which converges at a faster rate than `bidirectional_lstm_net` when passing same sample number. If you want to use bidirectional LSTM, just remove comment in the last line and comment `stacked_lstm_net`.
+
+## Testing
+
+Testing means evaluating the labeled validation set using trained model.
+
+```
+cd demo/sentiment
+./test.sh
+```
+
+test.sh:
+
+```bash
+function get_best_pass() {
+  cat $1  | grep -Pzo 'Test .*\n.*pass-.*' | \
+  sed  -r 'N;s/Test.* error=([0-9]+\.[0-9]+).*\n.*pass-([0-9]+)/\1 \2/g' | \
+  sort | head -n 1
+}
+
+log=train.log
+LOG=`get_best_pass $log`
+LOG=(${LOG})
+evaluate_pass="model_output/pass-${LOG[1]}"
+
+echo 'evaluating from pass '$evaluate_pass
+
+model_list=./model.list
+touch $model_list | echo $evaluate_pass > $model_list
+net_conf=trainer_config.py
+paddle train --config=$net_conf \
+             --model_list=$model_list \
+             --job=test \
+             --use_gpu=false \
+             --trainer_count=4 \
+             --config_args=is_test=1 \
+             2>&1 | tee 'test.log'
+```
+
+The function `get_best_pass` gets the best model by classification error rate for testing. In this example, We use test dataset of IMDB as validation by default. Unlike training, it needs to specify `--job=test` and model path, namely `--model_list=$model_list` here. If running successfully, the log is saved in path of `demo/sentiment/test.log`. For example, in our test, the best model is `model_output/pass-00002`, the classification error is 0.115645 as follows.
+
+```
+Pass=0 samples=24999 AvgCost=0.280471 Eval: classification_error_evaluator=0.115645
+```
+
+## Prediction
+
+`predict.py` provides a predicting interface. You should install python api of PaddlePaddle before using it. One example to predict unlabeled review of IMDB is as follows. Simply running:
+
+```
+cd demo/sentiment
+./predict.sh
+```
+predict.sh:
+
+```
+#Note the default model is pass-00002, you shold make sure the model path
+#exists or change the mode path.
+model=model_output/pass-00002/
+config=trainer_config.py
+label=data/pre-imdb/labels.list
+python predict.py \
+     -n $config\
+     -w $model \
+     -b $label \
+     -d data/pre-imdb/dict.txt \
+     -i data/aclImdb/test/pos/10007_10.txt
+```
+
+* `predict.py`: predicting interface.
+*  -n $config : set network configure.
+*  -w $model: set model path.
+*  -b $label: set dictionary about corresponding relation between integer label and string label.
+*  -d data/pre-imdb/dict.txt: set dictionary.
+*  -i data/aclImdb/test/pos/10014_7.txt: set one example file to predict.
+
+Note you should make sure the default model path `model_output/pass-00002`
+exists or change the model path.
+
+Predicting result of this example:
+
+```
+Loading parameters from model_output/pass-00002/
+./data/aclImdb/test/pos/10014_7.txt: predicting label is pos
+```
+We sincerely appreciate your interest and welcome your contributions.
+
+## Reference
+[1] Brendan O'Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010. [From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series](http://homes.cs.washington.edu/~nasmith/papers/oconnor+balasubramanyan+routledge+smith.icwsm10.pdf). In ICWSM-2010. <br>
+[2] Johan Bollen, Huina Mao, Xiaojun Zeng. 2011. [Twitter mood predicts the stock market](http://arxiv.org/abs/1010.3003), Journal of Computational Science.<br>
+[3] Alex Graves, Marcus Liwicki, Santiago Fernan- dez, Roman Bertolami, Horst Bunke, and Ju ̈rgen Schmidhuber. 2009. [A novel connectionist system for unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine In- telligence](http://www.cs.toronto.edu/~graves/tpami_2009.pdf), 31(5):855–868.<br>
+[4] Zachary C. Lipton, [A Critical Review of Recurrent Neural Networks for Sequence Learning](http://arxiv.org/abs/1506.00019v1), arXiv:1506.00019. <br>
+[5] Jie Zhou and Wei Xu; [End-to-end Learning of Semantic Role Labeling Using Recurrent Neural Networks](http://www.aclweb.org/anthology/P/P15/P15-1109.pdf); ACL-IJCNLP 2015. <br>

release_doc/0.9.0/doc/_sources/demo/text_generation/index.txt

+Text Generation Tutorial
+========================
+
+.. toctree::
+    :maxdepth: 3
+    :glob:
+
+    Training Locally <text_generation.md>
+    internal/cluster_train.md

release_doc/0.9.0/doc/_sources/demo/text_generation/text_generation.txt

+# Text generation Tutorial #
+
+Sequence to sequence has been proven to be a powerful model for language generation. It can be used for machine translation, query rewriting, image captioning, etc.
+
+This tutorial guides you through training a sequence to sequence model for neural machine translation (NMT) network that translates French to English.
+
+We follow the paper [Neural Machine Translation by Jointly Learning to Align and Translate](http://arxiv.org/abs/1409.0473) , which details the model architecture and training procedure for good performance on WMT-14 dataset. This tutorial reproduces this result in PaddlePaddle.
+
+We thank @caoying for the pull request that defines the model architecture and solver configurations.
+
+## Data Preparation ##
+### Download and Extract ###
+Download the WMT-14 dataset from [http://www-lium.univ-lemans.fr/~schwenk/cslm\_joint\_paper/](http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/), extract it, and divide Develop and Test data into separate folder.
+
+- **Train data**: [bitexts (after selection)](http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/bitexts.tgz)
+- **Develop and Test data**: [dev+test data](http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/dev+test.tgz)
+
+To do this, simply run the following commands in linux, otherwise, you need to download, extract, divide, and rename the file suffix respectively.
+
+```bash
+cd demo/seqToseq/data
+./wmt14_data.sh
+```
+
+We should find that the dataset `wmt14` has three folders as shown in the following table.
+<table border="2" cellspacing="0" cellpadding="6" rules="all" frame="border">
+<colgroup>
+<col  class="left" />
+<col  class="left" />
+<col  class="left" />
+<col  class="left" />
+</colgroup>
+
+<thead>
+<tr>
+<th scope="col" class="left">folder name</th>
+<th scope="col" class="left">French-English parallel corpora file</th>
+<th scope="col" class="left">number of total file</th>
+<th scope="col" class="left">size</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td class="left">train_data</td>
+<td class="left">ccb2_pc30.src, ccb2_pc30.trg, etc</td>
+<td class="left">twelve</td>
+<td class="left">3.55G</td>
+</tr>
+
+<tr>
+<td class="left">test_data</td>
+<td class="left">ntst1213.src, ntst1213.trg</td>
+<td class="left">two</td>
+<td class="left">1636k</td>
+</tr>
+
+<tr>
+<td class="left">gen_data</td>
+<td class="left">ntst14.src, ntst14.trg</td>
+<td class="left">two</td>
+<td class="left">864k</td>
+</tr>
+</tbody>
+</table>
+<br/>
+
+- Each folder has French-English parallel corpora
+- **XXX.src** are source French files; **XXX.trg** are target English files.
+- The number of lines of **XXX.src** and **XXX.trg** should be the same.
+- Each line is a French/English sentence.
+- There is a one-to-one correspondence between the sentence at the i-th line of **XXX.src** and **XXX.trg**.
+
+### User Defined Dataset ###
+
+If you need to do other sequence-to-sequence tasks, such as Paraphrasing, you only need to organize the data as follows, and place them in `demo/seqToseq/data`:
+
+    dataset
+      train
+        file1.src file1.trg
+        file2.src file2.trg
+        ......
+      test
+        file1.src file1.trg
+        file2.src file2.trg
+        ......
+      gen
+        file1.src file1.trg
+        file2.src file2.trg
+        ......
+- 1st directory: dataset folder name
+- 2nd directory: folder of train, test, and gen. The names of these three folders are fixed.
+- 3rd file: Source-Target parallel corpora files.
+  - **XXX.src** are source files, **XXX.trg** are target files.
+  - Each line of the file must be a sequence.
+  - There should be a one-to-one correspondence between the i-th sequence of **XXX.src** and **XXX.trg**.
+
+## Data Preprocess ##
+### Preprocessing Workflow ###
+- Concat each Source-Target parallel corpora to be one file:
+  - concat each **XXX.src** and **XXX.trg** to be **XXX**.
+  - the i-th line of **XXX** = the i-th line of **XXX.src** + '\t' + the i-th line of **XXX.trg**
+- Build source and target dictionary of train data, each dictionary has DICTSIZE words:
+  - the most frequent (DICTSIZE-3) words
+  - 3 special token:
+    - `<s>`: the start of a sequence
+    - `<e>`: the end of a sequence
+    - `<unk>`: a word not included in dictionary
+
+### Preprocessing Command and Result
+The general command for preprocessing the dataset is:
+
+```python
+cd demo/seqToseq/
+python preprocess.py -i INPUT [-d DICTSIZE] [-m]
+```
+
+- `-i INPUT`: the path of input original dataset
+- `-d DICTSIZE`: the specified word count of dictionary, if not set, dictionary will contain all the words in input dataset
+- `-m --mergeDict`: merge source and target dictionary, thus, two dictionaries have the same context
+
+And you will see messages like this:
+
+    concat parallel corpora for dataset
+    build source dictionary for train data
+    build target dictionary for train data
+    dictionary size is XXX
+
+Here, you can simply run the command:
+
+```python
+python preprocess.py -i data/wmt14 -d 30000
+```
+
+It will take several minutes, and store the preprocessed dataset in `demo/seqToseq/data/pre-wmt14`, the directory has following structure.
+
+    train test gen train.list test.list gen.list src.dict trg.dict
+
+- **train, test, gen**: folder contains French-English parallel corpora of train data, test data and gen data respectively. Each line of file in folder contains two parts, the former is a French sequence, and the latter is a corresponding English sequence.
+- **train.list, test.list, gen.list**: text contains a file list in train folder, test folder and gen folder respectively
+- **src.dict, trg.dict**: source (French) / target (English) dictionary, each dictionary has 30000 words: the most frequent 29997 words and 3 special token
+
+## Model Training ##
+### Introduction ###
+
+Neural machine translation (NMT) aims at building a single neural network that can be jointly tuned to maximize translation performance. Recently proposed NMT models often belong to a family of encoder–decoder models. Encoder-Decoder models encode a source sentence into a fixed-length vector from which a decoder generates a target sentence.
+
+In this task, we use an extension to the encoder–decoder model which learns to align and translate jointly. Each time the model generates a word in a translation, it searches for a set of positions in the source sentence for the most relevant information.  The decoder predicts a target word based on the context vectors associated with these source positions and all the previous generated target words. For more detailed explanation, readers can refer to paper [Neural Machine Translation by Jointly Learning to Align and Translate](http://arxiv.org/abs/1409.0473).
+
+The most distinguishing feature of this model is that it doesn't encode an input sentence into a single fixed-length vector. Instead, it encodes the input sentence into a sequence of vectors, where one vector corresponds to an input element. A subset of these vectors is chosen adaptively while decoding the translated sentence. This frees a NMT model from having to squash all the information of a source sentence, regardless of its length, into a fixed-length vector. The improvement of this model is more apparent for longer sentences, but the improvement can be observed for sentences of any length.
+<center>![](./encoder-decoder-attention-model.png)</center>
+<center>Figure 1. Encoder-Decoder-Attention-Model</center>
+
+### Training Model in PaddlePaddle ###
+We need to create a model config file before training. Here is an example `demo/seqToseq/translation/train.conf`. The first three lines import python function for defining network, and define the job_mode and attention_mode.
+
+```python
+from seqToseq_net import *
+is_generating = False
+
+### Data Definiation
+train_conf = seq_to_seq_data(data_dir = "./data/pre-wmt14",
+                             is_generating = is_generating)
+
+### Algorithm Configuration
+settings(
+    learning_method = AdamOptimizer(),
+    batch_size = 50,
+    learning_rate = 5e-4)
+
+### Network Architecture
+gru_encoder_decoder(train_conf, is_generating)
+```
+
+1. **Data Definiation**: We define a SeqToSeq train and test data in our example. It returns train_conf as the configuration, following is its input arguments:
+   - data_dir: directory of train data and test data
+   - is\_generating: whether this config is used for generating, here is false
+2. **Algorithm Configuration**: We use the SGD training algorithm (default), ADAM learning method in our example, specify batch_size as 50, and learning rate as 5e-4.
+3. **Network Architecture**: We use an attention version of GRU Encoder-Decoder network in our example. It consists a bidirectional GRU as an encoder and a decoder that emulates searching through a source sentence during decoding a translation.
+
+### Training Command and Result###
+After writing the model config, we can train the model by running the command:
+
+```bash
+cd demo/seqToseq/translation
+./train.sh
+```
+
+The `train.sh` is shown as follows:
+
+```bash
+paddle train \
+--config='translation/train.conf' \
+--save_dir='translation/model' \
+--use_gpu=false \
+--num_passes=16 \
+--show_parameter_stats_period=100 \
+--trainer_count=4 \
+--log_period=10 \
+--dot_period=5 \
+2>&1 | tee 'translation/train.log'
+```
+- config: set config of neural network
+- save_dir: set output path to save models
+- use_gpu: whether to use GPU to train, here use CPU
+- num_passes: set number of passes. One pass in paddle means training all samples in dataset one time
+- show_parameter_stats_period: here show parameter statistic every 100 batches
+- trainer_count: set number of CPU threads or GPU devices
+- log_period: here print log every 10 batches
+- dot_period: here print '.' every 5 batches
+
+The training loss function is printed every 10 batch by default, and you will see messages like this:
+
+    I0719 19:16:45.952062 15563 TrainerInternal.cpp:160]  Batch=10 samples=500 AvgCost=198.475 CurrentCost=198.475 Eval: classification_error_evaluator=0.737155  CurrentEval: classification_error_evaluator=0.737155
+    I0719 19:17:56.707319 15563 TrainerInternal.cpp:160]  Batch=20 samples=1000 AvgCost=157.479 CurrentCost=116.483 Eval: classification_error_evaluator=0.698392  CurrentEval: classification_error_evaluator=0.659065
+    .....
+- AvgCost: Average Cost from 0th batch to current batch
+- CurrentCost: Cost in current batch
+- classification\_error\_evaluator(Eval): False prediction rate for each word from 0th evaluation to current evaluation
+- classification\_error\_evaluator(CurrentEval): False prediction rate for each word in current evaluation
+
+And when the classification\_error\_evaluator is less than 0.35, the model is trained sucessfully.
+
+## Text Generation ##
+### Introduction ###
+
+Generally speaking, the NMT model is conditioned on the encodings of the source sentence, and then to predict the next target word by given the current target word. In the training process, the current word is always knowns as the ground truth, by contrast. In the generating process, the current word is the output of the decoder in last time step, which is accessed to from a memory in PaddlePaddle.
+
+Besides, we use Beam Search to generate sequences. Beam search uses breadth-first search to build its search tree. At each level of the tree, it generates all successors of the states at the current level, sorting them in increasing order of heuristic cost. However, it only stores a predetermined number of best states at each level (called the beam size).
+
+### Pretrained model ###
+We trained the model on a cluster with 50 nodes, each node has two 6-core CPUs. We trained 16 passes in 5 days, where each pass takes 7 hours. The model_dir has 16 sub-folder, each of which contains the whole model parameters with 202MB size. And we find pass-00012 model has the highest BLEU 27.77 (see paper [BLEU: a Method for Automatic Evaluation of Machine Translation](http://www.aclweb.org/anthology/P02-1040.pdf)). To download and extract this model, simply run the following commands in linux.
+
+```bash
+cd demo/seqToseq/data
+./wmt14_model.sh
+```
+
+### Generating Model in PaddlePaddle ###
+We need to create a model config file before translating French sequence. Here is an example `demo/seqToseq/translation/gen.conf`, the first three lines import python function for defining network, and define the job\_mode and attention\_mode.
+
+```python
+from seqToseq_net import *
+is_generating = True
+
+################## Data Definiation #####################
+gen_conf = seq_to_seq_data(data_dir = "./data/pre-wmt14",
+                           is_generating = is_generating,
+                           gen_result = "./translation/gen_result")
+
+############## Algorithm Configuration ##################
+settings(
+  learning_method = AdamOptimizer(),
+  batch_size = 1,
+  learning_rate = 0)
+
+################# Network configure #####################
+gru_encoder_decoder(gen_conf, is_generating)
+```
+
+1. **Data Definiation**: We defines an SeqToSeq gen data in our example. It returns gen_conf as the configuration, following is its input arguments:
+   - data\_dir: directory of gen data
+   - is\_generating: whether this config is used for generating, here is false
+   - gen\_result: file to store the generation result
+2. **Algorithm Configuration**: We use SGD traing algorithm in generation, and specify batch_size as 1 (each time generate one sequence), and learning rate as 0.
+3. **Network Architecture**: Essentially the same as the training model.
+
+### Generating Command and Result ###
+After writing the model config, we can do text translation from French to English by running the command:
+
+```bash
+cd demo/seqToseq/translation
+./gen.sh
+```
+
+The `gen.sh` is shown as follows, unlike training, there are some different arguments to specify:
+
+```bash
+paddle train \
+--job=test \
+--config='translation/gen.conf' \
+--save_dir='data/wmt14_model' \
+--use_gpu=true \
+--num_passes=13 \
+--test_pass=12 \
+--trainer_count=1 \
+2>&1 | tee 'translation/gen.log'
+```
+- job: set job mode to test
+- save_dir: the path of saved models
+- num_passes and test_pass: loading model parameters from test_pass to (num_passes - 1), here only loads `data/wmt14_model/pass-00012`
+
+You will see messages like this:
+
+    I0706 14:48:31.178915 31441 GradientMachine.cpp:143] Loading parameters from data/wmt14_model/pass-00012
+    I0706 14:48:40.012039 31441 Tester.cpp:125]  Batch=100 samples=100 AvgCost=0
+    I0706 14:48:48.898632 31441 Tester.cpp:125]  Batch=200 samples=200 AvgCost=0
+    ...
+
+And the generating result in `demo/seqToseq/translation/gen_result` likes:
+
+    0
+    0       -11.1314         The <unk> <unk> about the width of the seats while large controls are at stake <e>
+    1       -11.1519         The <unk> <unk> on the width of the seats while large controls are at stake <e>
+    2       -11.5988         The <unk> <unk> about the width of the seats while large controls are at stake . <e>
+
+    1
+    0       -24.4149         The dispute is between the major aircraft manufacturers about the width of the tourist seats on the <unk> flights , paving the way for a <unk> confrontation during the month of the Dubai <unk> . <e>
+    1       -26.9524         The dispute is between the major aircraft manufacturers about the width of the tourist seats on the <unk> flights , paving the way for a <unk> confrontation during the month of Dubai &apos; s <unk> . <e>
+    2       -27.9574         The dispute is between the major aircraft manufacturers about the width of the tourist seats on the <unk> flights , paving the way for a <unk> confrontation during the month of Dubai &apos; s Dubai <unk> . <e>
+    ...
+
+- This is the beam search result, where beam size is 3
+- '0' in 1st-line and '1' in 6th-line mean the sequence-id in gen data
+- Other six lines list the beam search results
+  - The 2nd-column is the score of beam search (from large to small)
+  - The 3rd-colunm is the generating English sequence
+- There is 2 special tokens:
+  - `<e>`: the end of a sequence
+  - `<unk>`: a word not included in dictionary
+
+### Bleu Evalutaion ###
+Human evaluations of machine translation are extensive but expensive. Paper [BLEU: a Method for Automatic Evaluation of Machine Translation](http://www.aclweb.org/anthology/P02-1040.pdf) presents a method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations. [Moses](http://www.statmt.org/moses/) is a statistical machine translation system, and we use [multi-bleu.perl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl) of it to do Bleu Evalution. To download this script, simply run the following command:
+
+```bash
+cd demo/seqToseq/translation
+./moses_bleu.sh
+```
+
+Since the standard translation is alrealy downloaded as `data/wmt14/gen/ntst14.trg`, we can do Bleu Evalution by running the command:
+
+```bash
+cd demo/seqToseq/translation
+./eval_bleu.sh FILE BEAMSIZE
+```
+
+- FILE: the generation result file
+- BEAMSIZE: expand width in beam search

release_doc/0.9.0/doc/_sources/dev/new_layer/index.txt

+Writing New Layers
+==================
+
+.. toctree::
+  :maxdepth: 3
+
+  new_layer.rst

release_doc/0.9.0/doc/_sources/index.txt

+PaddlePaddle Documentation
+==========================
+
+User Guide
+----------
+* [Introduction](introduction/index.md)
+* [Quick Start](demo/quick_start/index_en.md)
+* [Build and Installation](build/index.rst)
+* [Contribute Code](build/contribute_to_paddle.md)
+* [User Interface](ui/index.md)
+* [Model Config Interface](ui/api/trainer_config_helpers/index.rst)
+* [Example and Demo](demo/index.md)
+* [Cluster Train](cluster/index.md)
+
+Development Guide
+-----------------
+* [Layer Documents](layer.md)
+* [Writing New Layers](dev/new_layer/index.rst)
+* [Source Code Documents](source/index.md)
+
+Algorithm Tutorial
+------------------
+* [RNN Configuration](algorithm/rnn/rnn.rst)

release_doc/0.9.0/doc/_sources/introduction/index.txt

+# Introduction
+
+PaddlePaddle is a deep learning platform open-sourced by Baidu. With PaddlePaddle, you can easily train a classic neural network within a couple lines of configuration, or you can build sophisticated models that provide state-of-the-art performance on difficult learning tasks like sentiment analysis, machine translation, image caption and so on.
+
+## 1. A Classic Problem
+
+Now, to give you a hint of what using PaddlePaddle looks like, let's start with a fundamental learning problem - <a href="https://en.wikipedia.org/wiki/Simple_linear_regression">**simple linear regression**</a> : you have observed a set of two-dimensional data points of `X` and `Y`, where `X` is an explanatory variable and `Y` is corresponding dependent variable, and you want to recover the underlying correlation between `X` and `Y`. Linear regression can be used in many practical scenarios. For example, `X` can be a variable about house size, and `Y` a variable about house price. You can build a model that captures relationship between them by observing real estate markets.
+
+## 2. Prepare the Data
+
+Suppose the true relationship can be characterized as `Y = 2X + 0.3`, let's see how to recover this pattern only from observed data. Here is a piece of python code that feeds synthetic data to PaddlePaddle. The code is pretty self-explanatory, the only extra thing you need to add for PaddlePaddle is a definition of input data types.
+
+```python
+# dataprovider.py
+from paddle.trainer.PyDataProvider2 import *
+import random
+
+# define data types of input: 2 real numbers
+@provider(input_types=[dense_vector(1), dense_vector(1)],use_seq=False)
+def process(settings, input_file):
+    for i in xrange(2000):
+        x = random.random()
+        yield [x], [2*x+0.3]
+```
+
+## 3. Train a NeuralNetwork in PaddlePaddle
+
+To recover this relationship between `X` and `Y`, we use a neural network with one layer of linear activation units and a square error cost layer. Don't worry if you are not familiar with these terminologies, it's just saying that we are starting from a random line `Y' = wX + b` , then we gradually adapt `w` and `b` to minimize the difference between `Y'` and `Y`. Here is what it looks like in PaddlePaddle:
+
+```python
+# trainer_config.py
+from paddle.trainer_config_helpers import *
+
+# 1. read data. Suppose you saved above python code as dataprovider.py
+data_file = 'empty.list'
+with open(data_file, 'w') as f: f.writelines(' ')
+define_py_data_sources2(train_list=data_file, test_list=None, 
+        module='dataprovider', obj='process',args={})
+
+# 2. learning algorithm
+settings(batch_size=12, learning_rate=1e-3, learning_method=MomentumOptimizer())
+
+# 3. Network configuration
+x = data_layer(name='x', size=1)
+y = data_layer(name='y', size=1)
+y_predict = fc_layer(input=x, param_attr=ParamAttr(name='w'), size=1, act=LinearActivation(), bias_attr=ParamAttr(name='b'))
+cost = regression_cost(input=y_predict, label=y)
+outputs(cost)
+```
+
+Some of the most fundamental usages of PaddlePaddle are demonstrated:
+
+-  The first part shows how to feed data into PaddlePaddle. In general cases, PaddlePaddle reads raw data from a list of files, and then do some user-defined process to get real input. In this case, we only need to create a placeholder file since we are generating synthetic data on the fly.
+
+-  The second part describes learning algorithm. It defines in what ways adjustments are made to model parameters. PaddlePaddle provides a rich set of optimizers, but a simple momentum based optimizer will suffice here, and it processes 12 data points each time.
+
+-  Finally, the network configuration. It usually is as simple as "stacking" layers. Three kinds of layers are used in this configuration:
+	-  **Data Layer**: a network always starts with one or more data layers. They provide input data to the rest of the network. In this problem, two data layers are used respectively for `X` and `Y`.
+	-  **FC Layer**: FC layer is short for Fully Connected Layer, which connects all the input units to current layer and does the actual computation specified as activation function. Computation layers like this are the fundamental building blocks of a deeper model.
+	-  **Cost Layer**: in training phase, cost layers are usually the last layers of the network. They measure the performance of current model, and provide guidence to adjust parameters.
+
+Now that everything is ready, you can train the network with a simple command line call:
+ ```
+ paddle train --config=trainer_config.py --save_dir=./output --num_passes=30
+ ```
+
+This means that PaddlePaddle will train this network on the synthectic dataset for 30 passes, and save all the models under path `./output`. You will see from the messages printed out during training phase that the model cost is decreasing as time goes by, which indicates we are getting a closer guess.
+
+
+## 4. Evaluate the Model
+
+Usually, a different dataset that left out during training phase should be used to evalute the models. However, we are lucky enough to know the real answer: `w=2, b=0.3`, thus a better option is to check out model parameters directly.
+
+In PaddlePaddle, training is just to get a collection of model parameters, which are `w` and `b` in this case. Each parameter is saved in an individual file in the popular `numpy` array format. Here is the code that reads parameters from last pass.
+
+```python
+import numpy as np
+import os
+
+def load(file_name):
+    with open(file_name, 'rb') as f:
+        f.read(16) # skip header for float type.
+        return np.fromfile(f, dtype=np.float32)
+        
+print 'w=%.6f, b=%.6f' % (load('output/pass-00029/w'), load('output/pass-00029/b'))
+# w=1.999743, b=0.300137
+```
+
+<center> ![](./parameters.png) </center>
+
+Although starts from a random guess, you can see that value of `w` changes quickly towards 2 and `b` changes quickly towards 0.3. In the end, the predicted line is almost identical with real answer.
+
+There, you have recovered the underlying pattern between `X` and `Y` only from observed data.
+
+
+## 5. Where to Go from Here
+
+- <a href="../build/index.html"> Build and Installation </a>
+- <a href="../demo/quick_start/index_en.html">Quick Start</a>
+- <a href="../demo/index.html">Example and Demo</a>

release_doc/0.9.0/doc/_sources/layer.txt

+# Layer Documents
+
+* [Layer Source Code Document](source/gserver/layers/index.rst)
+* [Layer Python API Document](ui/api/trainer_config_helpers/layers_index.rst)

release_doc/0.9.0/doc/_sources/source/api/api.txt

+API
+========
+
+.. doxygenfile:: paddle/api/PaddleAPI.h
+.. doxygenfile:: paddle/api/Internal.h

release_doc/0.9.0/doc/_sources/source/cuda/cuda/cuda.txt

+Cuda
+=============
+
+Dynamic Link Libs
+--------------------------
+
+hl_dso_loader.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_dso_loader.h
+
+GPU Resources
+----------------
+
+hl_cuda.ph
+``````````````
+.. doxygenfile:: paddle/cuda/include/hl_cuda.ph
+
+hl_cuda.h
+``````````````
+.. doxygenfile:: paddle/cuda/include/hl_cuda.h
+
+CUDA Wrapper
+--------------
+
+hl_cuda_cublas.h
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_cuda_cublas.h
+
+hl_cuda_cudnn.h
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_cuda_cudnn.h
+
+hl_cuda_cudnn.h
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_cuda_cudnn.ph
+
+
+
+

release_doc/0.9.0/doc/_sources/source/cuda/cuda/index.txt

+CUDA
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  cuda.rst

release_doc/0.9.0/doc/_sources/source/cuda/matrix/index.txt

+Matrix
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  matrix.rst

release_doc/0.9.0/doc/_sources/source/cuda/matrix/matrix.txt

+Matrix
+=======
+
+Base Matrix
+-------------
+
+hl_matrix.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_matrix.h
+
+hl_matrix_base.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_matrix_base.cuh
+
+hl_matrix_apply.cuh
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_matrix_apply.cuh
+
+hl_matrix_ops.cuh
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_matrix_ops.cuh
+
+hl_matrix_type.cuh
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_matrix_type.cuh
+
+hl_sse_matrix_kernel.cuh
+``````````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_sse_matrix_kernel.cuh
+
+hl_batch_transpose.h
+``````````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_batch_transpose.h
+
+Sparse Matrix
+--------------
+
+hl_sparse.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_sparse.h
+
+hl_sparse.ph
+``````````````````````
+.. doxygenfile:: paddle/cuda/include/hl_sparse.ph
+
+Others
+---------------
+
+hl_aggregate.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_aggregate.h
+
+hl_table_apply.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_table_apply.h
+
+hl_top_k.h
+``````````````````
+.. doxygenfile:: paddle/cuda/include/hl_top_k.h
+
+

release_doc/0.9.0/doc/_sources/source/cuda/rnn/index.txt

+RNN
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  rnn.rst

release_doc/0.9.0/doc/_sources/source/cuda/rnn/rnn.txt

+Neural Networks
+==================
+
+Base
+-------
+.. doxygenfile:: paddle/cuda/include/hl_gpu.h
+.. doxygenfile:: paddle/cuda/include/hl_cnn.h
+.. doxygenfile:: paddle/cuda/include/hl_functions.h
+.. doxygenfile:: paddle/cuda/include/hl_avx_functions.h
+.. doxygenfile:: paddle/cuda/include/hl_device_functions.cuh
+.. doxygenfile:: paddle/cuda/include/hl_gpu_functions.cuh
+
+Activation Functions
+-----------------------
+.. doxygenfile:: paddle/cuda/include/hl_activation_functions.h
+
+RNN Related APIs
+-----------------
+
+.. doxygenfile:: paddle/cuda/include/hl_recurrent_apply.cuh
+.. doxygenfile:: paddle/cuda/include/hl_sequence.h
+
+LSTM Model
+``````````````
+.. doxygenfile:: paddle/cuda/include/hl_lstm.h
+.. dpxygenfile:: paddle/cuda/include/hl_cpu_lstm.cuh
+.. doxygenfile:: paddle/cuda/include/hl_gpu_lstm.cuh
+.. doxygenfile:: paddle/cuda/include/hl_lstm_ops.cuh
+
+GRU Model
+````````````````
+.. doxygenfile:: paddle/cuda/include/hl_gru_ops.cuh
+.. doxygenfile:: paddle/cuda/include/hl_cpu_gru.cuh
+.. doxygenfile:: paddle/cuda/include/hl_gpu_gru.cuh
+
+

release_doc/0.9.0/doc/_sources/source/cuda/utils/index.txt

+Utils
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  utils.rst

release_doc/0.9.0/doc/_sources/source/cuda/utils/utils.txt

+Utilities
+===========
+
+HPPL Base
+------------
+
+hl_base.h
+``````````````
+.. doxygenfile:: paddle/cuda/include/hl_base.h
+
+Timer
+-----------
+
+hl_time.h
+``````````````
+.. doxygenfile:: paddle/cuda/include/hl_time.h
+
+Thread Resource
+-----------
+
+hl_thread.ph
+``````````````
+.. doxygenfile:: paddle/cuda/include/hl_thread.ph

release_doc/0.9.0/doc/_sources/source/gserver/activations/index.txt

+Activations
+=============
+
+..  doxygenclass:: paddle::ActivationFunction
+    :members:

release_doc/0.9.0/doc/_sources/source/gserver/dataprovider/dataproviders.txt

+Data Providers
+================
+
+Base DataProvider
+------------------
+..  doxygenclass:: paddle::DataProvider
+    :members:
+
+DataProviderGroup
+-------------------
+..  doxygenclass:: paddle::DataProviderGroup
+    :members:
+
+MultiDataProvider
+-------------------
+..  doxygenclass:: paddle::MultiDataProvider
+    :members:
+
+PyDataProvider
+===================
+
+IFieldScanner
+-------------
+..  doxygenclass:: paddle::IFieldScanner
+    :members:
+
+DenseScanner
+-------------
+..  doxygenclass:: paddle::DenseScanner
+    :members:
+
+IndexScanner
+-------------
+..  doxygenclass:: paddle::IndexScanner
+    :members:
+
+SparseNonValueScanner
+---------------------
+..  doxygenclass:: paddle::SparseNonValueScanner
+    :members:
+
+SparseValueScanner
+------------------
+..  doxygenclass:: paddle::SparseValueScanner
+    :members:
+
+SequenceScanner
+------------------
+..  doxygenclass:: paddle::SparseValueScanner
+    :members:
+
+IPyDataProviderCache
+--------------------
+..  doxygenclass:: paddle::IPyDataProviderCache
+    :members:
+
+NoCacheStrategy
+---------------
+..  doxygenclass:: paddle::NoCacheStrategy
+    :members:
+
+CacheOnePassInMemory
+--------------------
+..  doxygenclass:: paddle::CacheOnePassInMemory
+    :members:
+
+IPyDataProvider
+---------------
+..  doxygenclass:: paddle::PyDataProvider2
+    :members:
+
+Proto Data Provider
+===================
+
+ProtoDataProvider
+----------------
+..  doxygenclass:: paddle::ProtoDataProvider
+    :members:
+
+ProtoSequenceDataProvider
+----------------
+..  doxygenclass:: paddle::ProtoSequenceDataProvider
+    :members:

release_doc/0.9.0/doc/_sources/source/gserver/dataprovider/index.txt

+Data Providers Documents
+==========================
+
+.. toctree::
+  :maxdepth: 3
+
+  dataproviders.rst

release_doc/0.9.0/doc/_sources/source/gserver/evaluators/evaluators.txt

+Base Evaluator
+==============
+
+Evaluator
+---------
+..  doxygenclass:: paddle::Evaluator
+    :members:
+
+
+Utils
+=====
+
+SumEvaluator
+------------
+..  doxygenclass:: paddle::SumEvaluator
+    :members:
+
+ColumnSumEvaluator
+------------------
+..  doxygenclass:: paddle::ColumnSumEvaluator
+    :members:
+
+Classification
+==============
+
+ClassificationErrorEvaluator
+---------------------------
+..  doxygenclass:: paddle::ClassificationErrorEvaluator
+    :members:
+
+SequenceClassificationErrorEvaluator
+------------------------------------
+..  doxygenclass:: paddle::SequenceClassificationErrorEvaluator
+    :members:
+
+AucEvaluator
+-------------
+..  doxygenclass:: paddle::AucEvaluator
+    :members:
+
+PrecisionRecallEvaluator
+------------------------
+..  doxygenclass:: paddle::PrecisionRecallEvaluator
+    :members:
+
+ChunkEvaluator
+--------------
+..  doxygenclass:: paddle::ChunkEvaluator
+    :members:
+
+CTCEvaluator
+------------
+..  doxygenclass:: paddle::CTCErrorEvaluator
+    :members:
+
+
+Rank
+====
+
+PnpairEvaluator
+-------------
+..  doxygenclass:: paddle::PnpairEvaluator
+    :members:
+
+AucEvaluator
+-------------
+..  doxygenclass:: paddle::RankAucEvaluator
+    :members:
+
+
+Printer
+=======
+
+ValuePrinter
+-------------
+..  doxygenclass:: paddle::ValuePrinter
+    :members:
+
+GradientPrinter
+---------------
+..  doxygenclass:: paddle::GradientPrinter
+    :members:
+
+MaxIdPrinter
+------------
+..  doxygenclass:: paddle::MaxIdPrinter
+    :members:
+
+MaxFramePrinter
+---------------
+..  doxygenclass:: paddle::MaxFramePrinter
+    :members:
+
+SequenceTextPrinter
+------------------
+..  doxygenclass:: paddle::SequenceTextPrinter
+    :members:
+
+ClassificationErrorPrinter
+--------------------------
+..  doxygenclass:: paddle::ClassificationErrorPrinter
+    :members:

release_doc/0.9.0/doc/_sources/source/gserver/evaluators/index.txt

+Evaluators
+==========
+
+.. toctree::
+  :maxdepth: 3
+
+  evaluators.rst

release_doc/0.9.0/doc/_sources/source/gserver/gradientmachines/gradientmachines.txt

+Gradient Machines
+================
+
+GradientMachine
+---------------------
+..  doxygenclass:: paddle::GradientMachine
+    :members:
+
+GradientMachineModel
+--------------------
+..  doxygenclass:: paddle::IGradientMachineMode
+    :members:
+
+MultiGradientMachine
+---------------------
+..  doxygenclass:: paddle::MultiGradientMachine
+    :members:
+
+TrainerThread
+`````````````
+..  doxygenclass:: paddle::TrainerThread
+    :members:
+
+Recurrent Gradient Machines
+---------------------------
+..  doxygenclass:: paddle::RecurrentGradientMachine
+    :members:
+
+Networks
+========
+
+NeuralNetwork
+-------------
+..  doxygenclass:: paddle::NeuralNetwork
+    :members:
+
+ParallelNeuralNetwork
+---------------------
+..  doxygenclass:: paddle::ParallelNeuralNetwork
+    :members:

release_doc/0.9.0/doc/_sources/source/gserver/gradientmachines/index.txt

+Gradient Machines Documents
+=============================
+
+.. toctree::
+  :maxdepth: 3
+
+  gradientmachines.rst

release_doc/0.9.0/doc/_sources/source/gserver/layers/index.txt

+Layers Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  layer.rst

release_doc/0.9.0/doc/_sources/source/index.txt

+# Source Code Documents
+
+## cuda
+
+- [CUDA](cuda/cuda/index.rst)
+- [Matrix](cuda/matrix/index.rst)
+- [RNN](cuda/rnn/index.rst)
+- [Utils](cuda/utils/index.rst)
+
+## gserver
+
+- [Activations](gserver/activations/index.rst)
+- [Data Providers](gserver/dataprovider/index.rst)
+- [Evaluators](gserver/evaluators/index.rst)
+- [Gradient Machines](gserver/gradientmachines/index.rst)
+- [Layers](gserver/layers/index.rst)
+
+## math
+
+- [Matrix](math/matrix/index.rst)
+- [Utils](math/utils/index.rst)
+
+## parameter
+
+- [Parameter](parameter/parameter/index.rst)
+- [Update](parameter/update/index.rst)
+- [Optimizer](parameter/optimizer/index.rst)
+
+## pserver
+
+- [Client](pserver/client/index.rst)
+- [Network](pserver/network/index.rst)
+- [Server](pserver/server/index.rst)
+
+## trainer
+
+- [Trainer](trainer/trainer.rst)
+
+## api
+
+- [API](api/api.rst)
+
+## utils
+
+- [CustomStackTrace](utils/customStackTrace.rst)
+- [Enumeration wrapper](utils/enum.rst)
+- [Lock](utils/lock.rst)
+- [Queue](utils/queue.rst)
+- [Thread](utils/thread.rst)

release_doc/0.9.0/doc/_sources/source/math/matrix/index.txt

+Matrix Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  matrix.rst

release_doc/0.9.0/doc/_sources/source/math/matrix/matrix.txt

+Matrix
+=======
+
+Base
+--------
+.. doxygenfile:: paddle/math/BaseMatrix.h
+
+Sparse Matrix
+----------------
+.. doxygenfile:: paddle/math/Matrix.h
+.. doxygenfile:: paddle/math/Vector.h
+.. doxygenfile:: paddle/math/MathUtils.h
+.. doxygenfile:: paddle/math/SparseMatrix.h
+.. doxygenfile:: paddle/math/SparseRowMatrix.h
+.. doxygenfile:: paddle/math/CpuSparseMatrix.h
+
+Others
+----------
+.. doxygenfile:: paddle/math/MathFunctions.h
+.. doxygenfile:: paddle/math/SIMDFunctions.h

release_doc/0.9.0/doc/_sources/source/math/utils/index.txt

+Utils Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  utils.rst

release_doc/0.9.0/doc/_sources/source/math/utils/utils.txt

+Utils
+=======
+
+Memory Handle
+--------------
+.. doxygenfile:: paddle/math/MemoryHandle.h
+.. doxygenfile:: paddle/math/Allocator.h
+.. doxygenfile:: paddle/math/PoolAllocator.h
+.. doxygenfile:: paddle/math/Storage.h

release_doc/0.9.0/doc/_sources/source/parameter/optimizer/index.txt

+Parameter Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  optimizer.rst

release_doc/0.9.0/doc/_sources/source/parameter/optimizer/optimizer.txt

+Optimizer
+============
+
+.. doxygenfile:: paddle/parameter/FirstOrderOptimizer.h
+.. doxygenfile:: paddle/parameter/AverageOptimizer.h
+.. doxygenfile:: paddle/parameter/ParameterOptimizer.h
+.. doxygenfile:: paddle/parameter/OptimizerWithRegularizer.h

release_doc/0.9.0/doc/_sources/source/parameter/parameter/index.txt

+Parameter Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  parameter.rst

release_doc/0.9.0/doc/_sources/source/parameter/parameter/parameter.txt

+Parameter
+=============
+
+Weight
+--------
+.. doxygenfile:: paddle/parameter/Weight.h
+
+Regularizer
+------------
+.. doxygenfile:: paddle/parameter/Regularizer.h
+
+Parameter
+-------------
+.. doxygenfile:: paddle/parameter/Argument.h
+.. doxygenfile:: paddle/parameter/Parameter.h
+.. doxygenfile:: paddle/parameter/ParallelParameter.h

release_doc/0.9.0/doc/_sources/source/parameter/update/index.txt

+Parameter Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  update.rst

release_doc/0.9.0/doc/_sources/source/parameter/update/update.txt

+Update
+==========
+
+.. doxygenfile:: paddle/parameter/ParameterUpdaterBase.h
+.. doxygenfile:: paddle/parameter/ParameterUpdaterHook.h
+.. doxygenfile:: paddle/parameter/ParameterUpdateFunctions.h
+

release_doc/0.9.0/doc/_sources/source/pserver/client/client.txt

+Client
+=========
+
+.. doxygenclass:: paddle::BaseClient
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:
+
+.. doxygenclass:: paddle::ParameterClient2
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:

release_doc/0.9.0/doc/_sources/source/pserver/client/index.txt

+Client Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  client.rst

release_doc/0.9.0/doc/_sources/source/pserver/network/index.txt

+Network Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  network.rst

release_doc/0.9.0/doc/_sources/source/pserver/network/network.txt

+Network
+==========
+
+Socket Server
+----------------
+.. doxygenclass:: paddle::SocketServer
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:
+
+Socket Worker
+----------------
+.. doxygenclass:: paddle::SocketWorker
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:
+
+Socket Client
+----------------
+.. doxygenclass:: paddle::SocketClient
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:
+
+Socket Channel
+---------------
+.. doxygenclass:: paddle::SocketChannel
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:
+
+Message Reader
+---------------
+.. doxygenclass:: paddle::MsgReader
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:

release_doc/0.9.0/doc/_sources/source/pserver/server/index.txt

+Server Documents
+====================
+
+.. toctree::
+  :maxdepth: 3
+
+  server.rst

release_doc/0.9.0/doc/_sources/source/pserver/server/server.txt

+Server
+==========
+
+.. doxygenclass:: paddle::ProtoServer
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:
+
+.. doxygenclass:: paddle::ParameterServer2
+    :members:
+    :protected-members:
+    :private-members:
+    :undoc-members:

release_doc/0.9.0/doc/_sources/source/trainer/trainer.txt

+Trainer
+=======
+
+TrainerStats
+------------
+
+..  doxygenclass:: paddle::TrainerStats
+    :members:
+
+RemoteParameterUpdater
+-----------------------
+
+..  doxygenclass:: paddle::RemoteParameterUpdater
+    :members:
+
+ConcurrentRemoteParameterUpdater
+---------------------------------
+
+..  doxygenclass:: paddle::ConcurrentRemoteParameterUpdater
+    :members:
+
+SparseRemoteParameterUpdater
+----------------------------
+
+..  doxygenclass:: paddle::SparseRemoteParameterUpdater
+    :members:
+
+SparseRemoteParameterUpdaterComposite
+-------------------------------------
+
+..  doxygenclass:: paddle::SparseRemoteParameterUpdaterComposite
+    :members:

release_doc/0.9.0/doc/_sources/source/utils/customStackTrace.txt

+CustomStackTrace
+================
+
+
+class CustomStackTrace
+----------------------
+
+..  doxygenclass:: paddle::CustomStackTrace
+    :members:

release_doc/0.9.0/doc/_sources/source/utils/enum.txt

+enumeration_wrapper
+===================
+
+
+namespace paddle::enumeration_wrapper
+-------------------------------------
+
+..  doxygennamespace:: paddle::enumeration_wrapper
+

release_doc/0.9.0/doc/_sources/source/utils/lock.txt

+Thread
+======
+
+
+class Thread 
+------------
+
+..  doxygenclass:: paddle::Thread
+    :members:
+
+
+class ThreadWorker
+------------------
+
+..  doxygenclass:: paddle::ThreadWorker
+    :members:
+    
+
+class SyncThreadPool 
+--------------------
+
+..  doxygenclass:: paddle::SyncThreadPool 
+    :members:
+    
+
+class MultiThreadWorker 
+-----------------------
+
+..  doxygenclass:: paddle::MultiThreadWorker 
+    :members:
+    
+
+class AsyncThreadPool 
+---------------------
+
+..  doxygenclass:: paddle::AsyncThreadPool 
+    :members:

release_doc/0.9.0/doc/_sources/source/utils/queue.txt

+Queue
+=====
+
+
+class Queue
+------------
+
+..  doxygenclass:: paddle::Queue
+    :members:
+
+
+class BlockingQueue 
+-------------------
+
+..  doxygenclass:: paddle::BlockingQueue 
+    :members:

release_doc/0.9.0/doc/_sources/source/utils/thread.txt

+Lock
+====
+
+
+class RWLock
+------------
+
+..  doxygenclass:: paddle::RWLock
+    :members:
+
+class ReadLockGuard
+-------------------
+
+..  doxygenclass:: paddle::ReadLockGuard
+    :members:
+
+class SpinLock
+--------------
+
+..  doxygenclass:: paddle::SpinLock
+    :members:
+
+class Semaphore
+---------------
+
+..  doxygenclass:: paddle::Semaphore
+    :members:
+
+class ThreadBarrier
+-------------------
+
+..  doxygenclass:: paddle::ThreadBarrier
+    :members:
+
+class LockedCondition
+---------------------
+
+..  doxygenclass:: paddle::LockedCondition
+    :members:
+

release_doc/0.9.0/doc/_sources/ui/api/trainer_config_helpers/activations.txt

+===========
+Activations
+===========
+
+BaseActivation
+==============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: BaseActivation
+    :noindex:
+    
+AbsActivation
+===============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: AbsActivation
+    :noindex:
+    
+ExpActivation
+===============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: ExpActivation
+    :noindex:
+    
+IdentityActivation
+==================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: IdentityActivation
+    :noindex:
+    
+LinearActivation
+==================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: LinearActivation
+    :noindex:
+
+LogActivation
+==================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: LogActivation
+    :noindex:
+    
+SquareActivation
+================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: SquareActivation
+    :noindex:
+    
+SigmoidActivation
+=================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: SigmoidActivation
+    :noindex:
+    
+SoftmaxActivation
+=================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: SoftmaxActivation
+    :noindex:
+    
+SequenceSoftmaxActivation
+=========================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: SequenceSoftmaxActivation
+    :noindex:
+    
+ReluActivation
+==============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: ReluActivation
+    :noindex:
+    
+BReluActivation
+===============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: BReluActivation
+    :noindex:
+    
+SoftReluActivation
+==================
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: SoftReluActivation
+    :noindex:
+    
+TanhActivation
+==============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: TanhActivation
+    :noindex:
+    
+STanhActivation
+===============
+
+..  automodule:: paddle.trainer_config_helpers.activations
+    :members: STanhActivation
+    :noindex:

release_doc/0.9.0/doc/_sources/ui/api/trainer_config_helpers/attrs.txt

+Parameter and Extra Layer Attribute
+===================================
+
+..  automodule:: paddle.trainer_config_helpers.attrs
+    :members:

release_doc/0.9.0/doc/_sources/ui/api/trainer_config_helpers/data_sources.txt

+DataSources
+===========
+
+..  automodule:: paddle.trainer_config_helpers.data_sources
+    :members: