Merge branch 'develop' into warpctc

CMakeLists.txt

@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 2.8)
 project(paddle CXX C)
 set(PADDLE_MAJOR_VERSION 0)
 set(PADDLE_MINOR_VERSION 9)
-set(PADDLE_PATCH_VERSION 0a0)
+set(PADDLE_PATCH_VERSION 0)
 set(PADDLE_VERSION ${PADDLE_MAJOR_VERSION}.${PADDLE_MINOR_VERSION}.${PADDLE_PATCH_VERSION})
 
 set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake")

RELEASE.md

+# Release v0.9.0
+
+## New Features:
+
+* New Layers
+  * bilinear interpolation layer.
+  * spatial pyramid-pool layer.
+  * de-convolution layer.
+  * maxout layer.
+* Support rectangle padding, stride, window and input for Pooling Operation.
+* Add —job=time in trainer, which can be used to print time info without compiler option -WITH_TIMER=ON.
+* Expose cost_weight/nce_layer in `trainer_config_helpers`
+* Add FAQ, concepts, h-rnn docs.
+* Add Bidi-LSTM and DB-LSTM to quick start demo @alvations
+* Add usage track scripts.
+
+## Improvements
+
+* Add Travis-CI for Mac OS X. Enable swig unittest in Travis-CI. Skip Travis-CI when only docs are changed.
+* Add code coverage tools.
+* Refine convolution layer to speedup and reduce GPU memory.
+* Speed up PyDataProvider2
+* Add ubuntu deb package build scripts.
+* Make Paddle use git-flow branching model.
+* PServer support no parameter blocks.
+
+## Bug Fixes
+
+* add zlib link to py_paddle
+* add input sparse data check for sparse layer at runtime
+* Bug fix for sparse matrix multiplication
+* Fix floating-point overflow problem of tanh
+* Fix some nvcc compile options
+* Fix a bug in yield dictionary in DataProvider
+* Fix SRL hang when exit.
+
+# Release v0.8.0beta.1
+New features:
+
+* Mac OSX is supported by source code. #138
+   * Both GPU and CPU versions of PaddlePaddle are supported.
+
+* Support CUDA 8.0
+
+* Enhance `PyDataProvider2`
+   * Add dictionary yield format. `PyDataProvider2` can yield a dictionary with key is data_layer's name, value is features.
+   * Add `min_pool_size` to control memory pool in provider.
+
+* Add `deb` install package & docker image for no_avx machines.
+   * Especially for cloud computing and virtual machines
+
+* Automatically disable `avx` instructions in cmake when machine's CPU don't support `avx` instructions.
+
+* Add Parallel NN api in trainer_config_helpers.
+
+* Add `travis ci` for Github
+
+Bug fixes:
+
+* Several bugs in trainer_config_helpers. Also complete the unittest for trainer_config_helpers
+* Check if PaddlePaddle is installed when unittest.
+* Fix bugs in GTX series GPU
+* Fix bug in MultinomialSampler
+
+Also more documentation was written since last release.
+
+# Release v0.8.0beta.0
+
+PaddlePaddle v0.8.0beta.0 release. The install package is not stable yet and it's a pre-release version.

doc/about/index.rst

+ABOUT
+=======
+
+PaddlPaddle is an easy-to-use, efficient, flexible and scalable deep learning platform,
+which is originally developed by Baidu scientists and engineers for the purpose of applying deep learning to many products at Baidu.
+
+PaddlePaddle is now open source but far from complete, which is intended to be built upon, improved, scaled, and extended.
+We hope to build an active open source community both by providing feedback and by actively contributing to the source code.
+
+
+Credits
+--------
+
+We owe many thanks to `all contributors and developers <https://github.com/PaddlePaddle/Paddle/blob/develop/authors>`_ of PaddlePaddle!

doc/algorithm/rnn/bi_lstm.jpg

-../../demo/sentiment_analysis/bi_lstm.jpg
\ No newline at end of file

doc/algorithm/rnn/encoder-decoder-attention-model.png

-../../demo/text_generation/encoder-decoder-attention-model.png
\ No newline at end of file

doc/ui/data_provider/index.rst → doc/api/data_provider/index.rst

-DataProvider Introduction
-=========================
+Introduction
+==============
 DataProvider is a module that loads training or testing data into cpu or gpu
 memory for the following triaining or testing process.
 

doc/ui/data_provider/pydataprovider2.rst → doc/api/data_provider/pydataprovider2.rst

-How to use PyDataProvider2
-==========================
+PyDataProvider2
+=================
 
 We highly recommand users to use PyDataProvider2 to provide training or testing
 data to PaddlePaddle. The user only needs to focus on how to read a single

doc/api/index.rst

+API
+====
+
+DataProvider API
+----------------
+
+..  toctree::
+  :maxdepth: 1
+
+  data_provider/index.rst
+  data_provider/pydataprovider2.rst
+
+Model Config API
+----------------
+
+..  toctree::
+  :maxdepth: 1
+
+  trainer_config_helpers/index.rst
+  trainer_config_helpers/optimizers.rst
+  trainer_config_helpers/data_sources.rst
+  trainer_config_helpers/layers.rst
+  trainer_config_helpers/activations.rst 
+  trainer_config_helpers/poolings.rst
+  trainer_config_helpers/networks.rst
+  trainer_config_helpers/evaluators.rst
+  trainer_config_helpers/attrs.rst
+
+
+Applications API
+----------------
+
+..  toctree::
+  :maxdepth: 1
+
+  predict/swig_py_paddle_en.rst
\ No newline at end of file

doc/ui/predict/swig_py_paddle_en.rst → doc/api/predict/swig_py_paddle_en.rst

-Python Prediction API
-=====================
+Python Prediction
+==================
 
 PaddlePaddle offers a set of clean prediction interfaces for python with the help of
 SWIG. The main steps of predict values in python are:

doc/ui/api/trainer_config_helpers/attrs.rst → doc/api/trainer_config_helpers/attrs.rst

-Parameter and Extra Layer Attribute
-===================================
+Parameter Attributes
+=======================
 
 ..  automodule:: paddle.trainer_config_helpers.attrs
     :members:

doc/cluster/index.rst

-Cluster Train
-====================
-
-.. toctree::
-  :glob:
-
-  opensource/cluster_train.md
-  internal/index.md

doc/dev/index.rst

-Development Guide
-=================
-
-..  toctree::
-  :maxdepth: 1
-
-  layer.md
-  new_layer/new_layer.rst
-  ../source/index.md

doc/dev/layer.md

-# Layer Documents
-
-* [Layer Source Code Document](../source/gserver/layers/index.rst)
-* [Layer Python API Document](../ui/api/trainer_config_helpers/index.rst)

doc/introduction/index.md → doc/getstarted/basic_usage/basic_usage.rst

-# Introduction
+Basic Usage
+=============
 
 PaddlePaddle is a deep learning platform open-sourced by Baidu. With PaddlePaddle, you can easily train a classic neural network within a couple lines of configuration, or you can build sophisticated models that provide state-of-the-art performance on difficult learning tasks like sentiment analysis, machine translation, image caption and so on.
 
-## 1. A Classic Problem
+1. A Classic Problem
+---------------------
 
-Now, to give you a hint of what using PaddlePaddle looks like, let's start with a fundamental learning problem - <a href="https://en.wikipedia.org/wiki/Simple_linear_regression">**simple linear regression**</a> : you have observed a set of two-dimensional data points of `X` and `Y`, where `X` is an explanatory variable and `Y` is corresponding dependent variable, and you want to recover the underlying correlation between `X` and `Y`. Linear regression can be used in many practical scenarios. For example, `X` can be a variable about house size, and `Y` a variable about house price. You can build a model that captures relationship between them by observing real estate markets.
+Now, to give you a hint of what using PaddlePaddle looks like, let's start with a fundamental learning problem - `simple linear regression <https://en.wikipedia.org/wiki/Simple_linear_regression>`_: you have observed a set of two-dimensional data points of ``X`` and ``Y``, where ``X`` is an explanatory variable and ``Y`` is corresponding dependent variable, and you want to recover the underlying correlation between ``X`` and ``Y``. Linear regression can be used in many practical scenarios. For example, ``X`` can be a variable about house size, and ``Y`` a variable about house price. You can build a model that captures relationship between them by observing real estate markets.
 
-## 2. Prepare the Data
+2. Prepare the Data
+--------------------
 
-Suppose the true relationship can be characterized as `Y = 2X + 0.3`, let's see how to recover this pattern only from observed data. Here is a piece of python code that feeds synthetic data to PaddlePaddle. The code is pretty self-explanatory, the only extra thing you need to add for PaddlePaddle is a definition of input data types.
+Suppose the true relationship can be characterized as ``Y = 2X + 0.3``, let's see how to recover this pattern only from observed data. Here is a piece of python code that feeds synthetic data to PaddlePaddle. The code is pretty self-explanatory, the only extra thing you need to add for PaddlePaddle is a definition of input data types.
 
-```python
-# dataprovider.py
-from paddle.trainer.PyDataProvider2 import *
-import random
+    .. code-block:: python
 
-# define data types of input: 2 real numbers
-@provider(input_types=[dense_vector(1), dense_vector(1)],use_seq=False)
-def process(settings, input_file):
+        # dataprovider.py
+        from paddle.trainer.PyDataProvider2 import *
+        import random
+
+        # define data types of input: 2 real numbers
+        @provider(input_types=[dense_vector(1), dense_vector(1)],use_seq=False)
+        def process(settings, input_file):
             for i in xrange(2000):
                 x = random.random()
                 yield [x], [2*x+0.3]
-```
 
-## 3. Train a NeuralNetwork in PaddlePaddle
+3. Train a NeuralNetwork
+-------------------------
+
+To recover this relationship between ``X`` and ``Y``, we use a neural network with one layer of linear activation units and a square error cost layer. Don't worry if you are not familiar with these terminologies, it's just saying that we are starting from a random line ``Y' = wX + b`` , then we gradually adapt ``w`` and ``b`` to minimize the difference between ``Y'`` and ``Y``. Here is what it looks like in PaddlePaddle:
 
-To recover this relationship between `X` and `Y`, we use a neural network with one layer of linear activation units and a square error cost layer. Don't worry if you are not familiar with these terminologies, it's just saying that we are starting from a random line `Y' = wX + b` , then we gradually adapt `w` and `b` to minimize the difference between `Y'` and `Y`. Here is what it looks like in PaddlePaddle:
+    .. code-block:: python
 
-```python
-# trainer_config.py
-from paddle.trainer_config_helpers import *
+        # trainer_config.py
+        from paddle.trainer_config_helpers import *
 
-# 1. read data. Suppose you saved above python code as dataprovider.py
-data_file = 'empty.list'
-with open(data_file, 'w') as f: f.writelines(' ')
-define_py_data_sources2(train_list=data_file, test_list=None, 
+        # 1. read data. Suppose you saved above python code as dataprovider.py
+        data_file = 'empty.list'
+        with open(data_file, 'w') as f: f.writelines(' ')
+        define_py_data_sources2(train_list=data_file, test_list=None, 
                 module='dataprovider', obj='process',args={})
 
-# 2. learning algorithm
-settings(batch_size=12, learning_rate=1e-3, learning_method=MomentumOptimizer())
+        # 2. learning algorithm
+        settings(batch_size=12, learning_rate=1e-3, learning_method=MomentumOptimizer())
 
-# 3. Network configuration
-x = data_layer(name='x', size=1)
-y = data_layer(name='y', size=1)
-y_predict = fc_layer(input=x, param_attr=ParamAttr(name='w'), size=1, act=LinearActivation(), bias_attr=ParamAttr(name='b'))
-cost = regression_cost(input=y_predict, label=y)
-outputs(cost)
-```
+        # 3. Network configuration
+        x = data_layer(name='x', size=1)
+        y = data_layer(name='y', size=1)
+        y_predict = fc_layer(input=x, param_attr=ParamAttr(name='w'), size=1, act=LinearActivation(), bias_attr=ParamAttr(name='b'))
+        cost = regression_cost(input=y_predict, label=y)
+        outputs(cost)
 
 Some of the most fundamental usages of PaddlePaddle are demonstrated:
 
@@ -55,46 +59,51 @@ Some of the most fundamental usages of PaddlePaddle are demonstrated:
 -  The second part describes learning algorithm. It defines in what ways adjustments are made to model parameters. PaddlePaddle provides a rich set of optimizers, but a simple momentum based optimizer will suffice here, and it processes 12 data points each time.
 
 -  Finally, the network configuration. It usually is as simple as "stacking" layers. Three kinds of layers are used in this configuration:
-	-  **Data Layer**: a network always starts with one or more data layers. They provide input data to the rest of the network. In this problem, two data layers are used respectively for `X` and `Y`.
+	-  **Data Layer**: a network always starts with one or more data layers. They provide input data to the rest of the network. In this problem, two data layers are used respectively for ``X`` and ``Y``.
 	-  **FC Layer**: FC layer is short for Fully Connected Layer, which connects all the input units to current layer and does the actual computation specified as activation function. Computation layers like this are the fundamental building blocks of a deeper model.
 	-  **Cost Layer**: in training phase, cost layers are usually the last layers of the network. They measure the performance of current model, and provide guidence to adjust parameters.
 
 Now that everything is ready, you can train the network with a simple command line call:
- ```
+
+    .. code-block:: bash
+ 
         paddle train --config=trainer_config.py --save_dir=./output --num_passes=30
- ```
  
-This means that PaddlePaddle will train this network on the synthectic dataset for 30 passes, and save all the models under path `./output`. You will see from the messages printed out during training phase that the model cost is decreasing as time goes by, which indicates we are getting a closer guess.
 
+This means that PaddlePaddle will train this network on the synthectic dataset for 30 passes, and save all the models under path ``./output``. You will see from the messages printed out during training phase that the model cost is decreasing as time goes by, which indicates we are getting a closer guess.
+
+
+4. Evaluate the Model
+-----------------------
 
-## 4. Evaluate the Model
+Usually, a different dataset that left out during training phase should be used to evalute the models. However, we are lucky enough to know the real answer: ``w=2, b=0.3``, thus a better option is to check out model parameters directly.
 
-Usually, a different dataset that left out during training phase should be used to evalute the models. However, we are lucky enough to know the real answer: `w=2, b=0.3`, thus a better option is to check out model parameters directly.
+In PaddlePaddle, training is just to get a collection of model parameters, which are ``w`` and ``b`` in this case. Each parameter is saved in an individual file in the popular ``numpy`` array format. Here is the code that reads parameters from last pass.
 
-In PaddlePaddle, training is just to get a collection of model parameters, which are `w` and `b` in this case. Each parameter is saved in an individual file in the popular `numpy` array format. Here is the code that reads parameters from last pass.
+    .. code-block:: python
 
-```python
-import numpy as np
-import os
+        import numpy as np
+        import os
 
-def load(file_name):
+        def load(file_name):
             with open(file_name, 'rb') as f:
                 f.read(16) # skip header for float type.
                 return np.fromfile(f, dtype=np.float32)
                 
-print 'w=%.6f, b=%.6f' % (load('output/pass-00029/w'), load('output/pass-00029/b'))
-# w=1.999743, b=0.300137
-```
+        print 'w=%.6f, b=%.6f' % (load('output/pass-00029/w'), load('output/pass-00029/b'))
+        # w=1.999743, b=0.300137
 
-<center> ![](./parameters.png) </center>
+    .. image:: parameters.png
+        :align: center
 
-Although starts from a random guess, you can see that value of `w` changes quickly towards 2 and `b` changes quickly towards 0.3. In the end, the predicted line is almost identical with real answer.
+Although starts from a random guess, you can see that value of ``w`` changes quickly towards 2 and ``b`` changes quickly towards 0.3. In the end, the predicted line is almost identical with real answer.
 
-There, you have recovered the underlying pattern between `X` and `Y` only from observed data.
+There, you have recovered the underlying pattern between ``X`` and ``Y`` only from observed data.
 
 
-## 5. Where to Go from Here
+5. Where to Go from Here
+-------------------------
 
-- <a href="../build/index.html"> Build and Installation </a>
-- <a href="../demo/quick_start/index_en.html">Quick Start</a>
-- <a href="../demo/index.html">Example and Demo</a>
+- `Install and Build <../build_and_install/index.html>`_
+- `Tutorials <../demo/quick_start/index_en.html>`_
+- `Example and Demo <../demo/index.html>`_

doc/build/index.rst → doc/getstarted/build_and_install/index.rst

@@ -8,8 +8,6 @@ Install PaddlePaddle
     :maxdepth: 1
     :glob:
 
-    install_*
-    internal/install_from_jumbo.md
     docker_install.rst
     ubuntu_install.rst
 
@@ -25,4 +23,3 @@ Build from Source
     :glob:
 
     build_from_source.md
\ No newline at end of file
-    contribute_to_paddle.md

doc/getstarted/index.rst

+GET STARTED
+============
+
+..  toctree::
+  :maxdepth: 2
+
+  build_and_install/index.rst
+  basic_usage/basic_usage.rst

doc/cluster/opensource/cluster_train.md → doc/howto/cluster/cluster_train.md

-# Distributed Training
+# How to Run Distributed Training
 
 In this article, we explain how to run distributed Paddle training jobs on clusters.  We will create the distributed version of the single-process training example, [recommendation](https://github.com/baidu/Paddle/tree/develop/demo/recommendation).
 
@@ -9,7 +9,7 @@ In this article, we explain how to run distributed Paddle training jobs on clust
 1. Aforementioned scripts use a Python library [fabric](http://www.fabfile.org/) to run SSH commands.  We can use `pip` to install fabric:
 
    ```bash
-pip install fabric
+   pip install fabric
    ```
 
 1. We need to install PaddlePaddle on all nodes in the cluster.  To enable GPUs, we need to install CUDA in `/usr/local/cuda`; otherwise Paddle would report errors at runtime.

doc/howto/cmd_parameter/index.md

+# How to Set Command-line Parameters
+
+* [Use Case](use_case.md)
+* [Arguments](arguments.md)
+* [Detailed Descriptions](detail_introduction.md)

doc/build/contribute_to_paddle.md → doc/howto/contribute_to_paddle.md

-# Contribute Code
+# How to Contribute Code
 
 We sincerely appreciate your contributions. You can use fork and pull request
 workflow to merge your code. 

doc/algorithm/index.rst → doc/howto/deep_model/index.rst

-Algorithm Tutorial
-==================
+How to Configure Deep Models
+============================
 
 ..  toctree::
   :maxdepth: 1

doc/algorithm/rnn/rnn.rst → doc/howto/deep_model/rnn/rnn.rst

@@ -42,7 +42,7 @@ Simple Gated Recurrent Neural Network
 
 Recurrent neural network process a sequence at each time step sequentially. An example of the architecture of LSTM is listed below.
 
-.. image:: ./bi_lstm.jpg
+.. image:: ../../../tutorials/sentiment_analysis/bi_lstm.jpg
 	 :align: center
 
 Generally speaking, a recurrent network perform the following operations from :math:`t=1` to :math:`t=T`, or reversely from :math:`t=T` to :math:`t=1`.
@@ -101,7 +101,7 @@ Sequence to Sequence Model with Attention
 -----------------------------------------
 We will use the sequence to sequence model with attention as an example to demonstrate how you can configure complex recurrent neural network models. An illustration of the sequence to sequence model with attention is shown in the following figure.
 
-.. image:: ./encoder-decoder-attention-model.png
+.. image:: ../../../tutorials/text_generation/encoder-decoder-attention-model.png
  	 :align: center
 
 In this model, the source sequence :math:`S = \{s_1, \dots, s_T\}` is encoded with a bidirectional gated recurrent neural networks. The hidden states of the bidirectional gated recurrent neural network :math:`H_S = \{H_1, \dots, H_T\}` is called *encoder vector* The decoder is a gated recurrent neural network. When decoding each token :math:`y_t`, the gated recurrent neural network generates a set of weights :math:`W_S^t = \{W_1^t, \dots, W_T^t\}`, which are used to compute a weighted sum of the encoder vector. The weighted sum of the encoder vector is utilized to condition the generation of the token :math:`y_t`.

doc/howto/index.rst

+HOW TO
+=======
+
+Usage
+-------
+
+..  toctree::
+  :maxdepth: 1
+
+  cmd_parameter/index.md
+  deep_model/index.rst
+  cluster/cluster_train.md
+
+Development
+------------
+
+..  toctree::
+  :maxdepth: 1
+
+  new_layer/index.rst
+  contribute_to_paddle.md
+
+Optimization
+-------------
+
+..  toctree::
+  :maxdepth: 1
+
+  optimization/index.rst

doc/dev/new_layer/new_layer.rst → doc/howto/new_layer/index.rst

-==================
-Writing New Layers
-==================
+=======================
+How to Write New Layers
+=======================
 
 This tutorial will guide you to write customized layers in PaddlePaddle. We will utilize fully connected layer as an example to guide you through the following steps for writing a new layer.
 

doc/optimization/index.rst → doc/howto/optimization/index.rst

-Performance Tuning
-==================
+How to Tune GPU Performance
+===========================
 
 .. toctree::
   :maxdepth: 3

doc/index.rst

@@ -4,8 +4,9 @@ PaddlePaddle Documentation
 ..  toctree::
   :maxdepth: 1
 
-  introduction/index.md
-  user_guide.rst
-  dev/index.rst
-  algorithm/index.rst
-  optimization/index.rst
+  getstarted/index.rst
+  tutorials/index.md
+  howto/index.rst
+  api/index.rst
+  about/index.rst 
+ 
\ No newline at end of file

doc/introduction/parameters.png

-../../doc_cn/introduction/parameters.png
\ No newline at end of file

doc/demo/index.md → doc/tutorials/index.md

-# Examples and demos
+# TUTORIALS
 There are serveral examples and demos here.
 
 ## Image