+
+```python
+def G(in):
+ # over-simplified example as G has only one layers:
+ return paddle.layer.fc(in, parameter_name="G")
+
+def D(in);
+ # again, over-simplified:
+ return paddle.layer.fc(in, parameter_name="D")
+
+# Construct the first topology, which contains both D and G.
+# By learning this topology, we update parameters of G.
+d0 = paddle.layer.should_be_false(D(G(paddle.layer.data())))
+
+# Construct a second topology d1, which contains only D. By
+# training this topology, we update parameters of D. Note
+# that d1 share parameters with d0.
+d1 = paddle.layer.should_be_true(D(paddle.layer.data()))
+
+# Create parameters from a list of multiple topologies (models) for
+# the chance to share parameters between these topologies.
+parameters = paddle.parameters.create([d0, d1])
+
+# Iterative training of GAN.
+for ...:
+ train(d0, parameters, reader=read_from_rng, immutable_parameters={"D"})
+ train(d1, parameters, reader=read_from_realistic_images)
+
+# Use d1 for inference:
+print "D thinks a batch of images are realistic ", infer(d1, parameters, read_mnist_images)
+```
+
+
+### Summarization
+
+
+Above two programs reveal some important design concerns:
+
+1. Users describe a topology as an expression of layers. Every layer
+ has a *parameter name*. If the users don't specify it explicitly, it's automatically generated as a unique name. By
+ specifying the parameter name, users can specify the sharing of
+ parameters between layers and even between topologies.
+
+1. `paddle.parameters.create` figures out parameters required by one
+ or more topologies from parameter names of layers. It creates these
+ parameters and returns a `ParameterSet` object, which is in essence
+ a map from *parameter names* to *parameters*.
+
+1. At training and inference time, `paddle.train` and `paddle.infer`
+ requires both a topology and the parameter set that holds the parameters of that topology. There are some reasons:
+
+ 1. This prevents users from forgetting to call
+ `paddle.parameters.create`.
+ 1. `paddle.train` needs to know which parameter set to update.
+ 1. Users could load another (pre-trained) parameter set and use it
+ with a topology in `train.infer`.
+
+1. By specifying the `immutable_parameters` parameter of
+ `paddle.train`, we can forbid the update of these parameters.
+
+
+## Reader
+
+Not all programming frameworks allow users to define I/O functions.
+An example is Google MapReduce, which can only read from text,
+SSTable, and RecordIO files. Hadoop MapReduce allows users to define
+readers and writers by deriving from base classes `Reader` and
+`Writer`. The former is less flexible but also less error-prone. We
+decide to provide the flexibility to users to define their readers.
+
+
+There are some open questions here:
+
+1. **Should a reader return a Python dictionary?**
+
+1. **How to map multiple outputs from a reader to multiple data layers?**
+
+1. **How to easily compose some existing readers to read more data and
+ feed a topology with more data layers?**
+
+
+## Training
+
+The recommended way to training a model is to call `paddle.train`,
+which simply calls `paddle.trainer.Default`, a global variable of
+type `paddle.trainer.SGD`. Equivalently, we can do
+
+```python
+opt = paddle.trainer.SGD(..., paddle.updater.Adam(...))
+opt.train(topology, parameters, reader=read, ...)
+```
+
+### Updater
+
+Please be aware that a trainer can accept an updater as its data
+member, where an updater is a class derived from
+`paddle.trainer.Updater`. This is to make it easier to customize
+trainers, as discussed
+[here](https://github.com/PaddlePaddle/Paddle/issues/1319).
+
+### Event Handler
+
+`paddle.train` and `paddle.trainer.XXX.train` take an optional
+parameter `event_handler`, which should be either `None` or a function
+that handle some events:
+
+1. BeginTraining
+1. EndTraining
+1. BeginIteration
+1. EndIteration
+1. BeginPass
+1. EndPass
+
+where EndPass is sent if and only if the reader yields
+`end_pass=True`.
+
+An example as follows:
+
+```python
+def event_handler(event):
+ if ininstance(event, paddle.event.EndIteration):
+ print paddle.test(...)
+
+paddle.train(topology, parameters, reader, event_handler)
+```
+
+If we are writing a PaddlePaddle program in and for iPython/Jypyter,
+we can use metaplotlib in the event handler to plot a curve of
+cost/error versus iterations, as shown
+[here](https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/).
+
+### Distributed Training
+
+If users want to do distributed training on a cluster, s/he should
+call `paddle.dist_train` and provides access tokens to the cluster as
+a parameter.
+
+For example, if the user has a TLS certificate that allows him to
+access a Kubernetes cluster, s/he should be able to call
+
+```python
+paddle.dist_train(model,
+ trainer=paddle.trainer.SGD(...,
+ paddle.updater.Adam(...)),
+ reader=read,
+ k8s_user="yi",
+ k8s_token="kube_cluster_tls.pem",
+ k8s_job="hello",
+ num_parameter_servers=15)
+```
+
+The pseudo code if `paddle.dist_train` is as follows:
+
+```python
+def dist_train(topology, parameters, trainer, reader, ...):
+ if os.getenv("KUBERNETES_SERVICE_HOST") == None:
+ image_name = k8s_user + '/' + k8s_job
+ docker_build(image_name)
+ docker_push()
+ kube_ctrl_start_job(image_name, k8s_user, k8s_token)
+ else:
+ rank = kube_list_containers_in_job_and_return_current_containers_rank()
+ if rank == 0:
+ master()
+ elif rank < 15:
+ parameter_server()
+ else:
+ trainer.train(model, reader=read)
+```
+
+Please be aware that if a process is running on the Kubernetes
+cluster, it will have some environment variables pre-defined.
+
+If `dist_train` doesn't see these environment variables, it knows
+that it's running on users' personal computer, and it should work as a
+*launcher*. Otherwise, it knows that it's running on the cluster and
+need to figure out its role as either the master, or a trainer, or a
+parameter server.
diff --git a/develop/doc/_sources/design/reader/README.md.txt b/develop/doc/_sources/design/reader/README.md.txt
new file mode 100644
index 0000000000000000000000000000000000000000..f21f7af520df5171798326818ecb97c3bcd14a12
--- /dev/null
+++ b/develop/doc/_sources/design/reader/README.md.txt
@@ -0,0 +1,202 @@
+# Python Data Reader Design Doc
+
+At training and testing time, PaddlePaddle programs need to read data. To ease the users' work to write data reading code, we define that
+
+- A *reader* is a function that reads data (from file, network, random number generator, etc) and yields data items.
+- A *reader creator* is a function that returns a reader function.
+- A *reader decorator* is a function, which accepts one or more readers, and returns a reader.
+- A *batch reader* is a function that reads data (from *reader*, file, network, random number generator, etc) and yields a batch of data items.
+
+and provide function which converts reader to batch reader, frequently used reader creators and reader decorators.
+
+## Data Reader Interface
+
+Indeed, *data reader* doesn't have to be a function that reads and yields data items. It can be any function with no parameter that creates a iterable (anything can be used in `for x in iterable`):
+
+```
+iterable = data_reader()
+```
+
+Element produced from the iterable should be a **single** entry of data, **not** a mini batch. That entry of data could be a single item, or a tuple of items. Item should be of [supported type](http://www.paddlepaddle.org/doc/ui/data_provider/pydataprovider2.html?highlight=dense_vector#input-types) (e.g., numpy 1d array of float32, int, list of int)
+
+An example implementation for single item data reader creator:
+
+```python
+def reader_creator_random_image(width, height):
+ def reader():
+ while True:
+ yield numpy.random.uniform(-1, 1, size=width*height)
+ return reader
+```
+
+An example implementation for multiple item data reader creator:
+```python
+def reader_creator_random_image_and_label(width, height, label):
+ def reader():
+ while True:
+ yield numpy.random.uniform(-1, 1, size=width*height), label
+ return reader
+```
+
+## Batch Reader Interface
+
+*batch reader* can be any function with no parameter that creates a iterable (anything can be used in `for x in iterable`). The output of the iterable should be a batch (list) of data items. Each item inside the list must be a tuple.
+
+Here are valid outputs:
+```python
+# a mini batch of three data items. Each data item consist three columns of data, each of which is 1.
+[(1, 1, 1),
+(2, 2, 2),
+(3, 3, 3)]
+
+# a mini batch of three data items, each data item is a list (single column).
+[([1,1,1],),
+([2,2,2],),
+([3,3,3],),
+```
+
+Please note that each item inside the list must be a tuple, below is an invalid output:
+```python
+ # wrong, [1,1,1] needs to be inside a tuple: ([1,1,1],).
+ # Otherwise it's ambiguous whether [1,1,1] means a single column of data [1, 1, 1],
+ # or three column of datas, each of which is 1.
+[[1,1,1],
+[2,2,2],
+[3,3,3]]
+```
+
+It's easy to convert from reader to batch reader:
+```python
+mnist_train = paddle.dataset.mnist.train()
+mnist_train_batch_reader = paddle.batch(mnist_train, 128)
+```
+
+Also easy to create custom batch reader:
+```python
+def custom_batch_reader():
+ while True:
+ batch = []
+ for i in xrange(128):
+ batch.append((numpy.random.uniform(-1, 1, 28*28),)) # note that it's a tuple being appended.
+ yield batch
+
+mnist_random_image_batch_reader = custom_batch_reader
+```
+
+## Usage
+
+batch reader, mapping from item(s) read to data layer, batch size and number of total pass will be passed into `paddle.train`:
+
+```python
+# two data layer is created:
+image_layer = paddle.layer.data("image", ...)
+label_layer = paddle.layer.data("label", ...)
+
+# ...
+batch_reader = paddle.batch(paddle.dataset.mnist.train(), 128)
+paddle.train(batch_reader, {"image":0, "label":1}, 128, 10, ...)
+```
+
+## Data Reader Decorator
+
+*Data reader decorator* takes a single or multiple data reader, returns a new data reader. It is similar to a [python decorator](https://wiki.python.org/moin/PythonDecorators), but it does not use `@` syntax.
+
+Since we have a strict interface for data readers (no parameter, return a single data item). Data reader can be used flexiable via data reader decorators. Following are a few examples:
+
+### Prefetch Data
+
+Since reading data may take time and training can not proceed without data. It is generally a good idea to prefetch data.
+
+Use `paddle.reader.buffered` to prefetch data:
+
+```python
+buffered_reader = paddle.reader.buffered(paddle.dataset.mnist.train(), 100)
+```
+
+`buffered_reader` will try to buffer (prefetch) `100` data entries.
+
+### Compose Multiple Data Readers
+
+For example, we want to use a source of real images (reusing mnist dataset), and a source of random images as input for [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661).
+
+We can do:
+
+```python
+def reader_creator_random_image(width, height):
+ def reader():
+ while True:
+ yield numpy.random.uniform(-1, 1, size=width*height)
+ return reader
+
+def reader_creator_bool(t):
+ def reader:
+ while True:
+ yield t
+ return reader
+
+true_reader = reader_creator_bool(True)
+false_reader = reader_creator_bool(False)
+
+reader = paddle.reader.compose(paddle.dataset.mnist.train(), data_reader_creator_random_image(20, 20), true_reader, false_reader)
+# Skipped 1 because paddle.dataset.mnist.train() produces two items per data entry.
+# And we don't care second item at this time.
+paddle.train(paddle.batch(reader, 128), {"true_image":0, "fake_image": 2, "true_label": 3, "false_label": 4}, ...)
+```
+
+### Shuffle
+
+Given shuffle buffer size `n`, `paddle.reader.shuffle` will return a data reader that buffers `n` data entries and shuffle them before a data entry is read.
+
+Example:
+```python
+reader = paddle.reader.shuffle(paddle.dataset.mnist.train(), 512)
+```
+
+## Q & A
+
+### Why reader return only a single entry, but not a mini batch?
+
+Always returning a single entry make reusing existing data readers much easier (e.g., if existing reader return not a single entry but 3 entries, training code will be more complex because it need to handle cases like batch size 2).
+
+We provide function `paddle.batch` to turn (single entry) reader into batch reader.
+
+### Why do we need batch reader, isn't train take reader and batch_size as arguments sufficient?
+
+In most of the case, train taking reader and batch_size as arguments would be sufficent. However sometimes user want to customize order of data entries inside a mini batch. Or even change batch size dynamically.
+
+### Why use a dictionary but not a list to provide mapping?
+
+We decided to use dictionary (`{"image":0, "label":1}`) instead of list (`["image", "label"]`) is because that user can easily resue item (e.g., using `{"image_a":0, "image_b":0, "label":1}`) or skip item (e.g., using `{"image_a":0, "label":2}`).
+
+### How to create custom data reader creator
+
+```python
+def image_reader_creator(image_path, label_path, n):
+ def reader():
+ f = open(image_path)
+ l = open(label_path)
+ images = numpy.fromfile(
+ f, 'ubyte', count=n * 28 * 28).reshape((n, 28 * 28)).astype('float32')
+ images = images / 255.0 * 2.0 - 1.0
+ labels = numpy.fromfile(l, 'ubyte', count=n).astype("int")
+ for i in xrange(n):
+ yield images[i, :], labels[i] # a single entry of data is created each time
+ f.close()
+ l.close()
+ return reader
+
+# images_reader_creator creates a reader
+reader = image_reader_creator("/path/to/image_file", "/path/to/label_file", 1024)
+paddle.train(paddle.batch(reader, 128), {"image":0, "label":1}, ...)
+```
+
+### How is `paddle.train` implemented
+
+An example implementation of paddle.train could be:
+
+```python
+def train(batch_reader, mapping, batch_size, total_pass):
+ for pass_idx in range(total_pass):
+ for mini_batch in batch_reader(): # this loop will never end in online learning.
+ do_forward_backward(mini_batch, mapping)
+```
diff --git a/develop/doc/_sources/getstarted/basic_usage/index_en.rst.txt b/develop/doc/_sources/getstarted/basic_usage/index_en.rst.txt
new file mode 100644
index 0000000000000000000000000000000000000000..c10b897d4292d0c2b062b5c8e23466505afa408a
--- /dev/null
+++ b/develop/doc/_sources/getstarted/basic_usage/index_en.rst.txt
@@ -0,0 +1,101 @@
+Simple Linear Regression
+========================
+
+PaddlePaddle is a deep learning platform open-sourced by Baidu. With PaddlePaddle, you can easily train a classic neural network within a couple lines of configuration, or you can build sophisticated models that provide state-of-the-art performance on difficult learning tasks like sentiment analysis, machine translation, image caption and so on.
+
+Problem Background
+------------------
+
+Now, to give you a hint of what using PaddlePaddle looks like, let's start with a fundamental learning problem - `simple linear regression | Optional | +Description | +
|---|---|
| WITH_GPU | Compile PaddlePaddle with NVIDIA GPU |
| WITH_AVX | Compile PaddlePaddle with AVX intrinsics |
| WITH_DSO | Compile PaddlePaddle with dynamic linked CUDA |
| WITH_TESTING | Compile PaddlePaddle with unit testing |
| WITH_SWIG_PY | Compile PaddlePaddle with inference api |
| WITH_STYLE_CHECK | Compile PaddlePaddle with style check |
| WITH_PYTHON | Compile PaddlePaddle with python interpreter |
| WITH_DOUBLE | Compile PaddlePaddle with double precision |
| WITH_RDMA | Compile PaddlePaddle with RDMA support |
| WITH_TIMER | Compile PaddlePaddle with stats timer |
| WITH_PROFILER | Compile PaddlePaddle with GPU profiler |
| WITH_DOC | Compile PaddlePaddle with documentation |
| ON_COVERALLS | Compile PaddlePaddle with code coverage |
| COVERALLS_UPLOAD | Package code coverage data to coveralls |
| ON_TRAVIS | Exclude special unit test on Travis CI |
| + | args | +local train | +cluster train | +local test | +cluster test | +
|---|---|---|---|---|---|
| common | +job | +√ | √ | √ | √ | +
| use_gpu | +√ | √ | √ | √ | +|
| local | +√ | √ | √ | √ | +|
| config | +√ | √ | √ | √ | +|
| config_args | +√ | √ | √ | √ | +|
| num_passes | +√ | √ | √ | √ | +|
| trainer_count | +√ | √ | √ | √ | +|
| version | +√ | √ | √ | √ | +|
| show_layer_stat | +√ | √ | √ | √ | +|
| train | dot_period | +√ | √ | + | |
| test_period | +√ | √ | + | ||
| saving_period | +√ | √ | + | ||
| show_parameter_stats_period | +√ | √ | + | ||
| init_model_path | +√ | √ | √ | + | |
| load_missing_parameter_strategy | +√ | √ | + | ||
| saving_period_by_batches | +√ | √ | + | ||
| use_old_updater | +√ | √ | + | ||
| enable_grad_share | +√ | √ | + | ||
| grad_share_block_num | +√ | √ | + | ||
| log_error_clipping | +√ | √ | + | ||
| log_clipping | +√ | √ | + | ||
| save_only_one | +√ | √ | + | ||
| start_pass | +√ | √ | + | ||
| train/test | save_dir | +√ | √ | √ | √ | +
| testing during training | test_period | +√ | √ | + | |
| average_test_period | +√ | √ | + | ||
| test | model_list | +√ | √ | +||
| test_wait | +√ | √ | +|||
| test_pass | +√ | √ | +|||
| predict_output_dir | +√ | √ | +|||
| distribute_test | +√ | √ | +|||
| Auc/PnpairValidation | predict_file | +√ | √ | +||
| GPU | gpu_id | +√ | √ | √ | √ | +
| parallel_nn | +√ | √ | √ | √ | +|
| allow_only_one_model_on_one_gpu | +√ | √ | √ | √ | +|
| cudnn_dir | +√ | √ | √ | √ | +|
| cuda_dir | +√ | √ | √ | √ | +|
| cudnn_conv_workspace_limit_in_mb | +√ | √ | √ | √ | +|
| RNN | +beam_size | +√ | √ | +||
| rnn_use_batch | +√ | √ | √ | √ | +|
| prev_batch_state | +√ | √ | + | ||
| diy_beam_search_prob_so | +√ | √ | +|||
| metric learning | external | +√ | √ | √ | √ | +
| data_server_port | +√ | √ | +|||
| PServer | start_pserver | +√ | √ | +||
| pservers | +√ | √ | +|||
| port | +√ | √ | +|||
| port_num | +√ | √ | +|||
| ports_num_for_sparse | +√ | √ | +|||
| nics | +√ | √ | +|||
| rdma_tcp | +√ | √ | +|||
| small_messages | +√ | + | |||
| loadsave_parameters_in_pserver | +√ | √ | +|||
| log_period_server | +√ | + | |||
| pserver_num_threads | +√ | + | |||
| sock_send_buf_size | +√ | + | |||
| sock_recv_buf_size | +√ | + | |||
| num_gradient_servers | +√ | + | |||
| parameter_block_size | +√ | + | |||
| parameter_block_size_for_sparse | +√ | + | |||
| Async SGD | async_count | +√ | + | ||
| async_lagged_ratio_min | +√ | + | |||
| async_lagged_ratio_default | +√ | + | |||
| Performance Tuning | log_barrier_abstract | +√ | + | ||
| log_barrier_lowest_nodes | +√ | + | |||
| log_barrier_show_log | +√ | + | |||
| check_sparse_distribution_batches | +√ | + | |||
| check_sparse_distribution_ratio | +√ | + | |||
| check_sparse_distribution_unbalance_degree | +√ | + | |||
| check_sparse_distribution_in_pserver | +√ | + | |||
| show_check_sparse_distribution_log | +√ | + | |||
| Data Provider | memory_threshold_on_load_data | +√ | √ | + | |
| RandomNumber | seed | +√ | √ | + | |
| thread_local_rand_use_global_seed | +√ | √ | + | ||
| UnitTest | checkgrad_eps | ++ | |||
| Matrix/Vector | enable_parallel_vector | +√ | √ | √ | √ | +
+ Figure 1. GAN-Model-Structure + figure credit +
+ +The generator and discriminator take turn to be trained using SGD. The objective function of the generator is for its generated images being classified as real by the discriminator, and the objective function of the discriminator is to correctly classify real and fake images. When the GAN model is trained to converge to the equilibrium state, the generator will transform the given noise distribution to the distribution of real images, and the discriminator will not be able to distinguish between real and fake images at all. + +## Implementation of GAN Model Structure +Since GAN model involves multiple neural networks, it requires to use paddle python API. So the code walk-through below can also partially serve as an introduction to the usage of Paddle Python API. + +There are three networks defined in gan_conf.py, namely **generator_training**, **discriminator_training** and **generator**. The relationship to the model structure we defined above is that **discriminator_training** is the discriminator, **generator** is the generator, and the **generator_training** combined the generator and discriminator since training generator would require the discriminator to provide loss function. This relationship is described in the following code: +```python +if is_generator_training: + noise = data_layer(name="noise", size=noise_dim) + sample = generator(noise) + +if is_discriminator_training: + sample = data_layer(name="sample", size=sample_dim) + +if is_generator_training or is_discriminator_training: + label = data_layer(name="label", size=1) + prob = discriminator(sample) + cost = cross_entropy(input=prob, label=label) + classification_error_evaluator( + input=prob, label=label, name=mode + '_error') + outputs(cost) + +if is_generator: + noise = data_layer(name="noise", size=noise_dim) + outputs(generator(noise)) +``` + +In order to train the networks defined in gan_conf.py, one first needs to initialize a Paddle environment, parse the config, create GradientMachine from the config and create trainer from GradientMachine as done in the code chunk below: +```python +import py_paddle.swig_paddle as api +# init paddle environment +api.initPaddle('--use_gpu=' + use_gpu, '--dot_period=10', + '--log_period=100', '--gpu_id=' + args.gpu_id, + '--save_dir=' + "./%s_params/" % data_source) + +# Parse config +gen_conf = parse_config(conf, "mode=generator_training,data=" + data_source) +dis_conf = parse_config(conf, "mode=discriminator_training,data=" + data_source) +generator_conf = parse_config(conf, "mode=generator,data=" + data_source) + +# Create GradientMachine +dis_training_machine = api.GradientMachine.createFromConfigProto( +dis_conf.model_config) +gen_training_machine = api.GradientMachine.createFromConfigProto( +gen_conf.model_config) +generator_machine = api.GradientMachine.createFromConfigProto( +generator_conf.model_config) + +# Create trainer +dis_trainer = api.Trainer.create(dis_conf, dis_training_machine) +gen_trainer = api.Trainer.create(gen_conf, gen_training_machine) +``` + +In order to balance the strength between generator and discriminator, we schedule to train whichever one is performing worse by comparing their loss function value. The loss function value can be calculated by a forward pass through the GradientMachine. +```python +def get_training_loss(training_machine, inputs): + outputs = api.Arguments.createArguments(0) + training_machine.forward(inputs, outputs, api.PASS_TEST) + loss = outputs.getSlotValue(0).copyToNumpyMat() + return numpy.mean(loss) +``` + +After training one network, one needs to sync the new parameters to the other networks. The code below demonstrates one example of such use case: +```python +# Train the gen_training +gen_trainer.trainOneDataBatch(batch_size, data_batch_gen) + +# Copy the parameters from gen_training to dis_training and generator +copy_shared_parameters(gen_training_machine, +dis_training_machine) +copy_shared_parameters(gen_training_machine, generator_machine) +``` + + +## A Toy Example +With the infrastructure explained above, we can now walk you through a toy example of generating two dimensional uniform distribution using 10 dimensional Gaussian noise. + +The Gaussian noises are generated using the code below: +```python +def get_noise(batch_size, noise_dim): + return numpy.random.normal(size=(batch_size, noise_dim)).astype('float32') +``` + +The real samples (2-D uniform) are generated using the code below: +```python +# synthesize 2-D uniform data in gan_trainer.py:114 +def load_uniform_data(): + data = numpy.random.rand(1000000, 2).astype('float32') + return data +``` + +The generator and discriminator network are built using fully-connected layer and batch_norm layer, and are defined in gan_conf.py. + +To train the GAN model, one can use the command below. The flag -d specifies the training data (cifar, mnist or uniform) and flag --useGpu specifies whether to use gpu for training (0 is cpu, 1 is gpu). +```bash +$python gan_trainer.py -d uniform --useGpu 1 +``` +The generated samples can be found in ./uniform_samples/ and one example is shown below as Figure 2. One can see that it roughly recovers the 2D uniform distribution. + ++ Figure 2. Uniform Sample +
+ +## MNIST Example +### Data preparation +To download the MNIST data, one can use the following commands: +```bash +$cd data/ +$./get_mnist_data.sh +``` + +### Model description +Following the DC-Gan paper (https://arxiv.org/abs/1511.06434), we use convolution/convolution-transpose layer in the discriminator/generator network to better deal with images. The details of the network structures are defined in gan_conf_image.py. + +### Training the model +To train the GAN model on mnist data, one can use the following command: +```bash +$python gan_trainer.py -d mnist --useGpu 1 +``` +The generated sample images can be found at ./mnist_samples/ and one example is shown below as Figure 3. ++ Figure 3. MNIST Sample +
diff --git a/develop/doc/_sources/tutorials/image_classification/index_en.md.txt b/develop/doc/_sources/tutorials/image_classification/index_en.md.txt new file mode 100644 index 0000000000000000000000000000000000000000..60c81a6a539944634773f38ec4c9a59709dd4afc --- /dev/null +++ b/develop/doc/_sources/tutorials/image_classification/index_en.md.txt @@ -0,0 +1,221 @@ +Image Classification Tutorial +============================== + +This tutorial will guide you through training a convolutional neural network to classify objects using the CIFAR-10 image classification dataset. +As shown in the following figure, the convolutional neural network can recognize the main object in images, and output the classification result. + +| ResNet | +Top-1 | +Model Size | +
|---|---|---|
| ResNet-50 | +24.9% | +99M | +
| ResNet-101 | +23.7% | +173M | +
| ResNet-152 | +23.2% | +234M | +
+ +## ResNet Model + +See ```demo/model_zoo/resnet/resnet.py```. This config contains network of 50, 101 and 152 layers. You can specify layer number by adding argument like ```--config_args=layer_num=50``` in command line arguments. + +### Network Visualization + +You can get a diagram of ResNet network by running the following commands. The script generates dot file and then converts dot file to PNG file, which needs to install graphviz to convert. + +``` +cd demo/model_zoo/resnet +./net_diagram.sh +``` + +### Model Download + +``` +cd demo/model_zoo/resnet +./get_model.sh +``` +You can run above command to download all models and mean file and save them in ```demo/model_zoo/resnet/model``` if downloading successfully. + +``` +mean_meta_224 resnet_101 resnet_152 resnet_50 +``` + * resnet_50: model of 50 layers. + * resnet_101: model of 101 layers. + * resnet_152: model of 152 layers. + * mean\_meta\_224: mean file with 3 x 224 x 224 size in **BGR** order. You also can use three mean values: 103.939, 116.779, 123.68. + +### Parameter Info + +* **Convolution Layer Weight** + + As batch normalization layer is connected after each convolution layer, there is no parameter of bias and only one weight in this layer. + shape: `(Co, ky, kx, Ci)` + * Co: channle number of output feature map. + * ky: filter size in vertical direction. + * kx: filter size in horizontal direction. + * Ci: channle number of input feature map. + + 2-Dim matrix: (Co * ky * kx, Ci), saved in row-major order. + +* **Fully connected Layer Weight** + + 2-Dim matrix: (input layer size, this layer size), saved in row-major order. + +* **[Batch Normalization](
| Parameter Name | +Number | +Meaning | +
|---|---|---|
| _res2_1_branch1_bn.w0 | +256 | +gamma, scale parameter | +
| _res2_1_branch1_bn.w1 | +256 | +mean value of feature map | +
| _res2_1_branch1_bn.w2 | +256 | +variance of feature map | +
| _res2_1_branch1_bn.wbias | +256 | +beta, shift parameter | +
+ +### Parameter Observation + +Users who want to observe the parameters can use Python to read: + +``` +import sys +import numpy as np + +def load(file_name): + with open(file_name, 'rb') as f: + f.read(16) # skip header for float type. + return np.fromfile(f, dtype=np.float32) + +if __name__=='__main__': + weight = load(sys.argv[1]) +``` + +or simply use following shell command: + +``` +od -j 16 -f _res2_1_branch1_bn.w0 +``` + +## Feature Extraction + +We provide both C++ and Python interfaces to extract features. The following examples use data in `demo/model_zoo/resnet/example` to show the extracting process in detail. + +### C++ Interface + +First, specify image data list in `define_py_data_sources2` in the config, see example `demo/model_zoo/resnet/resnet.py`. + +``` + train_list = 'train.list' if not is_test else None + # mean.meta is mean file of ImageNet dataset. + # mean.meta size : 3 x 224 x 224. + # If you use three mean value, set like: + # "mean_value:103.939,116.779,123.68;" + args={ + 'mean_meta': "model/mean_meta_224/mean.meta", + 'image_size': 224, 'crop_size': 224, + 'color': True,'swap_channel:': [2, 1, 0]} + define_py_data_sources2(train_list, + 'example/test.list', + module="example.image_list_provider", + obj="processData", + args=args) +``` + +Second, specify layers to extract features in `Outputs()` of `resnet.py`. For example, + +``` +Outputs("res5_3_branch2c_conv", "res5_3_branch2c_bn") +``` + +Third, specify model path and output directory in `extract_fea_c++.sh`, and then run the following commands. + +``` +cd demo/model_zoo/resnet +./extract_fea_c++.sh +``` + +If successful, features are saved in `fea_output/rank-00000` as follows. And you can use `load_feature_c` interface in `load_feature.py ` to load such a file. + +``` +-0.115318 -0.108358 ... -0.087884;-1.27664 ... -1.11516 -2.59123; +-0.126383 -0.116248 ... -0.00534909;-1.42593 ... -1.04501 -1.40769; +``` + +* Each line stores features of a sample. Here, the first line stores features of `example/dog.jpg` and second line stores features of `example/cat.jpg`. +* Features of different layers are splitted by `;`, and their order is consistent with the layer order in `Outputs()`. Here, the left features are `res5_3_branch2c_conv` layer and right features are `res5_3_branch2c_bn` layer. + +### Python Interface + +`demo/model_zoo/resnet/classify.py` is an example to show how to use Python to extract features. Following example still uses data of `./example/test.list`. Command is as follows: + +``` +cd demo/model_zoo/resnet +./extract_fea_py.sh +``` + +extract_fea_py.sh: + +``` +python classify.py \ + --job=extract \ + --conf=resnet.py\ + --use_gpu=1 \ + --mean=model/mean_meta_224/mean.meta \ + --model=model/resnet_50 \ + --data=./example/test.list \ + --output_layer="res5_3_branch2c_conv,res5_3_branch2c_bn" \ + --output_dir=features + +``` +* \--job=extract: specify job mode to extract feature. +* \--conf=resnet.py: network configure. +* \--use_gpu=1: speficy GPU mode. +* \--model=model/resnet_5: model path. +* \--data=./example/test.list: data list. +* \--output_layer="xxx,xxx": specify layers to extract features. +* \--output_dir=features: output diretcoty. + +If run successfully, you will see features saved in `features/batch_0`, this file is produced with cPickle. You can use `load_feature_py` interface in `load_feature.py` to open the file, and it returns a dictionary as follows: + +``` +{ +'cat.jpg': {'res5_3_branch2c_conv': array([[-0.12638293, -0.116248 , -0.11883899, ..., -0.00895038, 0.01994277, -0.00534909]], dtype=float32), 'res5_3_branch2c_bn': array([[-1.42593431, -1.28918779, -1.32414699, ..., -1.45933616, -1.04501402, -1.40769434]], dtype=float32)}, +'dog.jpg': {'res5_3_branch2c_conv': array([[-0.11531784, -0.10835785, -0.08809858, ...,0.0055237, 0.01505112, -0.08788397]], dtype=float32), 'res5_3_branch2c_bn': array([[-1.27663755, -1.18272924, -0.90937918, ..., -1.25178063, -1.11515927, -2.59122872]], dtype=float32)} +} +``` + +Observed carefully, these feature values are consistent with the above results extracted by C++ interface. + +## Prediction + +`classify.py` also can be used to predict. We provide an example script `predict.sh` to predict data in `example/test.list` using a ResNet model with 50 layers. + +``` +cd demo/model_zoo/resnet +./predict.sh +``` + +predict.sh calls the `classify.py`: + +``` +python classify.py \ + --job=predict \ + --conf=resnet.py\ + --multi_crop \ + --model=model/resnet_50 \ + --use_gpu=1 \ + --data=./example/test.list +``` +* \--job=extract: speficy job mode to predict. +* \--conf=resnet.py: network configure. +* \--multi_crop: use 10 crops and average predicting probability. +* \--use_gpu=1: speficy GPU mode. +* \--model=model/resnet_50: model path. +* \--data=./example/test.list: data list. + +If run successfully, you will see following results, where 156 and 285 are labels of the images. + +``` +Label of example/dog.jpg is: 156 +Label of example/cat.jpg is: 282 +``` diff --git a/develop/doc/_sources/tutorials/index_en.md.txt b/develop/doc/_sources/tutorials/index_en.md.txt new file mode 100644 index 0000000000000000000000000000000000000000..77331a703b6f0fdf92921ebcc476325b7327e976 --- /dev/null +++ b/develop/doc/_sources/tutorials/index_en.md.txt @@ -0,0 +1,14 @@ +# TUTORIALS +There are several examples and demos here. + +* [Quick Start](quick_start/index_en.md) +* [MovieLens Regression](rec/ml_regression_en.rst) +* [Image Classification](image_classification/index_en.md) +* [Sentiment Analysis](sentiment_analysis/index_en.md) +* [Semantic Role Labeling](semantic_role_labeling/index_en.md) +* [Text Generation](text_generation/index_en.md) +* [Image Auto-Generation](gan/index_en.md) + +## Model Zoo +* [ImageNet: ResNet](imagenet_model/resnet_model_en.md) +* [Embedding: Chinese Word](embedding_model/index_en.md) diff --git a/develop/doc/_sources/tutorials/quick_start/index_en.md.txt b/develop/doc/_sources/tutorials/quick_start/index_en.md.txt new file mode 100644 index 0000000000000000000000000000000000000000..ca110431cf921ae0480d3fb2b17c58f90a84cc0e --- /dev/null +++ b/develop/doc/_sources/tutorials/quick_start/index_en.md.txt @@ -0,0 +1,562 @@ +# Quick Start + +This tutorial will teach the basics of deep learning (DL), including how to implement many different models in PaddlePaddle. You will learn how to: + - Prepare data into the standardized format that PaddlePaddle accepts. + - Write data providers that read data into PaddlePaddle. + - Configure neural networks in PaddlePaddle layer by layer. + - Train models. + - Perform inference with trained models. + + +## Install + +To get started, please install PaddlePaddle on your computer. Throughout this tutorial, you will learn by implementing different DL models for text classification. + +To install PaddlePaddle, please follow the instructions here: Build and Install. + +## Overview +For the first step, you will use PaddlePaddle to build a **text classification** system. For example, suppose you run an e-commence website, and you want to analyze the sentiment of user reviews to evaluate product quality. + +For example, given the input + +``` +This monitor is fantastic. +``` + +Your classifier should output “positive”, since this text snippet shows that the user is satisfied with the product. Given this input: + +``` +The monitor breaks down two months after purchase. +``` + +the classifier should output “negative“. + +To build your text classification system, your code will need to perform five steps: +
| Network name | +Number of parameters | +Test error | + + + + +
|---|---|---|
| Logistic regression | +252 KB | +8.652% | +
+ +### Word Embedding Model +In order to use the word embedding model, you need to change the data provider a little bit to make the input words as a sequence of word IDs. The revised data provider `dataprovider_emb.py` is listed below. You only need to change initializer() for the type of the first input. It is changed from sparse_binary_vector to sequence of intergers. process() remains the same. This data provider can also be used for later sequence models. + +```python +def initializer(settings, dictionary, **kwargs): + # Put the word dictionary into settings + settings.word_dict = dictionary + settings.input_types = [ + # Define the type of the first input as a sequence of integers. + integer_value_sequence(len(dictionary)), + # Define the second input for label id + integer_value(2)] + +@provider(init_hook=initializer) +def process(settings, file_name): + ... + # omitted, it is same as the data provider for LR model +``` + +This model is very similar to the framework of logistic regression, but it uses word embedding vectors instead of a sparse vectors to represent words. +
| Network name | +Number of parameters | +Test error | + + + + +
|---|---|---|
| Word embedding model | +15 MB | +8.484% | +
+ +### Convolutional Neural Network Model +Convolutional neural network converts a sequence of word embeddings into a sentence representation using temporal convolutions. You will transform the fully connected layer of the word embedding model to 3 new sub-steps. +
| Network name | +Number of parameters | +Test error | + + + + +
|---|---|---|
| Convolutional model | +16 MB | +5.628% | +
+ +### Recurrent Model +
| Network name | +Number of parameters | +Test error | + + + + +
|---|---|---|
| Recurrent model | +16 MB | +4.812% | +
+ +## Optimization Algorithm +Optimization algorithms include Momentum, RMSProp, AdaDelta, AdaGrad, Adam, and Adamax. You can use Adam optimization method here, with L2 regularization and gradient clipping, because Adam has been proved to work very well for training recurrent neural network. + +```python +settings(batch_size=128, + learning_rate=2e-3, + learning_method=AdamOptimizer(), + regularization=L2Regularization(8e-4), + gradient_clipping_threshold=25) +``` + +## Training Model +After completing data preparation and network architecture specification, you will run the training script. +
| Network name | +Number of parameters | +Error rate | +Configuration file name | + + + + +
|---|---|---|---|
| Logistic regression model(BOW) | +252KB | +8.652% | +trainer_config.lr.py | +
| Word embedding | +15MB | +8.484% | +trainer_config.emb.py | +
| Convolution model | +16MB | +5.628% | +trainer_config.cnn.py | +
| Time sequence model | +16MB | +4.812% | +trainer_config.lstm.py | +
+ +## Appendix +### Command Line Argument + +* \--config：network architecture path. +* \--save_dir：model save directory. +* \--log_period：the logging period per batch. +* \--num_passes：number of training passes. One pass means the training would go over the whole training dataset once. +* \--config_args：Other configuration arguments. +* \--init_model_path：The path of the initial model parameter. + +By default, the trainer will save model every pass. You can also specify `saving_period_by_batches` to set the frequency of batch saving. You can use `show_parameter_stats_period` to print the statistics of the parameters, which are very useful for tuning parameters. Other command line arguments can be found in command line argument documentation。 + +### Log + +``` +TrainerInternal.cpp:160] Batch=20 samples=2560 AvgCost=0.628761 CurrentCost=0.628761 Eval: classification_error_evaluator=0.304297 CurrentEval: classification_error_evaluator=0.304297 +``` +During model training, you will see the log like the examples above: +
| Name | +Explanation | + + + +
|---|---|
| Batch=20 | +You have trained 20 batches. | +
| samples=2560 | +You have trained 2560 examples. | +
| AvgCost | +The average cost from the first batch to the current batch. | +
| CurrentCost | +the average cost of the last log_period batches | +
| Eval: classification_error_evaluator | +The average classification error from the first batch to the current batch. | +
| CurrentEval: classification_error_evaluator | +The average error rate of the last log_period batches | +
diff --git a/develop/doc/_sources/tutorials/rec/ml_dataset_en.md.txt b/develop/doc/_sources/tutorials/rec/ml_dataset_en.md.txt new file mode 100644 index 0000000000000000000000000000000000000000..25dea5c4afbf1ce1c1ac6195cbd245b116459e2e --- /dev/null +++ b/develop/doc/_sources/tutorials/rec/ml_dataset_en.md.txt @@ -0,0 +1,111 @@ +```eval_rst +.. _demo_ml_dataset: +``` + +# MovieLens Dataset + +The [MovieLens Dataset](http://grouplens.org/datasets/movielens/) was collected by GroupLens Research. +The data set contains some user information, movie information, and many movie ratings from \[1-5\]. +The data sets have many version depending on the size of set. +We use [MovieLens 1M Dataset](http://files.grouplens.org/datasets/movielens/ml-1m.zip) as a demo dataset, which contains +1 million ratings from 6000 users on 4000 movies. Released 2/2003. + +## Dataset Features + +In [ml-1m Dataset](http://files.grouplens.org/datasets/movielens/ml-1m.zip), there are many features in these dataset. +The data files (which have ".dat" extension) in [ml-1m Dataset](http://files.grouplens.org/datasets/movielens/ml-1m.zip) +is basically CSV file that delimiter is "::". The description in README we quote here. + +### RATINGS FILE DESCRIPTION(ratings.dat) + + +All ratings are contained in the file "ratings.dat" and are in the +following format: + +UserID::MovieID::Rating::Timestamp + +- UserIDs range between 1 and 6040 +- MovieIDs range between 1 and 3952 +- Ratings are made on a 5-star scale (whole-star ratings only) +- Timestamp is represented in seconds since the epoch as returned by time(2) +- Each user has at least 20 ratings + +### USERS FILE DESCRIPTION(users.dat) + +User information is in the file "users.dat" and is in the following +format: + +UserID::Gender::Age::Occupation::Zip-code + +All demographic information is provided voluntarily by the users and is +not checked for accuracy. Only users who have provided some demographic +information are included in this data set. + +- Gender is denoted by a "M" for male and "F" for female +- Age is chosen from the following ranges: + + * 1: "Under 18" + * 18: "18-24" + * 25: "25-34" + * 35: "35-44" + * 45: "45-49" + * 50: "50-55" + * 56: "56+" + +- Occupation is chosen from the following choices: + + * 0: "other" or not specified + * 1: "academic/educator" + * 2: "artist" + * 3: "clerical/admin" + * 4: "college/grad student" + * 5: "customer service" + * 6: "doctor/health care" + * 7: "executive/managerial" + * 8: "farmer" + * 9: "homemaker" + * 10: "K-12 student" + * 11: "lawyer" + * 12: "programmer" + * 13: "retired" + * 14: "sales/marketing" + * 15: "scientist" + * 16: "self-employed" + * 17: "technician/engineer" + * 18: "tradesman/craftsman" + * 19: "unemployed" + * 20: "writer" + +### MOVIES FILE DESCRIPTION(movies.dat) + +Movie information is in the file "movies.dat" and is in the following +format: + +MovieID::Title::Genres + +- Titles are identical to titles provided by the IMDB (including +year of release) +- Genres are pipe-separated and are selected from the following genres: + + * Action + * Adventure + * Animation + * Children's + * Comedy + * Crime + * Documentary + * Drama + * Fantasy + * Film-Noir + * Horror + * Musical + * Mystery + * Romance + * Sci-Fi + * Thriller + * War + * Western + +- Some MovieIDs do not correspond to a movie due to accidental duplicate +entries and/or test entries +- Movies are mostly entered by hand, so errors and inconsistencies may exist diff --git a/develop/doc/_sources/tutorials/rec/ml_regression_en.rst.txt b/develop/doc/_sources/tutorials/rec/ml_regression_en.rst.txt new file mode 100644 index 0000000000000000000000000000000000000000..993b9a516f134ff8b59e8755b721f76c8f32f0fd --- /dev/null +++ b/develop/doc/_sources/tutorials/rec/ml_regression_en.rst.txt @@ -0,0 +1,348 @@ +Regression MovieLens Ratting +============================ + +Here we demonstrate a **Cosine Similarity Regression** job in movie lens dataset. +This demo will show how paddle does (word) embedding job, +handles the similarity regression, +the character-level convolutional networks for text, and how does paddle handle +multiple types of inputs. +Note that the model structure is not fine-tuned and just a demo to show how paddle works. + + +YOU ARE WELCOME TO BUILD A BETTER DEMO +BY USING PADDLEPADDLE, AND LET US KNOW TO MAKE THIS DEMO BETTER. + +Data Preparation +```````````````` +Download and extract dataset +'''''''''''''''''''''''''''' +We use :ref:`demo_ml_dataset` here. +To download and unzip the dataset, simply run the following commands. + +.. code-block:: bash + + cd demo/recommendation/data + ./ml_data.sh + +And the directory structure of :code:`demo/recommendation/data/ml-1m` is: + +.. code-block:: text + + +--ml-1m + +--- movies.dat # movie features + +--- ratings.dat # ratings + +--- users.dat # user features + +--- README # dataset description + +Field config file +''''''''''''''''' +**Field config file** is used to specify the fields of the dataset and the file format, +i.e, specific **WHAT** type it is in each feature file. + +The field config file of ml-1m shows in :code:`demo/recommendation/data/config.json`. +It specifics the field types and file names: 1) there are four types of field for user file\: id, gender, age and occupation; +2) the filename is "users.dat", and the delimiter of file is "::". + +.. include:: ../../../demo/recommendation/data/config.json + :code: json + :literal: + +Preprocess Data +``````````````` +You need to install python 3rd party libraries. +IT IS HIGHLY RECOMMEND TO USE VIRTUALENV MAKE A CLEAN PYTHON ENVIRONMENT. + +.. code-block:: bash + + pip install -r requirements.txt + +The general command for preprocessing the dataset is: + +.. code-block:: bash + + cd demo/recommendation + ./preprocess.sh + +And the detail steps are introduced as follows. + +Extract Movie/User features to python object +''''''''''''''''''''''''''''''''''''''''''''' + +There are many features in movie or user in movielens 1m dataset. +Each line of rating file just provides a Movie/User id to refer each movie or user. +We process the movie/user feature file first, and pickle the feature (**Meta**) object as a file. + +Meta config file +................ + +**Meta config file** is used to specific **HOW** to parse each field in dataset. +It could be translated from field config file, or written by hand. +Its file format could be either json or yaml syntax file. Parser will automatically choose the file format by extension name. + +To convert Field config file to meta config file, just run: + +.. code-block:: bash + + cd demo/recommendation/data + python config_generator.py config.json > meta_config.json + +The meta config file shows below: + +.. include:: ../../../demo/recommendation/data/meta_config.json + :code: json + :literal: + +There are two kinds of features in meta\: movie and user. + +* in movie file, whose name is movies.dat + * we just split each line by "::" + * pos 0 is id. + * pos 1 feature: + * name is title. + * it uses regex to parse this feature. + * it is a char based word embedding feature. + * it is a sequence. + * pos 2 feature: + * name is genres. + * type is one hot dense vector. + * dictionary is auto generated by parsing, each key is split by '|' +* in user file, whose name is users.dat + * we just split each line by "::" + * pos 0 is id. + * pos 1 feature: + * name is gender + * just simple char based embedding. + * pos 2 feature: + * name is age + * just whole word embedding. + * embedding id will be sort by word. + * pos 3 feature: + * name is occupation. + * just simple whole word embedding. + + +Meta file +''''''''' + +After having meta config file, we can generate **Meta file**, a python pickle object which stores movie/user information. +The following commands could be run to generate it. + +.. code-block:: bash + + python meta_generator.py ml-1m meta.bin --config=meta_config.json + +And the structure of the meta file :code:`meta.bin` is: + +.. code-block:: text + + +--+ movie + | +--+ __meta__ + | | +--+ raw_meta # each feature meta config. list + | | | + + | | | | # ID Field, we use id as key + | | | +--+ {'count': 3883, 'max': 3952, 'is_key': True, 'type': 'id', 'min': 1} + | | | | + | | | | # Titile field, the dictionary list of embedding. + | | | +--+ {'dict': [ ... ], 'type': 'embedding', 'name': 'title', 'seq': 'sequence'} + | | | | + | | | | # Genres field, the genres dictionary + | | | +--+ {'dict': [ ... ], 'type': 'one_hot_dense', 'name': 'genres'} + | | | + | | +--+ feature_map [1, 2] # a list for raw_meta index for feature field. + | | # it means there are 2 features for each key. + | | # * 0 offset of feature is raw_meta[1], Title. + | | # * 1 offset of feature is raw_meta[2], Genres. + | | + | +--+ 1 # movie 1 features + | | + + | | +---+ [[...], [...]] # title ids, genres dense vector + | | + | +--+ 2 + | | + | +--+ ... + | + +--- user + +--+ __meta__ + | + + | +--+ raw_meta + | | + + | | +--+ id field as user + | | | + | | +--+ {'dict': ['F', 'M'], 'type': 'embedding', 'name': 'gender', 'seq': 'no_sequence'} + | | | + | | +--+ {'dict': ['1', '18', '25', '35', '45', '50', '56'], 'type': 'embedding', 'name': 'age', 'seq': 'no_sequence'} + | | | + | | +--+ {'dict': [...], 'type': 'embedding', 'name': 'occupation', 'seq': 'no_sequence'} + | | + | +--+ feature_map [1, 2, 3] + | + +--+ 1 # user 1 features + | + +--+ 2 + +--+ ... + + +Split Training/Testing files +'''''''''''''''''''''''''''' + +We split :code:`ml-1m/ratings.dat` into a training and testing file. The way to split file is for each user, we split the +rating by two parts. So each user in testing file will have some rating information in training file. + +Use :code:`separate.py` to separate the training and testing file. + +.. code-block:: bash + + python split.py ml-1m/ratings.dat --delimiter="::" --test_ratio=0.1 + +Then two files will be generated\: :code:`ml-1m/ratings.dat.train` and :code:`ml-1m/rating.data.test`. +Move them to workspace :code:`data`, shuffle the train file, and prepare the file list for paddle train. + +.. code-block:: bash + + shuf ml-1m/ratings.dat.train > ratings.dat.train + cp ml-1m/ratings.dat.test . + echo "./data/ratings.dat.train" > train.list + echo "./data/ratings.dat.test" > test.list + + +Neural Network Configuration +```````````````````````````` + +Trainer Config File +''''''''''''''''''' + +The network structure shows below. + +.. image:: rec_regression_network.png + :align: center + :alt: rec_regression_network + +The demo's neural network config file :code:`trainer_config.py` show as below. + +.. literalinclude:: ../../../demo/recommendation/trainer_config.py + :language: python + :lines: 15- + +In this :code:`trainer_config.py`, we just map each feature type to +a feature vector, following shows how to map each feature to a vector shows below. + +* :code:`id`\: Just simple embedding, and then add to fully connected layer. +* :code:`embedding`\: + - if is_sequence, get the embedding and do a text convolutional operation, + get the average pooling result. + - if not sequence, get the embedding and add to fully connected layer. +* :code:`one_host_dense`\: + - just two fully connected layer. + +Then we combine each features of movie into one movie feature by a +:code:`fc_layer` with multiple inputs, and do the same thing to user features, +get one user feature. Then we calculate the cosine similarity of these two +features. + +In these networks, we use several APIs in :ref:`api_trainer_config` . There are + +* Data Layer, :ref:`api_trainer_config_helpers_layers_data_layer` +* Fully Connected Layer, :ref:`api_trainer_config_helpers_layers_fc_layer` +* Embedding Layer, :ref:`api_trainer_config_helpers_layers_embedding_layer` +* Context Projection Layer, :ref:`api_trainer_config_helpers_layers_context_projection` +* Pooling Layer, :ref:`api_trainer_config_helpers_layers_pooling_layer` +* Cosine Similarity Layer, :ref:`api_trainer_config_helpers_layers_cos_sim` +* Text Convolution Pooling Layer, :ref:`api_trainer_config_helpers_network_text_conv_pool` +* Declare Python Data Sources :ref:`api_trainer_config_helpers_data_sources`. + +Data Provider +''''''''''''' + +.. literalinclude:: ../../../demo/recommendation/dataprovider.py + :language: python + :lines: 15- + +The data provider just read the meta.bin and rating file, yield each sample for training. +In this :code:`dataprovider.py`, we should set\: + +* obj.slots\: The feature types and dimension. +* use_seq\: Whether this :code:`dataprovider.py` in sequence mode or not. +* process\: Return each sample of data to :code:`paddle`. + +The data provider details document see :ref:`api_pydataprovider2`. + +Train +````` + +After prepare data, config network, writting data provider, now we can run paddle training. + +The :code:`run.sh` is shown as follow: + +.. literalinclude:: ../../../demo/recommendation/run.sh + :language: bash + :lines: 16- + +It just start a paddle training process, write the log to :code:`log.txt`, +then print it on screen. + +Each command line argument in :code:`run.sh`, please refer to the :ref:`cmd_line_index` page. The short description of these arguments is shown as follow. + +* config\: Tell paddle which file is neural network configuration. +* save_dir\: Tell paddle save model into :code:`./output`. +* use_gpu\: Use gpu or not. Default is false. +* trainer_count\: The compute thread in one machine. +* test_all_data_in_one_period\: Test All Data during one test period. Otherwise, + will test a :code:`batch_size` data in one test period. +* log_period\: Print log after train :code:`log_period` batches. +* dot_period\: Print a :code:`.` after train :code:`dot_period` batches. +* num_passes\: Train at most :code:`num_passes`. + +If training process starts successfully, the output likes follow: + +.. code-block:: text + + I0601 08:07:22.832059 10549 TrainerInternal.cpp:157] Batch=100 samples=160000 AvgCost=4.13494 CurrentCost=4.13494 Eval: CurrentEval: + + I0601 08:07:50.672627 10549 TrainerInternal.cpp:157] Batch=200 samples=320000 AvgCost=3.80957 CurrentCost=3.48421 Eval: CurrentEval: + + I0601 08:08:18.877369 10549 TrainerInternal.cpp:157] Batch=300 samples=480000 AvgCost=3.68145 CurrentCost=3.42519 Eval: CurrentEval: + + I0601 08:08:46.863963 10549 TrainerInternal.cpp:157] Batch=400 samples=640000 AvgCost=3.6007 CurrentCost=3.35847 Eval: CurrentEval: + + I0601 08:09:15.413025 10549 TrainerInternal.cpp:157] Batch=500 samples=800000 AvgCost=3.54811 CurrentCost=3.33773 Eval: CurrentEval: + I0601 08:09:36.058670 10549 TrainerInternal.cpp:181] Pass=0 Batch=565 samples=902826 AvgCost=3.52368 Eval: + I0601 08:09:46.215489 10549 Tester.cpp:101] Test samples=97383 cost=3.32155 Eval: + I0601 08:09:46.215966 10549 GradientMachine.cpp:132] Saving parameters to ./output/model/pass-00000 + I0601 08:09:46.233397 10549 ParamUtil.cpp:99] save dir ./output/model/pass-00000 + I0601 08:09:46.233438 10549 Util.cpp:209] copy trainer_config.py to ./output/model/pass-00000 + I0601 08:09:46.233541 10549 ParamUtil.cpp:147] fileName trainer_config.py + +The model is saved in :code:`output/` directory. You can use :code:`Ctrl-C` to stop training whenever you want. + +Evaluate and Predict +```````````````````` + +After training several passes, you can evaluate them and get the best pass. Just run + +.. code-block:: bash + + ./evaluate.sh + +You will see messages like this: + +.. code-block:: text + + Best pass is 00009, error is 3.06949, which means predict get error as 0.875998002281 + evaluating from pass output/pass-00009 + +Then, you can predict what any user will rate a movie. Just run + +.. code-block:: bash + + python prediction.py 'output/pass-00009/' + +Predictor will read user input, and predict scores. It has a command-line user interface as follows: + +.. code-block:: text + + Input movie_id: 9 + Input user_id: 4 + Prediction Score is 2.56 + Input movie_id: 8 + Input user_id: 2 + Prediction Score is 3.13 diff --git a/develop/doc/_sources/tutorials/semantic_role_labeling/index_en.md.txt b/develop/doc/_sources/tutorials/semantic_role_labeling/index_en.md.txt new file mode 100644 index 0000000000000000000000000000000000000000..92d7c634832119c718711a57c16f69492d405f28 --- /dev/null +++ b/develop/doc/_sources/tutorials/semantic_role_labeling/index_en.md.txt @@ -0,0 +1,204 @@ +```eval_rst +.. _semantic_role_labeling: +``` + +# Semantic Role labeling Tutorial # + +Semantic role labeling (SRL) is a form of shallow semantic parsing whose goal is to discover the predicate-argument structure of each predicate in a given input sentence. SRL is useful as an intermediate step in a wide range of natural language processing tasks, such as information extraction. automatic document categorization and question answering. An instance is as following [1]: + + [ A0 He ] [ AM-MOD would ][ AM-NEG n’t ] [ V accept] [ A1 anything of value ] from [A2 those he was writing about ]. + +- V: verb +- A0: acceptor +- A1: thing accepted +- A2: accepted-from +- A3: Attribute +- AM-MOD: modal +- AM-NEG: negation + +Given the verb "accept", the chunks in sentence would play certain semantic roles. Here, the label scheme is from Penn Proposition Bank. + +To this date, most of the successful SRL systems are built on top of some form of parsing results where pre-defined feature templates over the syntactic structure are used. This tutorial will present an end-to-end system using deep bidirectional long short-term memory (DB-LSTM)[2] for solving the SRL task, which largely outperforms the previous state-of-the-art systems. The system regards SRL task as the sequence labelling problem. + +## Data Description +The relevant paper[2] takes the data set in CoNLL-2005&2012 Shared Task for training and testing. Accordingto data license, the demo adopts the test data set of CoNLL-2005, which can be reached on website. + +To download and process the original data, user just need to execute the following command: + +```bash +cd data +./get_data.sh +``` +Several new files appear in the `data `directory as follows. +```bash +conll05st-release：the test data set of CoNll-2005 shared task +test.wsj.words：the Wall Street Journal data sentences +test.wsj.props: the propositional arguments +feature: the extracted features from data set +``` + +## Training +### DB-LSTM +Please refer to the Sentiment Analysis demo to learn more about the long short-term memory unit. + +Unlike Bidirectional-LSTM that used in Sentiment Analysis demo, the DB-LSTM adopts another way to stack LSTM layer. First a standard LSTM processes the sequence in forward direction. The input and output of this LSTM layer are taken by the next LSTM layer as input, processed in reversed direction. These two standard LSTM layers compose a pair of LSTM. Then we stack LSTM layers pair after pair to obtain the deep LSTM model. + +The following figure shows a temporal expanded 2-layer DB-LSTM network. +
+[2] Johan Bollen, Huina Mao, Xiaojun Zeng. 2011. [Twitter mood predicts the stock market](http://arxiv.org/abs/1010.3003), Journal of Computational Science.
+[3] Alex Graves, Marcus Liwicki, Santiago Fernan- dez, Roman Bertolami, Horst Bunke, and Ju ̈rgen Schmidhuber. 2009. [A novel connectionist system for unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine In- telligence](http://www.cs.toronto.edu/~graves/tpami_2009.pdf), 31(5):855–868.
+[4] Zachary C. Lipton, [A Critical Review of Recurrent Neural Networks for Sequence Learning](http://arxiv.org/abs/1506.00019v1), arXiv:1506.00019.
+[5] Jie Zhou and Wei Xu; [End-to-end Learning of Semantic Role Labeling Using Recurrent Neural Networks](http://www.aclweb.org/anthology/P/P15/P15-1109.pdf); ACL-IJCNLP 2015.
diff --git a/develop/doc/_sources/tutorials/text_generation/index_en.md.txt b/develop/doc/_sources/tutorials/text_generation/index_en.md.txt new file mode 100644 index 0000000000000000000000000000000000000000..5d8e667c20bd1fda64a6e11a88517d52112b72fa --- /dev/null +++ b/develop/doc/_sources/tutorials/text_generation/index_en.md.txt @@ -0,0 +1,338 @@ +# Text generation Tutorial # + +Sequence to sequence has been proven to be a powerful model for language generation. It can be used for machine translation, query rewriting, image captioning, etc. + +This tutorial guides you through training a sequence to sequence model for neural machine translation (NMT) network that translates French to English. + +We follow the paper [Neural Machine Translation by Jointly Learning to Align and Translate](http://arxiv.org/abs/1409.0473) , which details the model architecture and training procedure for good performance on WMT-14 dataset. This tutorial reproduces this result in PaddlePaddle. + +We thank @caoying for the pull request that defines the model architecture and solver configurations. + +## Data Preparation ## +### Download and Extract ### +Download the WMT-14 dataset from [http://www-lium.univ-lemans.fr/~schwenk/cslm\_joint\_paper/](http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/), extract it, and divide Develop and Test data into separate folder. + +- **Train data**: [bitexts (after selection)](http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/bitexts.tgz) +- **Develop and Test data**: [dev+test data](http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/dev+test.tgz) + +To do this, simply run the following commands in linux, otherwise, you need to download, extract, divide, and rename the file suffix respectively. + +```bash +cd demo/seqToseq/data +./wmt14_data.sh +``` + +We should find that the dataset `wmt14` has three folders as shown in the following table. +
| folder name | +French-English parallel corpora file | +number of total file | +size | +
|---|---|---|---|
| train_data | +ccb2_pc30.src, ccb2_pc30.trg, etc | +twelve | +3.55G | +
| test_data | +ntst1213.src, ntst1213.trg | +two | +1636k | +
| gen_data | +ntst14.src, ntst14.trg | +two | +864k | +
+ +- Each folder has French-English parallel corpora +- **XXX.src** are source French files; **XXX.trg** are target English files. +- The number of lines of **XXX.src** and **XXX.trg** should be the same. +- Each line is a French/English sentence. +- There is a one-to-one correspondence between the sentence at the i-th line of **XXX.src** and **XXX.trg**. + +### User Defined Dataset ### + +If you need to do other sequence-to-sequence tasks, such as Paraphrasing, you only need to organize the data as follows, and place them in `demo/seqToseq/data`: + + dataset + train + file1.src file1.trg + file2.src file2.trg + ...... + test + file1.src file1.trg + file2.src file2.trg + ...... + gen + file1.src file1.trg + file2.src file2.trg + ...... +- 1st directory: dataset folder name +- 2nd directory: folder of train, test, and gen. The names of these three folders are fixed. +- 3rd file: Source-Target parallel corpora files. + - **XXX.src** are source files, **XXX.trg** are target files. + - Each line of the file must be a sequence. + - There should be a one-to-one correspondence between the i-th sequence of **XXX.src** and **XXX.trg**. + +## Data Preprocess ## +### Preprocessing Workflow ### +- Concat each Source-Target parallel corpora to be one file: + - concat each **XXX.src** and **XXX.trg** to be **XXX**. + - the i-th line of **XXX** = the i-th line of **XXX.src** + '\t' + the i-th line of **XXX.trg** +- Build source and target dictionary of train data, each dictionary has DICTSIZE words: + - the most frequent (DICTSIZE-3) words + - 3 special token: + - `