# PaddlePaddle Design Doc

## Ingredients

As our design principle is starting from the essence: how could we allow users to express and solve their problems at neural networks. Some essential concepts that our API have to provide include:

1. A topology is an expression of layers.
2. A layer could be any kind of computation, including cost.
3. Some layers have parameters, some don’t. Most costs don’t have parameters.
4. In some topologies, layers share parameters. For example, the network for training a ranking model.
5. At programming time, users specify topologies and possible sharing of parameters. PaddlePaddle can figure out and create parameters required (and possibly shared) by one or more topologies.

## Starting from Examples

As a summarization of our disucssion, let us present two examples here:

### Example 1. Sharing Parameters between Layers

We use the 3-branch ranking model in this example. For your convenience, I copy-a-paste the model’s topology as follows:

    A -> f -\
    Q -> f --> cost
    B -> f -/

The following program trains the topology including the cost, and then use the sub-network in the trained topology in inference:

    def f(in):
        e = paddle.layer.embedding(in, parameter_name="embedding")
        o = paddle.layer.softmax(e, parameter_name="semantic")
        return o

    # Create 3 topologies (subnets), they share parameters because all
    # correspoinding layers have the same parameter names.
    fA = f(paddle.layer.data(input_name="A"))
    fB = f(paddle.layer.data(input_name="B"))
    fQ = f(paddle.layer.data(input_name="Q"))

    topology = paddle.layer.less_than(
                   paddle.layer.cross_entropy(fA, fQ),
                   paddle.layer.corss_entropy(fB, fQ))

    # Derive parameters required in topology and create them in model.
    parameters = paddle.parameters.create(topology)

    # Estimate parameters used in topology from data.
    paddle.train(topology, parameters, reader=read_ranking_model_data)

    # Inference using fA (or fB or fC, as they share their parameters).
    [testA, testB, testQ] = read_ranking_model_data()
    print "The sematic-vector of testA: ", paddle.infer(fA, parameters, testA)

### Example 2. Sharing Parameters between “Models”

We use GAN in this example. In the following example program, `d0` and `d1` correspond to the two networks in the following figure:

    def G(in):
        # over-simplified example as G has only one layers:
        return paddle.layer.fc(in, parameter_name="G")

    def D(in);
        # again, over-simplified:
        return paddle.layer.fc(in, parameter_name="D")

    # Construct the first topology, which contains both D and G.
    # By learning this topology, we update parameters of G.
    d0 = paddle.layer.should_be_false(D(G(paddle.layer.data())))

    # Construct a second topology d1, which contains only D. By
    # training this topology, we update parameters of D.  Note
    # that d1 share parameters with d0.
    d1 = paddle.layer.should_be_true(D(paddle.layer.data()))

    # Create parameters from a list of multiple topologies (models) for
    # the chance to share parameters between these topologies.
    parameters = paddle.parameters.create([d0, d1])

    # Iterative training of GAN.
    for ...:
        train(d0, parameters, reader=read_from_rng, immutable_parameters={"D"})
        train(d1, parameters, reader=read_from_realistic_images)

    # Use d1 for inference:
    print "D thinks a batch of images are realistic ", infer(d1, parameters, read_mnist_images)

### Summarization

Above two programs reveal some important design concerns:

1. Users describe a topology as an expression of layers. Every layer has a parameter name. If the users don’t specify it explicitly, it’s automatically generated as a unique name. By specifying the parameter name, users can specify the sharing of parameters between layers and even between topologies.
2. `paddle.parameters.create` figures out parameters required by one or more topologies from parameter names of layers. It creates these parameters and returns a `ParameterSet` object, which is in essence a map from parameter names to parameters.
3. At training and inference time, `paddle.train` and `paddle.infer` requires both a topology and the parameter set that holds the parameters of that topology. There are some reasons:
   1. This prevents users from forgetting to call `paddle.parameters.create`.
   2. `paddle.train` needs to know which parameter set to update.
   3. Users could load another (pre-trained) parameter set and use it with a topology in `train.infer`.
4. By specifying the `immutable_parameters` parameter of `paddle.train`, we can forbid the update of these parameters.

## Reader

Not all programming frameworks allow users to define I/O functions. An example is Google MapReduce, which can only read from text, SSTable, and RecordIO files. Hadoop MapReduce allows users to define readers and writers by deriving from base classes `Reader` and `Writer`. The former is less flexible but also less error-prone. We decide to provide the flexibility to users to define their readers.

There are some open questions here:

1. Should a reader return a Python dictionary?
2. How to map multiple outputs from a reader to multiple data layers?
3. How to easily compose some existing readers to read more data and feed a topology with more data layers?

## Training

The recommended way to training a model is to call `paddle.train`, which simply calls `paddle.trainer.Default`, a global variable of type `paddle.trainer.SGD`. Equivalently, we can do

    opt = paddle.trainer.SGD(..., paddle.updater.Adam(...))
    opt.train(topology, parameters, reader=read, ...)

### Updater

Please be aware that a trainer can accept an updater as its data member, where an updater is a class derived from `paddle.trainer.Updater`. This is to make it easier to customize trainers, as discussed here.

### Event Handler

`paddle.train` and `paddle.trainer.XXX.train` take an optional parameter `event_handler`, which should be either `None` or a function that handle some events:

1. BeginTraining
2. EndTraining
3. BeginIteration
4. EndIteration
5. BeginPass
6. EndPass

where EndPass is sent if and only if the reader yields `end_pass=True`.

An example as follows:

    def event_handler(event):
        if ininstance(event, paddle.event.EndIteration):
            print paddle.test(...)

    paddle.train(topology, parameters, reader, event_handler)

If we are writing a PaddlePaddle program in and for iPython/Jypyter, we can use metaplotlib in the event handler to plot a curve of cost/error versus iterations, as shown here.

### Distributed Training

If users want to do distributed training on a cluster, s/he should call `paddle.dist_train` and provides access tokens to the cluster as a parameter.

For example, if the user has a TLS certificate that allows him to access a Kubernetes cluster, s/he should be able to call

    paddle.dist_train(model,
                      trainer=paddle.trainer.SGD(...,
                                                 paddle.updater.Adam(...)),
                      reader=read,
                      k8s_user="yi",
                      k8s_token="kube_cluster_tls.pem",
                      k8s_job="hello",
                      num_parameter_servers=15)

The pseudo code if `paddle.dist_train` is as follows:

    def dist_train(topology, parameters, trainer, reader, ...):
        if os.getenv("KUBERNETES_SERVICE_HOST") == None:
            image_name = k8s_user + '/' + k8s_job
            docker_build(image_name)
            docker_push()
            kube_ctrl_start_job(image_name, k8s_user, k8s_token)
        else:
            rank = kube_list_containers_in_job_and_return_current_containers_rank()
            if rank == 0:
                master()
            elif rank < 15:
                parameter_server()
            else:
                trainer.train(model, reader=read)

Please be aware that if a process is running on the Kubernetes cluster, it will have some environment variables pre-defined.

If `dist_train` doesn’t see these environment variables, it knows that it’s running on users’ personal computer, and it should work as a launcher. Otherwise, it knows that it’s running on the cluster and need to figure out its role as either the master, or a trainer, or a parameter server.