# Design Doc: Computations as a Graph

A primary goal of the refactorization of PaddlePaddle is a more flexible representation of deep learning computation, in particular, a graph of operators and variables, instead of sequences of layers as before.

This document explains that the construction of a graph as three steps:

- construct the forward part
- construct the backward part
- construct the optimization part

## The Construction of a Graph

Let us take the problem of image classification as a simple example. The application program that trains the model looks like:

    x = layer.data("images")
    l = layer.data("label")
    y = layer.fc(x)
    cost = layer.mse(y, l)
    optimize(cost)
    train(cost, reader=mnist.train())

### Forward Part

The first four lines of above program build the forward part of the graph.

In particular, the first line `x = layer.data("images")` creates variable x and a Feed operator that copies a column from the minibatch to x. `y = layer.fc(x)` creates not only the FC operator and output variable y, but also two parameters, W and b, and the initialization operators.

Initialization operators are kind of “run-once” operators – the `Run` method increments a class data member counter so to run at most once. By doing so, a parameter wouldn’t be initialized repeatedly, say, in every minibatch.

In this example, all operators are created as `OpDesc` protobuf messages, and all variables are `VarDesc`. These protobuf messages are saved in a `BlockDesc` protobuf message.

### Backward Part

The fifth line `optimize(cost)` calls two functions, `ConstructBackwardGraph` and `ConstructOptimizationGraph`.

`ConstructBackwardGraph` traverses the forward graph in the `BlockDesc` protobuf message and builds the backward part.

According to the chain rule of gradient computation, `ConstructBackwardGraph` would

1. create a gradient operator G for each operator F,
2. make all inputs, outputs, and outputs’ gradient of F as inputs of G,
3. create gradients for all inputs of F, except for those who don’t have gradients, like x and l, and
4. make all these gradients as outputs of G.

### Optimization Part

For each parameter, like W and b created by `layer.fc`, marked as double circles in above graphs, `ConstructOptimizationGraph` creates an optimization operator to apply its gradient. Here results in the complete graph:

## Block and Graph

The word block and graph are interchangable in the desgin of PaddlePaddle. A Block is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.

A Block keeps operators in an array `BlockDesc::ops`

    message BlockDesc {
      repeated OpDesc ops = 1;
      repeated VarDesc vars = 2;
    }

in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.