PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.
PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.
The data structure to describe the compile time graph should be able to be serialized for distributed training. So we use proto message to describe the graph: OpDesc to describe computation and VarDesc to describe data.
PaddlePaddle use proto message to describe compile time graph for
PaddlePaddle will generate these data structure according to user's description and do some optimization, such as:
1. Computation graph should be able to be saved to a file.
1. In distributed trianing, graph will be serialized and send to multiple workers.
1. InferShape. Infer the Output size according to Input size and set them into VarDesc.
The computation graph is constructed by Data Node and Operation Node. The concept to represent them is in the table below.
1. memory optimise and reuse. Scan all the memory that will be used and reuse some memory that is allocated before but will not be used anymore to reduce memory.
VarDesc is used to describe different kinds of Variable value, such as Tensor, scalar, and scope:
| |compile time|runtime|
|---|---|---|
|Data|VarDesc(proto)|Variable(cpp)|
|Operation|OpDesc(proto)|Operator(cpp)|
## Definition of VarDesc in Proto
## Definition of VarDesc
A VarDesc should have a name and value, in PaddlePaddle, the value will always be a tensor. Since we use LoDTensor most of the time. We add a LoDTesnorDesc to represent it.
```proto
messageVarDesc{
requiredstringname=1;
optionalLoDTesnorDesclod_tensor=2;//
}
```
```
## Definition of LodTensorDesc
```proto
messageLoDTensorDesc{
messageLoDTensorDesc{
enumType{
enumType{
BOOL=0;
INT16=1;
INT16=1;
INT32=2;
INT32=2;
INT64=3;
INT64=3;
FP16=4;
FP16=4;
FP32=5;
FP32=5;
DOUBLE = 6
FP64=6
BOOL = 7;
}
}
Type element_type = 1;
Typedata_type=1;
repeated int dims = 2; // [UNK, UNK, 6000] is saved as [-1, -1, 6000]
repeatedintdims=2;// [UNK, 6000] is saved as [-1, 6000]