var_desc.md 3.7 KB
Newer Older
Q
qiaolongfei 已提交
1 2 3
## Background
PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.

P
Peng Li 已提交
4
PaddlePaddle use proto message to describe compile time graph because
Q
qiaolongfei 已提交
5

Q
qiaolongfei 已提交
6
1. Computation graph should be able to be saved to a file.
Q
typo  
qiaolongfei 已提交
7
1. In distributed training, the graph will be serialized and send to multiple workers.
Q
qiaolongfei 已提交
8

Q
qiaolongfei 已提交
9
The computation graph is constructed by Data Node and Operation Node. The concept to represent them is in the table below.
Q
qiaolongfei 已提交
10

Q
qiaolongfei 已提交
11 12 13 14
| |compile time|runtime|
|---|---|---|
|Data|VarDesc(proto)|Variable(cpp)|
|Operation|OpDesc(proto)|Operator(cpp)|
Q
qiaolongfei 已提交
15 16


Q
qiaolongfei 已提交
17 18 19 20 21 22 23
## Definition of VarDesc

A VarDesc should have a name and value, in PaddlePaddle, the value will always be a tensor. Since we use LoDTensor most of the time. We add a LoDTesnorDesc to represent it.

```proto
message VarDesc {
  required string name = 1;
Q
qiaolongfei 已提交
24
  optional LoDTensorDesc lod_tensor = 2;
Q
qiaolongfei 已提交
25
}
Q
qiaolongfei 已提交
26
```
Q
qiaolongfei 已提交
27 28 29 30

## Definition of LodTensorDesc

```proto
Q
qiaolongfei 已提交
31 32 33 34 35 36 37 38 39 40
enum DataType {
  BOOL = 0;
  INT16 = 1;
  INT32 = 2;
  INT64 = 3;
  FP16 = 4;
  FP32 = 5;
  FP64 = 6;
}

Q
qiaolongfei 已提交
41
message LoDTensorDesc {
Q
qiaolongfei 已提交
42 43 44
  required DataType data_type = 1;
  repeated int32 dims = 2; // [UNK, 640, 480] is saved as [-1, 640, 480]
  optional int32 lod_level = 3 [default=0];
Q
qiaolongfei 已提交
45 46 47 48 49
}
```

## Definition of Variable in Python

Q
qiaolongfei 已提交
50
In Python API, layer will take Variable as Input, and return Variable as Output. There should be a class `Variable` in python to help create and manage Variable.
Q
qiaolongfei 已提交
51 52

```python
Q
qiaolongfei 已提交
53
image = Variable(dims=[-1, 640, 480])
Q
qiaolongfei 已提交
54 55 56
# fc1 and fc2 are both Variable
fc1 = layer.fc(input=image, output_size=10)
fc2 = layer.fc(input=fc1, output_size=20)
Q
qiaolongfei 已提交
57
```
Q
qiaolongfei 已提交
58 59
### what should class `Variable` Have
1. `name`.a name of string type is used to mark the value of the Variable.
Q
typo  
qiaolongfei 已提交
60
1. `initializer`. Since our Tensor does not have value. we will always use some Operator to fullfill it when run. So we should have a initialize method to help add the init operator.
Q
qiaolongfei 已提交
61 62
1. `operator`. Variable should record which operator produce itself. The reaon is:
  - we use pd.eval(targets=[var1, var2]) to run the related ops to get the value of var1 and var2. var.op is used to trace the dependency of the current variable.
Q
qiaolongfei 已提交
63

Q
qiaolongfei 已提交
64 65
In PaddlePaddle, we use Block to describe Computation Graph, so in the code we will use Block but not Graph.

Q
qiaolongfei 已提交
66
```python
Q
qiaolongfei 已提交
67 68 69 70
import VarDesc
import LoDTensorDesc
import framework

Q
qiaolongfei 已提交
71
def AddInitialOperator(variable, initializer):
Q
qiaolongfei 已提交
72
	# add an initialize Operator to block to init this Variable
Q
qiaolongfei 已提交
73

Q
qiaolongfei 已提交
74
class Variable(object):
Q
qiaolongfei 已提交
75
   def __init__(self, name, dims, type, initializer):
Q
qiaolongfei 已提交
76
      self._block = get_default_block()
Q
qiaolongfei 已提交
77 78
      self._name = name
      self.op = None
Q
qiaolongfei 已提交
79

Q
qiaolongfei 已提交
80 81 82
      tensor_desc = LoDTensorDesc(data_type=type, dims=dims)
      _var_desc = VarDesc(name=name, lod_tensor=tensor_desc)
      self._var = framework.CreateVar(_var_desc)
Q
qiaolongfei 已提交
83
      self._block.add_var(self)
Q
qiaolongfei 已提交
84 85 86 87

      # add initial op according to initializer
      if initializer is not None:
          AddInitialOperator(self, initializer)
Q
qiaolongfei 已提交
88 89 90 91 92 93

   def dims(self):
      return self._var.dims()

   def data_type(self):
       return self._var.data_type()
Q
qiaolongfei 已提交
94 95 96

   def to_proto(self):
       pass
Q
qiaolongfei 已提交
97 98
```

Q
typo  
qiaolongfei 已提交
99
Then we can use this Variable to create a fc layer in Python.
Q
qiaolongfei 已提交
100 101

```python
Q
qiaolongfei 已提交
102 103 104 105 106 107 108 109 110
import paddle as pd

def flatten_size(X, num_flatten_dims):
  prod = 1 # of last num_flatten_dims
  for i in xrange(num_flatten_dims):
    prod = prod * X.dims[-i-1]
  return prod

def layer.fc(X, output_size, num_flatten_dims):
Q
qiaolongfei 已提交
111 112
  W = Variable(pd.random_uniform(), type=FP32, dims=[flatten_size(X, num_flatten_dims), output_size])
  b = Variable(pd.random_uniform(), type=FP32, dims=[output_size])
Q
qiaolongfei 已提交
113 114 115 116 117
  out = Variable(type=FP32)
  y = operator.fc(X, W, b, output=out) # fc will put fc op input into out
  pd.InferShape(y)
  return out

Q
qiaolongfei 已提交
118
x = Variable(dims=[-1, 640, 480])
Q
qiaolongfei 已提交
119 120 121
y = layer.fc(x, output_size=100)
z = layer.fc(y, output_size=200)

Q
qiaolongfei 已提交
122 123
paddle.eval(targets=[z], ...)
print(z)
Q
qiaolongfei 已提交
124
```