@@ -55,17 +55,23 @@ Let us consolidate the discussion by presenting some examples.
...
@@ -55,17 +55,23 @@ Let us consolidate the discussion by presenting some examples.
The following C++ programs shows how blocks are used with the `if-else` structure:
The following C++ programs shows how blocks are used with the `if-else` structure:
```c++
```c++
namespacepd=paddle;
intx=10;
intx=10;
inty=20;
inty=1;
intout;
intz=10;
boolcond=false;
boolcond=false;
into1,o2;
if(cond){
if(cond){
intz=x+y;
intz=x+y;
out=softmax(z);
o1=z;
o2=pd::layer::softmax(z);
}else{
}else{
intz=fc(x);
intd=pd::layer::fc(z);
out=z;
o1=d;
o2=d+1;
}
}
```
```
An equivalent PaddlePaddle program from the design doc of the [IfElseOp operator](./if_else_op.md) is as follows:
An equivalent PaddlePaddle program from the design doc of the [IfElseOp operator](./if_else_op.md) is as follows:
...
@@ -73,57 +79,55 @@ An equivalent PaddlePaddle program from the design doc of the [IfElseOp operator
...
@@ -73,57 +79,55 @@ An equivalent PaddlePaddle program from the design doc of the [IfElseOp operator
```python
```python
importpaddleaspd
importpaddleaspd
x=var(10)
x=minibatch([10,20,30])# shape=[None, 1]
y=var(20)
y=var(1)# shape=[1], value=1
cond=var(false)
z=minibatch([10,20,30])# shape=[None, 1]
ie=pd.create_ifelseop(inputs=[x],output_num=1)
cond=larger_than(x,15)# [false, true, true]
ie=pd.ifelse()
withie.true_block():
withie.true_block():
x=ie.inputs(true,0)
d=pd.layer.add_scalar(x,y)
z=operator.add(x,y)
ie.output(d,pd.layer.softmax(d))
ie.set_output(true,0,operator.softmax(z))
withie.false_block():
withie.false_block():
x=ie.inputs(false,0)
d=pd.layer.fc(z)
z=layer.fc(x)
ie.output(d,d+1)
ie.set_output(true,0,operator.softmax(z))
o1,o2=ie(cond)
out=b(cond)
```
```
In both examples, the left branch computes `softmax(x+y)` and the right branch computes`fc(x)`.
In both examples, the left branch computes `x+y` and `softmax(x+y)`, the right branch computes `x+1` and`fc(x)`.
A difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances. The `ie.input(true, 0)` invocation returns instances in the 0-th input, `x`, that corresponds to true values in `cond` as the local variable `x`, where `ie.input(false, 0)` returns instances corresponding to false values.
A difference is that variables in the C++ program contain scalar values, whereas those in the PaddlePaddle programs are mini-batches of instances. The `ie.input(true, 0)` invocation returns instances in the 0-th input, `x`, that corresponds to true values in `cond` as the local variable `x`, where `ie.input(false, 0)` returns instances corresponding to false values.
### Blocks with `for` and `RNNOp`
### Blocks with `for` and `RNNOp`
The following RNN model from the [RNN design doc](./rnn.md)
The following RNN model from the [RNN design doc](./rnn.md)
```python
```python
x=sequence([10,20,30])
x=sequence([10,20,30])# shape=[None, 1]
m=var(0)
m=var(0)# shape=[1]
W=tensor()
W=var(0.314,param=true)# shape=[1]
U=tensor()
U=var(0.375,param=true)# shape=[1]
rnn=create_rnn(inputs=[input])
rnn=pd.rnn()
withrnn.stepnet()asnet:
withrnn.step():
x=net.set_inputs(0)
h=rnn.memory(init=m)
h=net.add_memory(init=m)
hh=rnn.previous_memory(h)
fc_out=pd.matmul(W,x)
a=layer.fc(W,x)
hidden_out=pd.matmul(U,h.pre(n=1))
b=layer.fc(U,hh)
sum=pd.add_two(fc_out,hidden_out)
s=pd.add(a,b)
act=pd.sigmoid(sum)
act=pd.sigmoid(s)
h.update(act)# update memory with act
rnn.update_memory(h,act)
net.set_outputs(0,act,hidden_out)# two outputs
rnn.output(a,b)
o1,o2=rnn()
o1,o2=rnn()
printo1,o2
```
```
has its equivalent C++ program as follows
has its equivalent C++ program as follows
```c++
```c++
int*x={10,20,30};
int*x={10,20,30};
intm=0;
int*m={0};
intW=some_value();
int*W={0.314};
intU=some_other_value();
int*U={0.375};
intmem[sizeof(x)/sizeof(x[0])+1];
intmem[sizeof(x)/sizeof(x[0])+1];
into1[sizeof(x)/sizeof(x[0])+1];
into1[sizeof(x)/sizeof(x[0])+1];
...
@@ -131,20 +135,16 @@ int o2[sizeof(x) / sizeof(x[0]) + 1];
...
@@ -131,20 +135,16 @@ int o2[sizeof(x) / sizeof(x[0]) + 1];
for(inti=1;i<=sizeof(x)/sizeof(x[0]);++i){
for(inti=1;i<=sizeof(x)/sizeof(x[0]);++i){
intx=x[i-1];
intx=x[i-1];
if(i==1)mem[0]=m;
if(i==1)mem[0]=m;
intfc_out=W*x;
inta=W*x;
inthidden_out=Y*mem[i-1];
intb=Y*mem[i-1];
intsum=fc_out+hidden_out;
ints=fc_out+hidden_out;
intact=sigmoid(sum);
intact=sigmoid(sum);
mem[i]=act;
mem[i]=act;
o1[i]=act;
o1[i]=act;
o2[i]=hidden_out;
o2[i]=hidden_out;
}
}
print_array(o1);
print_array(o2);
```
```
## Compilation and Execution
## Compilation and Execution
Like TensorFlow programs, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest part executes the message for training or inference.
Like TensorFlow programs, a PaddlePaddle program is written in Python. The first part describes a neural network as a protobuf message, and the rest part executes the message for training or inference.
...
@@ -210,11 +210,11 @@ a = pd.Varaible(shape=[20, 20])
...
@@ -210,11 +210,11 @@ a = pd.Varaible(shape=[20, 20])
IfOp should have only one branch. An IfOp operator takes a `cond` variable whose value must be a vector of N boolean elements. Its return value has N instances. If cond[i] == True, input instance input[i] will go through true_block() and generate output[i]; otherwise it will produce output from false_bloack().
# The `IfElse` Operator
```python
PaddlePaddle's `IfElse` operator differs from TensorFlow's:
importpaddleaspd
x=var()
- the TensorFlow version takes a scalar boolean value as the condition so that the whole mini-batch goes to either the true or the false branch, whereas
y=var()
- the PaddlePaddle version takes a vector of boolean value as the condition, and instances corresponding to true values go to the true branch, those corresponding to false values go to the false branch.
cond=var()
default_value=var()
## Example
b=pd.create_ifelseop(inputs=[x],output_num=1)
withb.true_block():
The following PaddlePaddle program shows the usage of the IfElse operator:
x=b.inputs(0)
z=operator.add(x,y)
b.set_output(0,operator.softmax(z))
withb.false_block():
x=b.inputs(0)
z=layer.fc(x)
b.set_output(0,operator.softmax(z))
out=b(cond)
```
If only true_block is set in an IfElseOp, a special case is that we can have a default value for false as:
A challenge to implement the `IfElse` operator is to infer those variables to be split, or, say, to identify the variable of the mini-batch or those derived from the mini-batch.
An equivalent C++ program is as follows:
```c++
namespacepd=paddle;
intx=10;
inty=1;
intz=10;
boolcond=false;
into1,o2;
if(cond){
intd=x+y;
o1=z;
o2=pd::layer::softmax(z);
}else{
intd=pd::layer::fc(z);
o1=d;
o2=d+1;
}
```
```
where default_value is a list of vars for `cond` == False.
The basic structure of a PaddlePaddle program is some nested blocks, as a C++ or Java program.
## Compile and Execution
A PaddlePaddle program consists of two parts -- the first generates a `ProgramDesc` protobuf message that describes the program, and the second runs this message using a C++ class `Executor`.
As described in [graph.md](./graph.md), the first five lines of the following PaddlePaddle program
A simple example PaddlePaddle program can be found in [graph.md](./graph.md):
```python
```python
x=layer.data("images")
x=layer.data("images")
...
@@ -13,36 +15,112 @@ optimize(cost)
...
@@ -13,36 +15,112 @@ optimize(cost)
train(cost,reader=mnist.train())
train(cost,reader=mnist.train())
```
```
generates, or compiles, a PaddelPaddle program, which is represented by the following protobuf message:
The first five lines of the following PaddlePaddle program generates, or, compiles, the `ProgramDesc` message. The last line runs it.
```protobuf
## Programs and Blocks
messageProgramDesc{
repeatedBlockDescblocks=1;
The basic structure of a PaddlePaddle program is some nested blocks, as a C++ or Java program.
- program: some nested blocks
-[block](./block.md):
- some local variable definitions, and
- a sequence of operators
The concept of block comes from usual programs. For example, the following C++ program has three blocks:
```c++
intmain(){// block 0
inti=0;
if(i<10){// block 1
for(intj=0;j<10;j++){// block 2
}
}
return0;
}
}
```
The following PaddlePaddle program has three blocks:
```python
importpaddleaspd//block0
x=minibatch([10,20,30])# shape=[None, 1]
y=var(1)# shape=[1], value=1
z=minibatch([10,20,30])# shape=[None, 1]
cond=larger_than(x,15)# [false, true, true]
ie=pd.ifelse()
withie.true_block()://block1
d=pd.layer.add_scalar(x,y)
ie.output(d,pd.layer.softmax(d))
withie.false_block()://block2
d=pd.layer.fc(z)
ie.output(d,d+1)
o1,o2=ie(cond)
```
## `BlockDesc` and `ProgramDesc`
All protobuf messages are defined in `framework.proto`.
`BlockDesc` is straight-forward -- it includes local variable definitions, `vars`, and a sequence of operators, `ops`.
```protobuf
messageBlockDesc{
messageBlockDesc{
requiredint32parent=1;
requiredint32parent=1;
repeatedVarDescvars=2;
repeatedVarDescvars=2;
repeatedOpDescops=3;
repeatedOpDescops=3;
}
}
```
The parent ID indicates the parent block so that operators in a block can refer to variables defined locally and also those defined in their ancestor blocks.
All hierarchical blocks in a program are flattened and stored in an array. The block ID is the index of the block in this array.
```protobuf
messageProgramDesc{
repeatedBlockDescblocks=1;
}
```
### Global Block
The global block is the first one in the above array.
## Operators that Use Blocks
In the above example, the operator `IfElseOp` has two blocks -- the true branch and the false branch.
The definition of `OpDesc` shows that an operator could have some attributes:
```protobuf
messageOpDesc{
messageOpDesc{
AttrDescattrs=1;
AttrDescattrs=1;
...
...
}
}
```
and an attribute could be of type block, which is, in fact, a block ID as described above:
```
message AttrDesc {
message AttrDesc {
requiredAttrTypetype=1;
required string name = 1;
// index into ProgramDesc::blocks when type==BLOCK
enum AttrType {
optionalint32block=2;
INT = 1,
STRING = 2,
...
BLOCK = ...
}
required AttrType type = 2;
optional int32 block = 10; // when type == BLOCK
...
...
}
}
```
```
When each of the first five lines runs, related Python function, e.g., `layer.fc`, calls C++ InferShape functions. This InferShape function needs to access the properties of VarDesc's accessed by the current OpDesc. These VarDesc's might not be defined in the current block, but in some ancestor blocks. This requires that we can trace the parent of a block.
## InferShape
A nested block is often an attribute of an operator, most likely, an IfElseOp or a WhileOp. In above solution, all blocks are in `ProgramDesc::blocks`, this implicitly assigns a zero-based ID to each block -- the index of the block in `ProgramDesc::blocks`. So that `AttrDesc::block` could be an integer block ID.
With this design, the InferShape function should take the following parameters:
With this design, the InferShape function should take the following parameters:
Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call `states[step_id]` will get the state in `step_id`th time step.
An RNN can be implemented with the following pseudocode
output_segments[step]=states[step]// take state as output
step++;
}
```
According to the [RNN roadmap](https://github.com/PaddlePaddle/Paddle/issues/4561), there are several different RNNs that PaddlePaddle will eventually support.
Currently, the basic RNN implementation supported by PaddlePaddle is the `recurrent_op` which takes tensors as input and splits them into `input_segments`.
Since a tensor cannot store variable-length sequences directly, PaddlePaddle implements the tensor with level of details (`LoDTensor` for short).
Segmenting the `LoDTensor` is much more complicated than splitting a tensor, that makes it necessary to refactor the `recurrent_op` with `LoDTensor` segmenting support.
As the next step in RNN support, `dynamic_recurrent_op` should be introduced to handle inputs with variable-length sequences.
The implementation is similar to `recurrent_op`.
The key difference is the way **the original input `LoDTensors` and outupts are split to get the `input_segments` and the `output_segments`.**
Though it can't be built over `recurrent_op` or `dynamic_recurrent_op` directly,
the logic behind splitting a tensor or a LoD tensor into `input_segments` remains the same.
## Why `TensorArray`
The logic behind splitting the inputs to segments, states and outputs is similar and can be shared in a seperate module.
The array of `states`, `input_segments` and `output_segments` would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes.
So there should be an array-like container, which can store the segments of a tensor or LoD tensor.
**This container can store an array of tensors and provides several methods to split a tensor or a LoD tensor** .
This is where the notion of `TensorArray` comes from.
## Introduce TensorArray to uniform all the three RNNs
TensorArray as a new concept is borrowed from TensorFlow,
TensorArray as a new concept is borrowed from TensorFlow,
it is meant to be used with dynamic iteration primitives such as `while_loop` and `map_fn`.
it is meant to be used with dynamic iteration primitives such as `while_loop` and `map_fn`.
This concept can be used to support our new design of dynamic operations, and help to refactor some existing variant-sentence-related layers,
This concept can be used to support our new design of dynamic operations, and help to refactor some existing variant-sentence-related layers,
such as `RecurrentGradientMachine`.
such as `recurrent_op`, `RecurrentGradientMachine`.
In [our design for dynamic RNN](https://github.com/PaddlePaddle/Paddle/pull/4401),
In [our design for dynamic RNN](https://github.com/PaddlePaddle/Paddle/pull/4401),
`TensorArray` is used to segment inputs and store states in all time steps.
`TensorArray` is used to segment inputs and store states in all time steps.
By providing some methods similar to a C++ array,
By providing some methods similar to a C++ array,
the definition of some state-based dynamic models such as RNN could be more natural and highly flexible.
the definition of some state-based dynamic models such as RNN can be more natural and highly flexible.
## Dynamic-Related Methods
## Dynamic-operations on TensorArray
Some basic methods should be proposed as follows:
`TensorArray` will be used directly when defining dynamic models, so some operators listed below should be implemented
### stack()
Pack the values in a `TensorArray` into a tensor with rank one higher than each tensor in `values`.
```python
### unstack(axis=0)
# several helper operators for TensorArray
Unpacks the given dimension of a rank-`R` tensor into rank-`(R-1)` tensors.
deftensor_array_stack(ta,tensor):
### concat()
'''
Return the values in the `TensorArray` as a concatenated Tensor.
get a tensor array `ta`, return a packed `tensor`.
### write(index, value, data_shared=true)
'''
Write value into index of the TensorArray.
pass
### read(index)
Read the value at location `index` in the `TensorArray`.
deftensor_array_unstack(tensor,ta):
### size()
'''
Return the number of values.
get a `tensor`, unstack it and get a tensor array `ta`.
get a `tensor` and a scalar tensor `index`, write `tensor` into index-th
value of the tensor array `ta`.
`data_shared` is an attribute that specifies whether to copy or reference the tensors.
'''
pass
deftensor_array_read(ta,index,tensor):
'''
get a tensor array `ta`, a scalar tensor `index`, read the index-th value of
`ta` and return as the `tensor`.
'''
pass
deftensor_array_size(ta,tensor):
'''
get a tensor array `ta`, return the size of `ta` and return as the scalar `tensor`.
'''
pass
```
It is trivial for users to use so many low-level operators, so some helper methods should be proposed in python wrapper to make `TensorArray` easier to use,
for example
```python
classTensorArray:
def__init__(self,name):
self.name=name
self.desc=TensorArrayDesc()
defstack(self,name=None):
'''
Pack the values in a `TensorArray` into a tensor with rank one higher
than each tensor in `values`.
`stack` can be used to split tensor into time steps for RNN or whileloop.
@name: str
the name of the variable to output.
'''
tensor=NewVar(name)
tensor_array_stack(self.name,tensor)
returntensor
defunstack(self,input):
'''
Unpacks the given dimension of a rank-`R` tensor into rank-`(R-1)` tensors.
`unstack` can be used to concatenate all the time steps for RNN or whileloop.
@input: str
the name of input tensor
'''
tensor_array_unstack(tensor,self.name)
defwrite(self,index,value,data_shared=True):
'''
Write value into index of the TensorArray.
If `data_shared` is set to True, than the index-th value in TensorArray will
Read the value at location `index` in the `TensorArray`.
@index: str
name of a scalar tensor
@output:
name of a output variable
'''
tensor_array_read(self.name,index,output)
defsize(self,output):
'''
Return the number of values.
@output: str
name of a scalar tensor
'''
tensor_array_size(self.name,output)
```
## LoDTensor-related Supports
## LoDTensor-related Supports
The `RecurrentGradientMachine` in Paddle serves as a flexible RNN layer; it takes variant length sequences as input,
The `RecurrentGradientMachine` in Paddle serves as a flexible RNN layer; it takes varience-length sequences as input, and output sequences too.
because each step of RNN could only take a tensor-represented batch of data as input,
Since each step of RNN can only take a tensor-represented batch of data as input,
some preprocess should be taken on the inputs such as sorting the sentences by their length in descending order and cut each word and pack to new batches.
some preprocess should be taken on the inputs such as sorting the sentences by their length in descending order and cut each word and pack to new batches.
Such cut-like operations can be embedded into `TensorArray` as general methods called `unpack` and `pack`.
Such cut-like operations can be embedded into `TensorArray` as general methods called `unpack` and `pack`,
these two operations are similar to `stack` and `unstack` except that they operate on variable-length sequences formated as a LoD tensor rather than a tensor.
Some definitions are like
```python
defunpack(level):
'''
Split LodTensor in some `level` and generate batches, if set `sort_by_length`,
will sort by length.
With these two methods, a variant-sentence-RNN can be implemented like
Returns:
- a new `TensorArray`, whose values are LodTensors and represents batches
of data.
- an int32 Tensor, which stores the map from the new batch's indices to
original LoDTensor
'''
pass
defpack(level,indices_map):
'''
Recover the original LoD-arranged LoDTensor with the values in a `TensorArray`
and `level` and `indices_map`.
'''
pass
```
With these two methods, a varience-length sentence supported RNN can be implemented like
the code above shows that by embedding the LoDTensor-related preprocess operations into `TensorArray`,
the code above shows that by embedding the LoDTensor-related preprocess operations into `TensorArray`,
the implementation of a RNN that supports varient-length sentences is far more concise than `RecurrentGradientMachine` because the latter mixes all the codes together, hard to read and extend.
the implementation of a RNN that supports varient-length sentences is far more concise than `RecurrentGradientMachine` because the latter mixes all the codes together, hard to read and extend.
some details are as follows.
### unpack(level, sort_by_length)
Split LodTensor in some `level` and generate batches, if set `sort_by_length`, will sort by length.
Returns:
- a new `TensorArray`, whose values are LodTensors and represents batches of data.
- an int32 Tensor, which stores the map from the new batch's indices to original LoDTensor
### pack(level, indices_map)
Recover the original LoD-arranged LoDTensor with the values in a `TensorArray` and `level` and `indices_map`.
