IfOp should have only one branch. An IfOp operator takes a `cond` variable whose value must be a vector of N boolean elements. Its return value has N instances. If cond[i] == True, input instance input[i] will go through true_block() and generate output[i]; otherwise it will produce output from false_bloack().
# The `IfElse` Operator
```python
import paddle as pd
PaddlePaddle's `IfElse` operator differs from TensorFlow's:
x = var()
y = var()
cond = var()
default_value = var()
b = pd.create_ifelseop(inputs=[x], output_num=1)
with b.true_block():
x = b.inputs(0)
z = operator.add(x, y)
b.set_output(0, operator.softmax(z))
with b.false_block():
x = b.inputs(0)
z = layer.fc(x)
b.set_output(0, operator.softmax(z))
out = b(cond)
```
- the TensorFlow version takes a scalar boolean value as the condition so that the whole mini-batch goes to either the true or the false branch, whereas
- the PaddlePaddle version takes a vector of boolean value as the condition, and instances corresponding to true values go to the true branch, those corresponding to false values go to the false branch.
## Example
The following PaddlePaddle program shows the usage of the IfElse operator:
If only true_block is set in an IfElseOp, a special case is that we can have a default value for false as:
```python
import paddle as pd
x = var()
y = var()
cond = var()
default_value = var()
b = pd.create_ifelseop(inputs=[x], output_num=1, default_value)
with b.true_block():
x = b.inputs(0)
z = operator.add(x, y)
b.set_output(0, operator.softmax(z))
x = minibatch([10, 20, 30]) # shape=[None, 1]
y = var(1) # shape=[1], value=1
z = minibatch([10, 20, 30]) # shape=[None, 1]
cond = larger_than(x, 15) # [false, true, true]
ie = pd.ifelse()
with ie.true_block():
d = pd.layer.add(x, y)
ie.output(d, pd.layer.softmax(d))
with ie.false_block():
d = pd.layer.fc(z)
ie.output(d, d+1)
o1, o2 = ie(cond)
```
out = b(cond)
A challenge to implement the `IfElse` operator is to infer those variables to be split, or, say, to identify the variable of the mini-batch or those derived from the mini-batch.
An equivalent C++ program is as follows:
```c++
namespace pd = paddle;
int x = 10;
int y = 1;
int z = 10;
bool cond = false;
int o1, o2;
if (cond) {
int d = x + y;
o1 = z;
o2 = pd::layer::softmax(z);
} else {
int d = pd::layer::fc(z);
o1 = d;
o2 = d+1;
}
```
where default_value is a list of vars for `cond` == False.
This design doc presents the necessity of a new C++ class `TensorArray`.
In addition to the very simple C++ implementation
```c++
class TensorArray {
public:
explicit TensorArray(const LoDTensor&);
explicit TensorArray(size_t size);
private:
vector<LoDTensor> values_;
};
```
We also need to expose it to PaddlePaddle's Python API,
because users would want to use it with our very flexible operators `WhileLoop`.
An example for a RNN based on dynamic operators is
```python
input = pd.data(...)
num_steps = Var(12)
TensorArray states(size=num_steps)
TensorArray step_inputs(unstack_from=input)
TensorArray step_outputs(size=num_steps)
W = Tensor(...)
U = Tensor(...)
default_state = some_op()
step = Var(1)
wloop = paddle.create_whileloop(loop_vars=[step])
with wloop.frame():
wloop.break_if(pd.equal(step, num_steps)
pre_state = states.read(step-1, default_state)
step_input = step_inputs.read(step)
state = pd.sigmoid(pd.matmul(U, pre_state) + pd.matmul(W, step_input))
states.write(step, state)
step_outputs.write(step, state) # output state
step.update(state+1)
output = step_outputs.stack()
```
## Background
Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call `states[step_id]` will get the state in `step_id`th time step.
An RNN can be implemented with the following pseudocode
```c++
Array states;
Array input_segments;
Array output_segments;
Parameter W, U;
step = 1
seq_len = 12
while_loop {
if (step == seq_len) break;
states[step] = sigmoid(W * states[step-1] + U * input_segments[step]);
output_segments[step] = states[step] // take state as output
step++;
}
```
According to the [RNN roadmap](https://github.com/PaddlePaddle/Paddle/issues/4561), there are several different RNNs that PaddlePaddle will eventually support.
Currently, the basic RNN implementation supported by PaddlePaddle is the `recurrent_op` which takes tensors as input and splits them into `input_segments`.
Since a tensor cannot store variable-length sequences directly, PaddlePaddle implements the tensor with level of details (`LoDTensor` for short).
Segmenting the `LoDTensor` is much more complicated than splitting a tensor, that makes it necessary to refactor the `recurrent_op` with `LoDTensor` segmenting support.
As the next step in RNN support, `dynamic_recurrent_op` should be introduced to handle inputs with variable-length sequences.
The implementation is similar to `recurrent_op`.
The key difference is the way **the original input `LoDTensors` and outupts are split to get the `input_segments` and the `output_segments`.**
Though it can't be built over `recurrent_op` or `dynamic_recurrent_op` directly,
the logic behind splitting a tensor or a LoD tensor into `input_segments` remains the same.
## Why `TensorArray`
The logic behind splitting the inputs to segments, states and outputs is similar and can be shared in a seperate module.
The array of `states`, `input_segments` and `output_segments` would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes.
So there should be an array-like container, which can store the segments of a tensor or LoD tensor.
**This container can store an array of tensors and provides several methods to split a tensor or a LoD tensor** .
This is where the notion of `TensorArray` comes from.
## Introduce TensorArray to uniform all the three RNNs
TensorArray as a new concept is borrowed from TensorFlow,
it is meant to be used with dynamic iteration primitives such as `while_loop` and `map_fn`.
This concept can be used to support our new design of dynamic operations, and help to refactor some existing variant-sentence-related layers,
such as `RecurrentGradientMachine`.
such as `recurrent_op`, `RecurrentGradientMachine`.
In [our design for dynamic RNN](https://github.com/PaddlePaddle/Paddle/pull/4401),
`TensorArray` is used to segment inputs and store states in all time steps.
By providing some methods similar to a C++ array,
the definition of some state-based dynamic models such as RNN could be more natural and highly flexible.
## Dynamic-Related Methods
Some basic methods should be proposed as follows:
### stack()
Pack the values in a `TensorArray` into a tensor with rank one higher than each tensor in `values`.
### unstack(axis=0)
Unpacks the given dimension of a rank-`R` tensor into rank-`(R-1)` tensors.
### concat()
Return the values in the `TensorArray` as a concatenated Tensor.
### write(index, value, data_shared=true)
Write value into index of the TensorArray.
### read(index)
Read the value at location `index` in the `TensorArray`.
### size()
Return the number of values.
the definition of some state-based dynamic models such as RNN can be more natural and highly flexible.
## Dynamic-operations on TensorArray
`TensorArray` will be used directly when defining dynamic models, so some operators listed below should be implemented
```python
# several helper operators for TensorArray
def tensor_array_stack(ta, tensor):
'''
get a tensor array `ta`, return a packed `tensor`.
'''
pass
def tensor_array_unstack(tensor, ta):
'''
get a `tensor`, unstack it and get a tensor array `ta`.
get a `tensor` and a scalar tensor `index`, write `tensor` into index-th
value of the tensor array `ta`.
`data_shared` is an attribute that specifies whether to copy or reference the tensors.
'''
pass
def tensor_array_read(ta, index, tensor):
'''
get a tensor array `ta`, a scalar tensor `index`, read the index-th value of
`ta` and return as the `tensor`.
'''
pass
def tensor_array_size(ta, tensor):
'''
get a tensor array `ta`, return the size of `ta` and return as the scalar `tensor`.
'''
pass
```
It is trivial for users to use so many low-level operators, so some helper methods should be proposed in python wrapper to make `TensorArray` easier to use,
for example
```python
class TensorArray:
def __init__(self, name):
self.name = name
self.desc = TensorArrayDesc()
def stack(self, name=None):
'''
Pack the values in a `TensorArray` into a tensor with rank one higher
than each tensor in `values`.
`stack` can be used to split tensor into time steps for RNN or whileloop.
@name: str
the name of the variable to output.
'''
tensor = NewVar(name)
tensor_array_stack(self.name, tensor)
return tensor
def unstack(self, input):
'''
Unpacks the given dimension of a rank-`R` tensor into rank-`(R-1)` tensors.
`unstack` can be used to concatenate all the time steps for RNN or whileloop.
@input: str
the name of input tensor
'''
tensor_array_unstack(tensor, self.name)
def write(self, index, value, data_shared=True):
'''
Write value into index of the TensorArray.
If `data_shared` is set to True, than the index-th value in TensorArray will
Read the value at location `index` in the `TensorArray`.
@index: str
name of a scalar tensor
@output:
name of a output variable
'''
tensor_array_read(self.name, index, output)
def size(self, output):
'''
Return the number of values.
@output: str
name of a scalar tensor
'''
tensor_array_size(self.name, output)
```
## LoDTensor-related Supports
The `RecurrentGradientMachine` in Paddle serves as a flexible RNN layer; it takes variant length sequences as input,
because each step of RNN could only take a tensor-represented batch of data as input,
The `RecurrentGradientMachine` in Paddle serves as a flexible RNN layer; it takes varience-length sequences as input, and output sequences too.
Since each step of RNN can only take a tensor-represented batch of data as input,
some preprocess should be taken on the inputs such as sorting the sentences by their length in descending order and cut each word and pack to new batches.
Such cut-like operations can be embedded into `TensorArray` as general methods called `unpack` and `pack`.
Such cut-like operations can be embedded into `TensorArray` as general methods called `unpack` and `pack`,
these two operations are similar to `stack` and `unstack` except that they operate on variable-length sequences formated as a LoD tensor rather than a tensor.
Some definitions are like
```python
def unpack(level):
'''
Split LodTensor in some `level` and generate batches, if set `sort_by_length`,
will sort by length.
With these two methods, a variant-sentence-RNN can be implemented like
Returns:
- a new `TensorArray`, whose values are LodTensors and represents batches
of data.
- an int32 Tensor, which stores the map from the new batch's indices to
original LoDTensor
'''
pass
def pack(level, indices_map):
'''
Recover the original LoD-arranged LoDTensor with the values in a `TensorArray`
and `level` and `indices_map`.
'''
pass
```
With these two methods, a varience-length sentence supported RNN can be implemented like
the code above shows that by embedding the LoDTensor-related preprocess operations into `TensorArray`,
the implementation of a RNN that supports varient-length sentences is far more concise than `RecurrentGradientMachine` because the latter mixes all the codes together, hard to read and extend.
some details are as follows.
### unpack(level, sort_by_length)
Split LodTensor in some `level` and generate batches, if set `sort_by_length`, will sort by length.
Returns:
- a new `TensorArray`, whose values are LodTensors and represents batches of data.
- an int32 Tensor, which stores the map from the new batch's indices to original LoDTensor
### pack(level, indices_map)
Recover the original LoD-arranged LoDTensor with the values in a `TensorArray` and `level` and `indices_map`.
<p>IfOp should have only one branch. An IfOp operator takes a <codeclass="docutils literal"><spanclass="pre">cond</span></code> variable whose value must be a vector of N boolean elements. Its return value has N instances. If cond[i] == True, input instance input[i] will go through true_block() and generate output[i]; otherwise it will produce output from false_bloack().</p>
<divclass="section"id="the-ifelse-operator">
<spanid="the-ifelse-operator"></span><h1>The <codeclass="docutils literal"><spanclass="pre">IfElse</span></code> Operator<aclass="headerlink"href="#the-ifelse-operator"title="Permalink to this headline">¶</a></h1>
<p>PaddlePaddle’s <codeclass="docutils literal"><spanclass="pre">IfElse</span></code> operator differs from TensorFlow’s:</p>
<ulclass="simple">
<li>the TensorFlow version takes a scalar boolean value as the condition so that the whole mini-batch goes to either the true or the false branch, whereas</li>
<li>the PaddlePaddle version takes a vector of boolean value as the condition, and instances corresponding to true values go to the true branch, those corresponding to false values go to the false branch.</li>
</ul>
<divclass="section"id="example">
<spanid="example"></span><h2>Example<aclass="headerlink"href="#example"title="Permalink to this headline">¶</a></h2>
<p>The following PaddlePaddle program shows the usage of the IfElse operator:</p>
<p>A challenge to implement the <codeclass="docutils literal"><spanclass="pre">IfElse</span></code> operator is to infer those variables to be split, or, say, to identify the variable of the mini-batch or those derived from the mini-batch.</p>
<spanid="design-for-tensorarray"></span><h1>Design for TensorArray<aclass="headerlink"href="#design-for-tensorarray"title="Permalink to this headline">¶</a></h1>
<p>This design doc presents the necessity of a new C++ class <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>.
In addition to the very simple C++ implementation</p>
<spanid="background"></span><h2>Background<aclass="headerlink"href="#background"title="Permalink to this headline">¶</a></h2>
<p>Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call <codeclass="docutils literal"><spanclass="pre">states[step_id]</span></code> will get the state in <codeclass="docutils literal"><spanclass="pre">step_id</span></code>th time step.</p>
<p>An RNN can be implemented with the following pseudocode</p>
<spanclass="n">output_segments</span><spanclass="p">[</span><spanclass="n">step</span><spanclass="p">]</span><spanclass="o">=</span><spanclass="n">states</span><spanclass="p">[</span><spanclass="n">step</span><spanclass="p">]</span><spanclass="c1">// take state as output</span>
<p>According to the <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/issues/4561">RNN roadmap</a>, there are several different RNNs that PaddlePaddle will eventually support.</p>
<p>Currently, the basic RNN implementation supported by PaddlePaddle is the <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code> which takes tensors as input and splits them into <codeclass="docutils literal"><spanclass="pre">input_segments</span></code>.</p>
<p>Since a tensor cannot store variable-length sequences directly, PaddlePaddle implements the tensor with level of details (<codeclass="docutils literal"><spanclass="pre">LoDTensor</span></code> for short).
Segmenting the <codeclass="docutils literal"><spanclass="pre">LoDTensor</span></code> is much more complicated than splitting a tensor, that makes it necessary to refactor the <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code> with <codeclass="docutils literal"><spanclass="pre">LoDTensor</span></code> segmenting support.</p>
<p>As the next step in RNN support, <codeclass="docutils literal"><spanclass="pre">dynamic_recurrent_op</span></code> should be introduced to handle inputs with variable-length sequences.</p>
<p>The implementation is similar to <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code>.
The key difference is the way <strong>the original input <codeclass="docutils literal"><spanclass="pre">LoDTensors</span></code> and outupts are split to get the <codeclass="docutils literal"><spanclass="pre">input_segments</span></code> and the <codeclass="docutils literal"><spanclass="pre">output_segments</span></code>.</strong></p>
<p>Though it can’t be built over <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code> or <codeclass="docutils literal"><spanclass="pre">dynamic_recurrent_op</span></code> directly,
the logic behind splitting a tensor or a LoD tensor into <codeclass="docutils literal"><spanclass="pre">input_segments</span></code> remains the same.</p>
</div>
<divclass="section"id="why-tensorarray">
<spanid="why-tensorarray"></span><h2>Why <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code><aclass="headerlink"href="#why-tensorarray"title="Permalink to this headline">¶</a></h2>
<p>The logic behind splitting the inputs to segments, states and outputs is similar and can be shared in a seperate module.</p>
<p>The array of <codeclass="docutils literal"><spanclass="pre">states</span></code>, <codeclass="docutils literal"><spanclass="pre">input_segments</span></code> and <codeclass="docutils literal"><spanclass="pre">output_segments</span></code> would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes.</p>
<p>So there should be an array-like container, which can store the segments of a tensor or LoD tensor.</p>
<p><strong>This container can store an array of tensors and provides several methods to split a tensor or a LoD tensor</strong> .
This is where the notion of <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> comes from.</p>
<spanid="introduce-tensorarray-to-uniform-all-the-three-rnns"></span><h2>Introduce TensorArray to uniform all the three RNNs<aclass="headerlink"href="#introduce-tensorarray-to-uniform-all-the-three-rnns"title="Permalink to this headline">¶</a></h2>
<p>TensorArray as a new concept is borrowed from TensorFlow,
it is meant to be used with dynamic iteration primitives such as <codeclass="docutils literal"><spanclass="pre">while_loop</span></code> and <codeclass="docutils literal"><spanclass="pre">map_fn</span></code>.</p>
<p>This concept can be used to support our new design of dynamic operations, and help to refactor some existing variant-sentence-related layers,
such as <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code>.</p>
such as <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code>, <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code>.</p>
<p>In <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/pull/4401">our design for dynamic RNN</a>,
<codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> is used to segment inputs and store states in all time steps.
By providing some methods similar to a C++ array,
the definition of some state-based dynamic models such as RNN could be more natural and highly flexible.</p>
<divclass="section"id="dynamic-related-methods">
<spanid="dynamic-related-methods"></span><h2>Dynamic-Related Methods<aclass="headerlink"href="#dynamic-related-methods"title="Permalink to this headline">¶</a></h2>
<p>Some basic methods should be proposed as follows:</p>
<divclass="section"id="stack">
<spanid="stack"></span><h3>stack()<aclass="headerlink"href="#stack"title="Permalink to this headline">¶</a></h3>
<p>Pack the values in a <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> into a tensor with rank one higher than each tensor in <codeclass="docutils literal"><spanclass="pre">values</span></code>.</p>
</div>
<divclass="section"id="unstack-axis-0">
<spanid="unstack-axis-0"></span><h3>unstack(axis=0)<aclass="headerlink"href="#unstack-axis-0"title="Permalink to this headline">¶</a></h3>
<p>Unpacks the given dimension of a rank-<codeclass="docutils literal"><spanclass="pre">R</span></code> tensor into rank-<codeclass="docutils literal"><spanclass="pre">(R-1)</span></code> tensors.</p>
the definition of some state-based dynamic models such as RNN can be more natural and highly flexible.</p>
</div>
<divclass="section"id="concat">
<spanid="concat"></span><h3>concat()<aclass="headerlink"href="#concat"title="Permalink to this headline">¶</a></h3>
<p>Return the values in the <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> as a concatenated Tensor.</p>
<spanid="write-index-value-data-shared-true"></span><h3>write(index, value, data_shared=true)<aclass="headerlink"href="#write-index-value-data-shared-true"title="Permalink to this headline">¶</a></h3>
<p>Write value into index of the TensorArray.</p>
</div>
<divclass="section"id="read-index">
<spanid="read-index"></span><h3>read(index)<aclass="headerlink"href="#read-index"title="Permalink to this headline">¶</a></h3>
<p>Read the value at location <codeclass="docutils literal"><spanclass="pre">index</span></code> in the <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>.</p>
<spanid="dynamic-operations-on-tensorarray"></span><h2>Dynamic-operations on TensorArray<aclass="headerlink"href="#dynamic-operations-on-tensorarray"title="Permalink to this headline">¶</a></h2>
<p><codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> will be used directly when defining dynamic models, so some operators listed below should be implemented</p>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="c1"># several helper operators for TensorArray</span>
<spanclass="sd"> get a tensor array `ta`, return the size of `ta` and return as the scalar `tensor`.</span>
<spanclass="sd">'''</span>
<spanclass="k">pass</span>
</pre></div>
</div>
<divclass="section"id="size">
<spanid="size"></span><h3>size()<aclass="headerlink"href="#size"title="Permalink to this headline">¶</a></h3>
<p>Return the number of values.</p>
<p>It is trivial for users to use so many low-level operators, so some helper methods should be proposed in python wrapper to make <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> easier to use,
<spanid="lodtensor-related-supports"></span><h2>LoDTensor-related Supports<aclass="headerlink"href="#lodtensor-related-supports"title="Permalink to this headline">¶</a></h2>
<p>The <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code> in Paddle serves as a flexible RNN layer; it takes variant length sequences as input,
because each step of RNN could only take a tensor-represented batch of data as input,
<p>The <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code> in Paddle serves as a flexible RNN layer; it takes varience-length sequences as input, and output sequences too.</p>
<p>Since each step of RNN can only take a tensor-represented batch of data as input,
some preprocess should be taken on the inputs such as sorting the sentences by their length in descending order and cut each word and pack to new batches.</p>
<p>Such cut-like operations can be embedded into <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> as general methods called <codeclass="docutils literal"><spanclass="pre">unpack</span></code> and <codeclass="docutils literal"><spanclass="pre">pack</span></code>.</p>
<p>With these two methods, a variant-sentence-RNN can be implemented like</p>
<p>Such cut-like operations can be embedded into <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> as general methods called <codeclass="docutils literal"><spanclass="pre">unpack</span></code> and <codeclass="docutils literal"><spanclass="pre">pack</span></code>,
these two operations are similar to <codeclass="docutils literal"><spanclass="pre">stack</span></code> and <codeclass="docutils literal"><spanclass="pre">unstack</span></code> except that they operate on variable-length sequences formated as a LoD tensor rather than a tensor.</p>
@@ -245,20 +426,6 @@ some preprocess should be taken on the inputs such as sorting the sentences by t
</div>
<p>the code above shows that by embedding the LoDTensor-related preprocess operations into <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>,
the implementation of a RNN that supports varient-length sentences is far more concise than <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code> because the latter mixes all the codes together, hard to read and extend.</p>
<spanid="unpack-level-sort-by-length"></span><h3>unpack(level, sort_by_length)<aclass="headerlink"href="#unpack-level-sort-by-length"title="Permalink to this headline">¶</a></h3>
<p>Split LodTensor in some <codeclass="docutils literal"><spanclass="pre">level</span></code> and generate batches, if set <codeclass="docutils literal"><spanclass="pre">sort_by_length</span></code>, will sort by length.</p>
<p>Returns:</p>
<ulclass="simple">
<li>a new <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>, whose values are LodTensors and represents batches of data.</li>
<li>an int32 Tensor, which stores the map from the new batch’s indices to original LoDTensor</li>
</ul>
</div>
<divclass="section"id="pack-level-indices-map">
<spanid="pack-level-indices-map"></span><h3>pack(level, indices_map)<aclass="headerlink"href="#pack-level-indices-map"title="Permalink to this headline">¶</a></h3>
<p>Recover the original LoD-arranged LoDTensor with the values in a <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> and <codeclass="docutils literal"><spanclass="pre">level</span></code> and <codeclass="docutils literal"><spanclass="pre">indices_map</span></code>.</p>
IfOp should have only one branch. An IfOp operator takes a `cond` variable whose value must be a vector of N boolean elements. Its return value has N instances. If cond[i] == True, input instance input[i] will go through true_block() and generate output[i]; otherwise it will produce output from false_bloack().
# The `IfElse` Operator
```python
import paddle as pd
PaddlePaddle's `IfElse` operator differs from TensorFlow's:
x = var()
y = var()
cond = var()
default_value = var()
b = pd.create_ifelseop(inputs=[x], output_num=1)
with b.true_block():
x = b.inputs(0)
z = operator.add(x, y)
b.set_output(0, operator.softmax(z))
with b.false_block():
x = b.inputs(0)
z = layer.fc(x)
b.set_output(0, operator.softmax(z))
out = b(cond)
```
- the TensorFlow version takes a scalar boolean value as the condition so that the whole mini-batch goes to either the true or the false branch, whereas
- the PaddlePaddle version takes a vector of boolean value as the condition, and instances corresponding to true values go to the true branch, those corresponding to false values go to the false branch.
## Example
The following PaddlePaddle program shows the usage of the IfElse operator:
If only true_block is set in an IfElseOp, a special case is that we can have a default value for false as:
```python
import paddle as pd
x = var()
y = var()
cond = var()
default_value = var()
b = pd.create_ifelseop(inputs=[x], output_num=1, default_value)
with b.true_block():
x = b.inputs(0)
z = operator.add(x, y)
b.set_output(0, operator.softmax(z))
x = minibatch([10, 20, 30]) # shape=[None, 1]
y = var(1) # shape=[1], value=1
z = minibatch([10, 20, 30]) # shape=[None, 1]
cond = larger_than(x, 15) # [false, true, true]
ie = pd.ifelse()
with ie.true_block():
d = pd.layer.add(x, y)
ie.output(d, pd.layer.softmax(d))
with ie.false_block():
d = pd.layer.fc(z)
ie.output(d, d+1)
o1, o2 = ie(cond)
```
out = b(cond)
A challenge to implement the `IfElse` operator is to infer those variables to be split, or, say, to identify the variable of the mini-batch or those derived from the mini-batch.
An equivalent C++ program is as follows:
```c++
namespace pd = paddle;
int x = 10;
int y = 1;
int z = 10;
bool cond = false;
int o1, o2;
if (cond) {
int d = x + y;
o1 = z;
o2 = pd::layer::softmax(z);
} else {
int d = pd::layer::fc(z);
o1 = d;
o2 = d+1;
}
```
where default_value is a list of vars for `cond` == False.
This design doc presents the necessity of a new C++ class `TensorArray`.
In addition to the very simple C++ implementation
```c++
class TensorArray {
public:
explicit TensorArray(const LoDTensor&);
explicit TensorArray(size_t size);
private:
vector<LoDTensor> values_;
};
```
We also need to expose it to PaddlePaddle's Python API,
because users would want to use it with our very flexible operators `WhileLoop`.
An example for a RNN based on dynamic operators is
```python
input = pd.data(...)
num_steps = Var(12)
TensorArray states(size=num_steps)
TensorArray step_inputs(unstack_from=input)
TensorArray step_outputs(size=num_steps)
W = Tensor(...)
U = Tensor(...)
default_state = some_op()
step = Var(1)
wloop = paddle.create_whileloop(loop_vars=[step])
with wloop.frame():
wloop.break_if(pd.equal(step, num_steps)
pre_state = states.read(step-1, default_state)
step_input = step_inputs.read(step)
state = pd.sigmoid(pd.matmul(U, pre_state) + pd.matmul(W, step_input))
states.write(step, state)
step_outputs.write(step, state) # output state
step.update(state+1)
output = step_outputs.stack()
```
## Background
Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call `states[step_id]` will get the state in `step_id`th time step.
An RNN can be implemented with the following pseudocode
```c++
Array states;
Array input_segments;
Array output_segments;
Parameter W, U;
step = 1
seq_len = 12
while_loop {
if (step == seq_len) break;
states[step] = sigmoid(W * states[step-1] + U * input_segments[step]);
output_segments[step] = states[step] // take state as output
step++;
}
```
According to the [RNN roadmap](https://github.com/PaddlePaddle/Paddle/issues/4561), there are several different RNNs that PaddlePaddle will eventually support.
Currently, the basic RNN implementation supported by PaddlePaddle is the `recurrent_op` which takes tensors as input and splits them into `input_segments`.
Since a tensor cannot store variable-length sequences directly, PaddlePaddle implements the tensor with level of details (`LoDTensor` for short).
Segmenting the `LoDTensor` is much more complicated than splitting a tensor, that makes it necessary to refactor the `recurrent_op` with `LoDTensor` segmenting support.
As the next step in RNN support, `dynamic_recurrent_op` should be introduced to handle inputs with variable-length sequences.
The implementation is similar to `recurrent_op`.
The key difference is the way **the original input `LoDTensors` and outupts are split to get the `input_segments` and the `output_segments`.**
Though it can't be built over `recurrent_op` or `dynamic_recurrent_op` directly,
the logic behind splitting a tensor or a LoD tensor into `input_segments` remains the same.
## Why `TensorArray`
The logic behind splitting the inputs to segments, states and outputs is similar and can be shared in a seperate module.
The array of `states`, `input_segments` and `output_segments` would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes.
So there should be an array-like container, which can store the segments of a tensor or LoD tensor.
**This container can store an array of tensors and provides several methods to split a tensor or a LoD tensor** .
This is where the notion of `TensorArray` comes from.
## Introduce TensorArray to uniform all the three RNNs
TensorArray as a new concept is borrowed from TensorFlow,
it is meant to be used with dynamic iteration primitives such as `while_loop` and `map_fn`.
This concept can be used to support our new design of dynamic operations, and help to refactor some existing variant-sentence-related layers,
such as `RecurrentGradientMachine`.
such as `recurrent_op`, `RecurrentGradientMachine`.
In [our design for dynamic RNN](https://github.com/PaddlePaddle/Paddle/pull/4401),
`TensorArray` is used to segment inputs and store states in all time steps.
By providing some methods similar to a C++ array,
the definition of some state-based dynamic models such as RNN could be more natural and highly flexible.
## Dynamic-Related Methods
Some basic methods should be proposed as follows:
### stack()
Pack the values in a `TensorArray` into a tensor with rank one higher than each tensor in `values`.
### unstack(axis=0)
Unpacks the given dimension of a rank-`R` tensor into rank-`(R-1)` tensors.
### concat()
Return the values in the `TensorArray` as a concatenated Tensor.
### write(index, value, data_shared=true)
Write value into index of the TensorArray.
### read(index)
Read the value at location `index` in the `TensorArray`.
### size()
Return the number of values.
the definition of some state-based dynamic models such as RNN can be more natural and highly flexible.
## Dynamic-operations on TensorArray
`TensorArray` will be used directly when defining dynamic models, so some operators listed below should be implemented
```python
# several helper operators for TensorArray
def tensor_array_stack(ta, tensor):
'''
get a tensor array `ta`, return a packed `tensor`.
'''
pass
def tensor_array_unstack(tensor, ta):
'''
get a `tensor`, unstack it and get a tensor array `ta`.
get a `tensor` and a scalar tensor `index`, write `tensor` into index-th
value of the tensor array `ta`.
`data_shared` is an attribute that specifies whether to copy or reference the tensors.
'''
pass
def tensor_array_read(ta, index, tensor):
'''
get a tensor array `ta`, a scalar tensor `index`, read the index-th value of
`ta` and return as the `tensor`.
'''
pass
def tensor_array_size(ta, tensor):
'''
get a tensor array `ta`, return the size of `ta` and return as the scalar `tensor`.
'''
pass
```
It is trivial for users to use so many low-level operators, so some helper methods should be proposed in python wrapper to make `TensorArray` easier to use,
for example
```python
class TensorArray:
def __init__(self, name):
self.name = name
self.desc = TensorArrayDesc()
def stack(self, name=None):
'''
Pack the values in a `TensorArray` into a tensor with rank one higher
than each tensor in `values`.
`stack` can be used to split tensor into time steps for RNN or whileloop.
@name: str
the name of the variable to output.
'''
tensor = NewVar(name)
tensor_array_stack(self.name, tensor)
return tensor
def unstack(self, input):
'''
Unpacks the given dimension of a rank-`R` tensor into rank-`(R-1)` tensors.
`unstack` can be used to concatenate all the time steps for RNN or whileloop.
@input: str
the name of input tensor
'''
tensor_array_unstack(tensor, self.name)
def write(self, index, value, data_shared=True):
'''
Write value into index of the TensorArray.
If `data_shared` is set to True, than the index-th value in TensorArray will
Read the value at location `index` in the `TensorArray`.
@index: str
name of a scalar tensor
@output:
name of a output variable
'''
tensor_array_read(self.name, index, output)
def size(self, output):
'''
Return the number of values.
@output: str
name of a scalar tensor
'''
tensor_array_size(self.name, output)
```
## LoDTensor-related Supports
The `RecurrentGradientMachine` in Paddle serves as a flexible RNN layer; it takes variant length sequences as input,
because each step of RNN could only take a tensor-represented batch of data as input,
The `RecurrentGradientMachine` in Paddle serves as a flexible RNN layer; it takes varience-length sequences as input, and output sequences too.
Since each step of RNN can only take a tensor-represented batch of data as input,
some preprocess should be taken on the inputs such as sorting the sentences by their length in descending order and cut each word and pack to new batches.
Such cut-like operations can be embedded into `TensorArray` as general methods called `unpack` and `pack`.
Such cut-like operations can be embedded into `TensorArray` as general methods called `unpack` and `pack`,
these two operations are similar to `stack` and `unstack` except that they operate on variable-length sequences formated as a LoD tensor rather than a tensor.
Some definitions are like
```python
def unpack(level):
'''
Split LodTensor in some `level` and generate batches, if set `sort_by_length`,
will sort by length.
With these two methods, a variant-sentence-RNN can be implemented like
Returns:
- a new `TensorArray`, whose values are LodTensors and represents batches
of data.
- an int32 Tensor, which stores the map from the new batch's indices to
original LoDTensor
'''
pass
def pack(level, indices_map):
'''
Recover the original LoD-arranged LoDTensor with the values in a `TensorArray`
and `level` and `indices_map`.
'''
pass
```
With these two methods, a varience-length sentence supported RNN can be implemented like
the code above shows that by embedding the LoDTensor-related preprocess operations into `TensorArray`,
the implementation of a RNN that supports varient-length sentences is far more concise than `RecurrentGradientMachine` because the latter mixes all the codes together, hard to read and extend.
some details are as follows.
### unpack(level, sort_by_length)
Split LodTensor in some `level` and generate batches, if set `sort_by_length`, will sort by length.
Returns:
- a new `TensorArray`, whose values are LodTensors and represents batches of data.
- an int32 Tensor, which stores the map from the new batch's indices to original LoDTensor
### pack(level, indices_map)
Recover the original LoD-arranged LoDTensor with the values in a `TensorArray` and `level` and `indices_map`.
<p>IfOp should have only one branch. An IfOp operator takes a <codeclass="docutils literal"><spanclass="pre">cond</span></code> variable whose value must be a vector of N boolean elements. Its return value has N instances. If cond[i] == True, input instance input[i] will go through true_block() and generate output[i]; otherwise it will produce output from false_bloack().</p>
<p>PaddlePaddle’s <codeclass="docutils literal"><spanclass="pre">IfElse</span></code> operator differs from TensorFlow’s:</p>
<ulclass="simple">
<li>the TensorFlow version takes a scalar boolean value as the condition so that the whole mini-batch goes to either the true or the false branch, whereas</li>
<li>the PaddlePaddle version takes a vector of boolean value as the condition, and instances corresponding to true values go to the true branch, those corresponding to false values go to the false branch.</li>
<p>A challenge to implement the <codeclass="docutils literal"><spanclass="pre">IfElse</span></code> operator is to infer those variables to be split, or, say, to identify the variable of the mini-batch or those derived from the mini-batch.</p>
<p>Steps are one of the core concepts of RNN. In each time step of RNN, there should be several input segments, states, and output segments; all these components act like arrays, for example, call <codeclass="docutils literal"><spanclass="pre">states[step_id]</span></code> will get the state in <codeclass="docutils literal"><spanclass="pre">step_id</span></code>th time step.</p>
<p>An RNN can be implemented with the following pseudocode</p>
<spanclass="n">output_segments</span><spanclass="p">[</span><spanclass="n">step</span><spanclass="p">]</span><spanclass="o">=</span><spanclass="n">states</span><spanclass="p">[</span><spanclass="n">step</span><spanclass="p">]</span><spanclass="c1">// take state as output</span>
<p>According to the <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/issues/4561">RNN roadmap</a>, there are several different RNNs that PaddlePaddle will eventually support.</p>
<p>Currently, the basic RNN implementation supported by PaddlePaddle is the <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code> which takes tensors as input and splits them into <codeclass="docutils literal"><spanclass="pre">input_segments</span></code>.</p>
<p>Since a tensor cannot store variable-length sequences directly, PaddlePaddle implements the tensor with level of details (<codeclass="docutils literal"><spanclass="pre">LoDTensor</span></code> for short).
Segmenting the <codeclass="docutils literal"><spanclass="pre">LoDTensor</span></code> is much more complicated than splitting a tensor, that makes it necessary to refactor the <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code> with <codeclass="docutils literal"><spanclass="pre">LoDTensor</span></code> segmenting support.</p>
<p>As the next step in RNN support, <codeclass="docutils literal"><spanclass="pre">dynamic_recurrent_op</span></code> should be introduced to handle inputs with variable-length sequences.</p>
<p>The implementation is similar to <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code>.
The key difference is the way <strong>the original input <codeclass="docutils literal"><spanclass="pre">LoDTensors</span></code> and outupts are split to get the <codeclass="docutils literal"><spanclass="pre">input_segments</span></code> and the <codeclass="docutils literal"><spanclass="pre">output_segments</span></code>.</strong></p>
<p>Though it can’t be built over <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code> or <codeclass="docutils literal"><spanclass="pre">dynamic_recurrent_op</span></code> directly,
the logic behind splitting a tensor or a LoD tensor into <codeclass="docutils literal"><spanclass="pre">input_segments</span></code> remains the same.</p>
<p>The logic behind splitting the inputs to segments, states and outputs is similar and can be shared in a seperate module.</p>
<p>The array of <codeclass="docutils literal"><spanclass="pre">states</span></code>, <codeclass="docutils literal"><spanclass="pre">input_segments</span></code> and <codeclass="docutils literal"><spanclass="pre">output_segments</span></code> would be exposed to users when writing a dynamic RNN model similar to the above pseudo codes.</p>
<p>So there should be an array-like container, which can store the segments of a tensor or LoD tensor.</p>
<p><strong>This container can store an array of tensors and provides several methods to split a tensor or a LoD tensor</strong> .
This is where the notion of <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> comes from.</p>
<spanid="introduce-tensorarray-to-uniform-all-the-three-rnns"></span><h2>Introduce TensorArray to uniform all the three RNNs<aclass="headerlink"href="#introduce-tensorarray-to-uniform-all-the-three-rnns"title="永久链接至标题">¶</a></h2>
<p>TensorArray as a new concept is borrowed from TensorFlow,
it is meant to be used with dynamic iteration primitives such as <codeclass="docutils literal"><spanclass="pre">while_loop</span></code> and <codeclass="docutils literal"><spanclass="pre">map_fn</span></code>.</p>
<p>This concept can be used to support our new design of dynamic operations, and help to refactor some existing variant-sentence-related layers,
such as <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code>.</p>
such as <codeclass="docutils literal"><spanclass="pre">recurrent_op</span></code>, <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code>.</p>
<p>In <aclass="reference external"href="https://github.com/PaddlePaddle/Paddle/pull/4401">our design for dynamic RNN</a>,
<codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> is used to segment inputs and store states in all time steps.
By providing some methods similar to a C++ array,
the definition of some state-based dynamic models such as RNN could be more natural and highly flexible.</p>
<p>Pack the values in a <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> into a tensor with rank one higher than each tensor in <codeclass="docutils literal"><spanclass="pre">values</span></code>.</p>
<p>Unpacks the given dimension of a rank-<codeclass="docutils literal"><spanclass="pre">R</span></code> tensor into rank-<codeclass="docutils literal"><spanclass="pre">(R-1)</span></code> tensors.</p>
the definition of some state-based dynamic models such as RNN can be more natural and highly flexible.</p>
<p>Read the value at location <codeclass="docutils literal"><spanclass="pre">index</span></code> in the <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>.</p>
<spanid="dynamic-operations-on-tensorarray"></span><h2>Dynamic-operations on TensorArray<aclass="headerlink"href="#dynamic-operations-on-tensorarray"title="永久链接至标题">¶</a></h2>
<p><codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> will be used directly when defining dynamic models, so some operators listed below should be implemented</p>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="c1"># several helper operators for TensorArray</span>
<p>It is trivial for users to use so many low-level operators, so some helper methods should be proposed in python wrapper to make <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> easier to use,
<p>The <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code> in Paddle serves as a flexible RNN layer; it takes variant length sequences as input,
because each step of RNN could only take a tensor-represented batch of data as input,
<p>The <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code> in Paddle serves as a flexible RNN layer; it takes varience-length sequences as input, and output sequences too.</p>
<p>Since each step of RNN can only take a tensor-represented batch of data as input,
some preprocess should be taken on the inputs such as sorting the sentences by their length in descending order and cut each word and pack to new batches.</p>
<p>Such cut-like operations can be embedded into <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> as general methods called <codeclass="docutils literal"><spanclass="pre">unpack</span></code> and <codeclass="docutils literal"><spanclass="pre">pack</span></code>.</p>
<p>With these two methods, a variant-sentence-RNN can be implemented like</p>
<p>Such cut-like operations can be embedded into <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> as general methods called <codeclass="docutils literal"><spanclass="pre">unpack</span></code> and <codeclass="docutils literal"><spanclass="pre">pack</span></code>,
these two operations are similar to <codeclass="docutils literal"><spanclass="pre">stack</span></code> and <codeclass="docutils literal"><spanclass="pre">unstack</span></code> except that they operate on variable-length sequences formated as a LoD tensor rather than a tensor.</p>
@@ -259,20 +440,6 @@ some preprocess should be taken on the inputs such as sorting the sentences by t
</div>
<p>the code above shows that by embedding the LoDTensor-related preprocess operations into <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>,
the implementation of a RNN that supports varient-length sentences is far more concise than <codeclass="docutils literal"><spanclass="pre">RecurrentGradientMachine</span></code> because the latter mixes all the codes together, hard to read and extend.</p>
<p>Split LodTensor in some <codeclass="docutils literal"><spanclass="pre">level</span></code> and generate batches, if set <codeclass="docutils literal"><spanclass="pre">sort_by_length</span></code>, will sort by length.</p>
<p>Returns:</p>
<ulclass="simple">
<li>a new <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code>, whose values are LodTensors and represents batches of data.</li>
<li>an int32 Tensor, which stores the map from the new batch’s indices to original LoDTensor</li>
<p>Recover the original LoD-arranged LoDTensor with the values in a <codeclass="docutils literal"><spanclass="pre">TensorArray</span></code> and <codeclass="docutils literal"><spanclass="pre">level</span></code> and <codeclass="docutils literal"><spanclass="pre">indices_map</span></code>.</p>
