Skip to content

P
Paddle

PaddlePaddle / Paddle
大约 2 年 前同步成功

通知 2325
Star 20933
Fork 5424
P
Paddle
“215dbff9db14ffd6f2f4b6d4d9e6e91d0ee0cbb7”上不存在“src/operators/kernel/fpga/V1/concat_kernel.cpp”
提交 c622759a 编写于 作者: T Travis CI
浏览文件
操作

Deploy to GitHub Pages: e0e3a8a5

上级 bf265015
隐藏空白更改
内联 并排
Showing with 324 addition and 388 deletion
develop/doc/_sources/design/ops/rnn.md.txt
浏览文件 @ c622759a
# RNNOp design # RNNOp design
This document is about an RNN operator which requires that instances in a mini-batch have the same length. We will have a more flexible RNN operator. This document describes the RNN (Recurrent Neural Network) operator and how it is implemented in PaddlePaddle. The RNN op requires that all instances in a mini-batch have the same length. We will have a more flexible dynamic RNN operator in the future.
## RNN Algorithm Implementation ## RNN Algorithm Implementation
<p aligh="center"> <p align="center">
<img src="./images/rnn.jpg"/> <img src="./images/rnn.jpg"/>
</p> </p>
The above diagram shows an RNN unrolled into a full network. The above diagram shows an RNN unrolled into a full network.
There are several important concepts: There are several important concepts here:
- *step-net*: the sub-graph to run at each step, - *step-net*: the sub-graph that runs at each step.
- *memory*, $h_t$, the state of the current step, - *memory*, $h_t$, the state of the current step.
- *ex-memory*, $h_{t-1}$, the state of the previous step, - *ex-memory*, $h_{t-1}$, the state of the previous step.
- *initial memory value*, the ex-memory of the first step. - *initial memory value*, the memory of the first (initial) step.
### Step-scope ### Step-scope
There could be local variables defined in step-nets. PaddlePaddle runtime realizes these variables in *step-scopes* -- scopes created for each step. There could be local variables defined in each step-net. PaddlePaddle runtime realizes these variables in *step-scopes* which are created for each step.
<p aligh="center"> <p align="center">
<img src="./images/rnn.png"/><br/> <img src="./images/rnn.png"/><br/>
Figure 2 the RNN's data flow Figure 2 illustrates the RNN's data flow
</p> </p>
Please be aware that all steps run the same step-net. Each step Please be aware that every step runs the same step-net. Each step does the following:
1. creates the step-scope, 1. Creates the step-scope.
2. realizes local variables, including step-outputs, in the step-scope, and 2. Initializes the local variables including step-outputs, in the step-scope.
3. runs the step-net, which could use these variables. 3. Runs the step-net, which uses the above mentioned variables.
The RNN operator will compose its output from step outputs in step scopes. The RNN operator will compose its output from step outputs in each of the step scopes.
### Memory and Ex-memory ### Memory and Ex-memory
Let's give more details about memory and ex-memory via a simply example: Let's give more details about memory and ex-memory using a simple example:
$$ $$
h_t = U h_{t-1} + W x_t h_t = U h_{t-1} + W x_t
$$, $$,
where $h_t$ and $h_{t-1}$ are the memory and ex-memory of step $t$'s respectively. where $h_t$ and $h_{t-1}$ are the memory and ex-memory (previous memory) of step $t$ respectively.
In the implementation, we can make an ex-memory variable either "refers to" the memory variable of the previous step, In the implementation, we can make an ex-memory variable either "refer to" the memory variable of the previous step,
or copy the value of the previous memory value to the current ex-memory variable. or copy the memory value of the previous step to the current ex-memory variable.
### Usage in Python ### Usage in Python
For more information on Block, please refer to the [design doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md). For more information on Block, please refer to the [design doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md).
We can define an RNN's step-net using Block: We can define an RNN's step-net using a Block:
```python ```python
import paddle as pd import paddle as pd
X = some_op() # x is some operator's output, and is a LoDTensor X = some_op() # x is some operator's output and is a LoDTensor
a = some_op() a = some_op()
# declare parameters # declare parameters
...@@ -68,7 +68,7 @@ with rnn.stepnet(): ...@@ -68,7 +68,7 @@ with rnn.stepnet():
x = rnn.add_input(X) x = rnn.add_input(X)
# declare a memory (rnn's step) # declare a memory (rnn's step)
h = rnn.add_memory(init=a) h = rnn.add_memory(init=a)
# h.pre_state() means previous memory of rnn # h.pre_state(), the previous memory of rnn
new_state = pd.add_two( pd.matmul(W, x) + pd.matmul(U, h.pre_state())) new_state = pd.add_two( pd.matmul(W, x) + pd.matmul(U, h.pre_state()))
# update current memory # update current memory
h.update(new_state) h.update(new_state)
...@@ -80,19 +80,19 @@ out = rnn() ...@@ -80,19 +80,19 @@ out = rnn()
Python API functions in above example: Python API functions in above example:
- `rnn.add_input` indicates the parameter is a variable that will be segmented into step-inputs. - `rnn.add_input`: indicates that the parameter is a variable that will be segmented into step-inputs.
- `rnn.add_memory` creates a variable used as the memory. - `rnn.add_memory`: creates a variable used as the memory.
- `rnn.add_outputs` mark the variables that will be concatenated across steps into the RNN output. - `rnn.add_outputs`: marks the variables that will be concatenated across steps into the RNN output.
### Nested RNN and LoDTensor ### Nested RNN and LoDTensor
An RNN whose step-net includes other RNN operators is known as an *nested RNN*. An RNN whose step-net includes other RNN operators is known as an *nested RNN*.
For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences. For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences. Each step of the higher level RNN also receives an input from the corresponding step of the lower level, and additionally the output from the previous time step at the same level.
The following figure illustrates the feeding of text into the lower level, one sentence each step, and the feeding of step outputs to the top level. The final top level output is about the whole text. The following figure illustrates feeding in text into the lower level, one sentence at a step, and the feeding in step outputs to the top level. The final top level output is about the whole text.
<p aligh="center"> <p align="center">
<img src="./images/2_level_rnn.png"/> <img src="./images/2_level_rnn.png"/>
</p> </p>
...@@ -110,7 +110,7 @@ a = some_op() ...@@ -110,7 +110,7 @@ a = some_op()
# chapter_data is a set of 128-dim word vectors # chapter_data is a set of 128-dim word vectors
# the first level of LoD is sentence # the first level of LoD is sentence
# the second level of LoD is chapter # the second level of LoD is a chapter
chapter_data = pd.Variable(shape=[None, 128], type=pd.lod_tensor, level=2) chapter_data = pd.Variable(shape=[None, 128], type=pd.lod_tensor, level=2)
def lower_level_rnn(paragraph): def lower_level_rnn(paragraph):
...@@ -138,14 +138,14 @@ with top_level_rnn.stepnet(): ...@@ -138,14 +138,14 @@ with top_level_rnn.stepnet():
pd.matmul(W0, paragraph_data) + pd.matmul(U0, h.pre_state())) pd.matmul(W0, paragraph_data) + pd.matmul(U0, h.pre_state()))
top_level_rnn.add_outputs(h) top_level_rnn.add_outputs(h)
# just output the last step # output the last step
chapter_out = top_level_rnn(output_all_steps=False) chapter_out = top_level_rnn(output_all_steps=False)
``` ```
in above example, the construction of the `top_level_rnn` calls `lower_level_rnn`. The input is a LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences. In the above example, the construction of the `top_level_rnn` calls `lower_level_rnn`. The input is an LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.
By default, the `RNNOp` will concatenate the outputs from all the time steps, By default, the `RNNOp` will concatenate the outputs from all the time steps.
if the `output_all_steps` set to False, it will only output the final time step. If the `output_all_steps` is set to False, it will only output the final time step.
<p align="center"> <p align="center">
......
develop/doc/_sources/design/ops/sequence_decoder.md.txt
浏览文件 @ c622759a
# Design: Sequence Decoder Generating LoDTensors # Design: Sequence Decoder Generating LoDTensors
In tasks such as machine translation and image to text, In tasks such as machine translation and visual captioning,
a [sequence decoder](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md) is necessary to generate sequences. a [sequence decoder](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md) is necessary to generate sequences, one word at a time.
This documentation describes how to implement the sequence decoder as an operator. This documentation describes how to implement the sequence decoder as an operator.
## Beam Search based Decoder ## Beam Search based Decoder
The [beam search algorithm](https://en.wikipedia.org/wiki/Beam_search) is necessary when generating sequences, The [beam search algorithm](https://en.wikipedia.org/wiki/Beam_search) is necessary when generating sequences. It is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.
it is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.
In the old version of PaddlePaddle, a C++ class `RecurrentGradientMachine` implements the general sequence decoder based on beam search, In the old version of PaddlePaddle, the C++ class `RecurrentGradientMachine` implements the general sequence decoder based on beam search, due to the complexity involved, the implementation relies on a lot of special data structures that are quite trivial and hard to be customized by users.
due to the complexity, the implementation relays on a lot of special data structures,
quite trivial and hard to be customized by users.
There are a lot of heuristic tricks in the sequence generation tasks, There are a lot of heuristic tricks in the sequence generation tasks, so the flexibility of sequence decoder is very important to users.
so the flexibility of sequence decoder is very important to users.
During PaddlePaddle's refactoring work, During the refactoring of PaddlePaddle, some new concepts are proposed such as: [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md) and [TensorArray](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md) that can better support the sequence usage, and they can also help make the implementation of beam search based sequence decoder **more transparent and modular** .
some new concept is proposed such as [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md) and [TensorArray](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md) that can better support sequence usage,
and they can help to make the implementation of beam search based sequence decoder **more transparent and modular** .
For example, the RNN sates, candidates IDs and probabilities of beam search can be represented as `LoDTensors`; For example, the RNN states, candidates IDs and probabilities of beam search can be represented all as `LoDTensors`;
the selected candidate's IDs in each time step can be stored in a `TensorArray`, and `Packed` to the sentences translated. the selected candidate's IDs in each time step can be stored in a `TensorArray`, and `Packed` to the sentences translated.
## Changing LoD's absolute offset to relative offsets ## Changing LoD's absolute offset to relative offsets
The current `LoDTensor` is designed to store levels of variable-length sequences, The current `LoDTensor` is designed to store levels of variable-length sequences. It stores several arrays of integers where each represents a level.
it stores several arrays of integers each represents a level.
The integers in each level represents the begin and end (not inclusive) offset of a sequence **in the underlying tensor**, The integers in each level represent the begin and end (not inclusive) offset of a sequence **in the underlying tensor**,
let's call this format the **absolute-offset LoD** for clear. let's call this format the **absolute-offset LoD** for clarity.
The relative-offset LoD can fast retrieve any sequence but fails to represent empty sequences, for example, a two-level LoD is as follows The relative-offset LoD can retrieve any sequence very quickly but fails to represent empty sequences, for example, a two-level LoD is as follows
```python ```python
[[0, 3, 9] [[0, 3, 9]
[0, 2, 3, 3, 3, 9]] [0, 2, 3, 3, 3, 9]]
...@@ -41,10 +34,9 @@ The first level tells that there are two sequences: ...@@ -41,10 +34,9 @@ The first level tells that there are two sequences:
while on the second level, there are several empty sequences that both begin and end at `3`. while on the second level, there are several empty sequences that both begin and end at `3`.
It is impossible to tell how many empty second-level sequences exist in the first-level sequences. It is impossible to tell how many empty second-level sequences exist in the first-level sequences.
There are many scenarios that relay on empty sequence representation, There are many scenarios that rely on empty sequence representation, for example in machine translation or visual captioning, one instance has no translation or the empty candidate set for a prefix.
such as machine translation or image to text, one instance has no translations or the empty candidate set for a prefix.
So let's introduce another format of LoD, So let's introduce another format of LoD,
it stores **the offsets of the lower level sequences** and is called **relative-offset** LoD. it stores **the offsets of the lower level sequences** and is called **relative-offset** LoD.
For example, to represent the same sequences of the above data For example, to represent the same sequences of the above data
...@@ -54,19 +46,18 @@ For example, to represent the same sequences of the above data ...@@ -54,19 +46,18 @@ For example, to represent the same sequences of the above data
[0, 2, 3, 3, 3, 9]] [0, 2, 3, 3, 3, 9]]
``` ```
the first level represents that there are two sequences, the first level represents that there are two sequences,
their offsets in the second-level LoD is `[0, 3)` and `[3, 5)`. their offsets in the second-level LoD is `[0, 3)` and `[3, 5)`.
The second level is the same with the relative offset example because the lower level is a tensor. The second level is the same with the relative offset example because the lower level is a tensor.
It is easy to find out the second sequence in the first-level LoD has two empty sequences. It is easy to find out the second sequence in the first-level LoD has two empty sequences.
The following demos are based on relative-offset LoD. The following examples are based on relative-offset LoD.
## Usage in a simple machine translation model ## Usage in a simple machine translation model
Let's start from a simple machine translation model that is simplified from [machine translation chapter](https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation) to draw a simple blueprint of what a sequence decoder can do and how to use it. Let's start from a simple machine translation model that is simplified from the [machine translation chapter](https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation) to draw a blueprint of what a sequence decoder can do and how to use it.
The model has an encoder that learns the semantic vector from a sequence, The model has an encoder that learns the semantic vector from a sequence, and a decoder which uses the sequence encoder to generate new sentences.
and a decoder which uses the sequence decoder to generate new sentences.
**Encoder** **Encoder**
```python ```python
...@@ -117,7 +108,7 @@ def generate(): ...@@ -117,7 +108,7 @@ def generate():
# which means there are 2 sentences to translate # which means there are 2 sentences to translate
# - the first sentence has 1 translation prefixes, the offsets are [0, 1) # - the first sentence has 1 translation prefixes, the offsets are [0, 1)
# - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6) # - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6)
# the target_word.lod is # the target_word.lod is
# [[0, 1, 6] # [[0, 1, 6]
# [0, 2, 4, 7, 9 12]] # [0, 2, 4, 7, 9 12]]
# which means 2 sentences to translate, each has 1 and 5 prefixes # which means 2 sentences to translate, each has 1 and 5 prefixes
...@@ -154,37 +145,36 @@ def generate(): ...@@ -154,37 +145,36 @@ def generate():
translation_ids, translation_scores = decoder() translation_ids, translation_scores = decoder()
``` ```
The `decoder.beam_search` is a operator that given the candidates and the scores of translations including the candidates, The `decoder.beam_search` is an operator that, given the candidates and the scores of translations including the candidates,
return the result of the beam search algorithm. returns the result of the beam search algorithm.
In this way, users can customize anything on the inputs or outputs of beam search, for example, two ways to prune some translation prefixes In this way, users can customize anything on the input or output of beam search, for example:
1. meke the correspondind elements in `topk_generated_scores` zero or some small values, beam_search will discard this candidate. 1. Make the corresponding elements in `topk_generated_scores` zero or some small values, beam_search will discard this candidate.
2. remove some specific candidate in `selected_ids` 2. Remove some specific candidate in `selected_ids`.
3. get the final `translation_ids`, remove the translation sequence in it. 3. Get the final `translation_ids`, remove the translation sequence in it.
The implementation of sequence decoder can reuse the C++ class [RNNAlgorithm](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30), The implementation of sequence decoder can reuse the C++ class: [RNNAlgorithm](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30),
so the python syntax is quite similar to a [RNN](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop). so the python syntax is quite similar to that of an [RNN](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop).
Both of them are two-level `LoDTensors` Both of them are two-level `LoDTensors`:
- the first level represents `batch_size` of (source) sentences; - The first level represents `batch_size` of (source) sentences.
- the second level represents the candidate ID sets for translation prefix. - The second level represents the candidate ID sets for translation prefix.
for example, 3 source sentences to translate, and has 2, 3, 1 candidates. For example, 3 source sentences to translate, and has 2, 3, 1 candidates.
Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, and an `lod_expand` operator is used to expand the LoD of the previous state to fit the current state.
a `lod_expand` operator is used to expand the LoD of the previous state to fit the current state.
For example, the previous state For example, the previous state:
* LoD is `[0, 1, 3][0, 2, 5, 6]` * LoD is `[0, 1, 3][0, 2, 5, 6]`
* content of tensor is `a1 a2 b1 b2 b3 c1` * content of tensor is `a1 a2 b1 b2 b3 c1`
the current state stored in `encoder_ctx_expanded` the current state is stored in `encoder_ctx_expanded`:
* LoD is `[0, 2, 7][0 3 5 8 9 11 11]` * LoD is `[0, 2, 7][0 3 5 8 9 11 11]`
* the content is * the content is
- a1 a1 a1 (a1 has 3 candidates, so the state should be copied 3 times for each candidates) - a1 a1 a1 (a1 has 3 candidates, so the state should be copied 3 times for each candidates)
- a2 a2 - a2 a2
- b1 b1 b1 - b1 b1 b1
...@@ -192,54 +182,48 @@ the current state stored in `encoder_ctx_expanded` ...@@ -192,54 +182,48 @@ the current state stored in `encoder_ctx_expanded`
- b3 b3 - b3 b3
- None (c1 has 0 candidates, so c1 is dropped) - None (c1 has 0 candidates, so c1 is dropped)
Benefit from the relative offset LoD, empty candidate set can be represented naturally. The benefit from the relative offset LoD is that the empty candidate set can be represented naturally.
the status in each time step can be stored in `TensorArray`, and `Pack`ed to a final LoDTensor, the corresponding syntax is The status in each time step can be stored in `TensorArray`, and `Pack`ed to a final LoDTensor. The corresponding syntax is:
```python ```python
decoder.output(selected_ids) decoder.output(selected_ids)
decoder.output(selected_generation_scores) decoder.output(selected_generation_scores)
``` ```
the `selected_ids` is the candidate ids for the prefixes, The `selected_ids` are the candidate ids for the prefixes, and will be `Packed` by `TensorArray` to a two-level `LoDTensor`, where the first level represents the source sequences and the second level represents generated sequences.
it will be `Packed` by `TensorArray` to a two-level `LoDTensor`,
the first level represents the source sequences,
the second level represents generated sequences.
Pack the `selected_scores` will get a `LoDTensor` that stores scores of each candidate of translations. Packing the `selected_scores` will get a `LoDTensor` that stores scores of each translation candidate.
Pack the `selected_generation_scores` will get a `LoDTensor`, and each tail is the probability of the translation. Packing the `selected_generation_scores` will get a `LoDTensor`, and each tail is the probability of the translation.
## LoD and shape changes during decoding ## LoD and shape changes during decoding
<p align="center"> <p align="center">
<img src="./images/LOD-and-shape-changes-during-decoding.jpg"/> <img src="./images/LOD-and-shape-changes-during-decoding.jpg"/>
</p> </p>
According the image above, the only phrase to change LoD is beam search. According to the image above, the only phase that changes the LoD is beam search.
## Beam search design ## Beam search design
The beam search algorthm will be implemented as one method of the sequence decoder, it has 3 inputs The beam search algorithm will be implemented as one method of the sequence decoder and has 3 inputs:
1. `topk_ids`, top K candidate ids for each prefix. 1. `topk_ids`, the top K candidate ids for each prefix.
2. `topk_scores`, the corresponding scores for `topk_ids` 2. `topk_scores`, the corresponding scores for `topk_ids`
3. `generated_scores`, the score of the prefixes. 3. `generated_scores`, the score of the prefixes.
All of the are LoDTensors, so that the sequence affilication is clear. All of these are LoDTensors, so that the sequence affiliation is clear. Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.
Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.
It will return three variables It will return three variables:
1. `selected_ids`, the final candidate beam search function selected for the next step. 1. `selected_ids`, the final candidate beam search function selected for the next step.
2. `selected_scores`, the scores for the candidates. 2. `selected_scores`, the scores for the candidates.
3. `generated_scores`, the updated scores for each prefixes (with the new candidates appended). 3. `generated_scores`, the updated scores for each prefix (with the new candidates appended).
## Introducing the LoD-based `Pack` and `Unpack` methods in `TensorArray` ## Introducing the LoD-based `Pack` and `Unpack` methods in `TensorArray`
The `selected_ids`, `selected_scores` and `generated_scores` are LoDTensors, The `selected_ids`, `selected_scores` and `generated_scores` are LoDTensors that exist at each time step,
and they exist in each time step,
so it is natural to store them in arrays. so it is natural to store them in arrays.
Currently, PaddlePaddle has a module called `TensorArray` which can store an array of tensors, Currently, PaddlePaddle has a module called `TensorArray` which can store an array of tensors. It is better to store the results of beam search in a `TensorArray`.
the results of beam search are better to store in a `TensorArray`.
The `Pack` and `UnPack` in `TensorArray` are used to package tensors in the array to a `LoDTensor` or split the `LoDTensor` to an array of tensors. The `Pack` and `UnPack` in `TensorArray` are used to pack tensors in the array to an `LoDTensor` or split the `LoDTensor` to an array of tensors.
It needs some extensions to support pack or unpack an array of `LoDTensors`. It needs some extensions to support the packing or unpacking an array of `LoDTensors`.
develop/doc/design/ops/rnn.html
浏览文件 @ c622759a
...@@ -190,50 +190,50 @@ ...@@ -190,50 +190,50 @@
<div class="section" id="rnnop-design"> <div class="section" id="rnnop-design">
<span id="rnnop-design"></span><h1>RNNOp design<a class="headerlink" href="#rnnop-design" title="Permalink to this headline"></a></h1> <span id="rnnop-design"></span><h1>RNNOp design<a class="headerlink" href="#rnnop-design" title="Permalink to this headline"></a></h1>
<p>This document is about an RNN operator which requires that instances in a mini-batch have the same length. We will have a more flexible RNN operator.</p> <p>This document describes the RNN (Recurrent Neural Network) operator and how it is implemented in PaddlePaddle. The RNN op requires that all instances in a mini-batch have the same length. We will have a more flexible dynamic RNN operator in the future.</p>
<div class="section" id="rnn-algorithm-implementation"> <div class="section" id="rnn-algorithm-implementation">
<span id="rnn-algorithm-implementation"></span><h2>RNN Algorithm Implementation<a class="headerlink" href="#rnn-algorithm-implementation" title="Permalink to this headline"></a></h2> <span id="rnn-algorithm-implementation"></span><h2>RNN Algorithm Implementation<a class="headerlink" href="#rnn-algorithm-implementation" title="Permalink to this headline"></a></h2>
<p aligh="center"> <p align="center">
<img src="./images/rnn.jpg"/> <img src="./images/rnn.jpg"/>
</p><p>The above diagram shows an RNN unrolled into a full network.</p> </p><p>The above diagram shows an RNN unrolled into a full network.</p>
<p>There are several important concepts:</p> <p>There are several important concepts here:</p>
<ul class="simple"> <ul class="simple">
<li><em>step-net</em>: the sub-graph to run at each step,</li> <li><em>step-net</em>: the sub-graph that runs at each step.</li>
<li><em>memory</em>, $h_t$, the state of the current step,</li> <li><em>memory</em>, $h_t$, the state of the current step.</li>
<li><em>ex-memory</em>, $h_{t-1}$, the state of the previous step,</li> <li><em>ex-memory</em>, $h_{t-1}$, the state of the previous step.</li>
<li><em>initial memory value</em>, the ex-memory of the first step.</li> <li><em>initial memory value</em>, the memory of the first (initial) step.</li>
</ul> </ul>
<div class="section" id="step-scope"> <div class="section" id="step-scope">
<span id="step-scope"></span><h3>Step-scope<a class="headerlink" href="#step-scope" title="Permalink to this headline"></a></h3> <span id="step-scope"></span><h3>Step-scope<a class="headerlink" href="#step-scope" title="Permalink to this headline"></a></h3>
<p>There could be local variables defined in step-nets. PaddlePaddle runtime realizes these variables in <em>step-scopes</em> &#8211; scopes created for each step.</p> <p>There could be local variables defined in each step-net. PaddlePaddle runtime realizes these variables in <em>step-scopes</em> which are created for each step.</p>
<p aligh="center"> <p align="center">
<img src="./images/rnn.png"/><br/> <img src="./images/rnn.png"/><br/>
Figure 2 the RNN's data flow Figure 2 illustrates the RNN's data flow
</p><p>Please be aware that all steps run the same step-net. Each step</p> </p><p>Please be aware that every step runs the same step-net. Each step does the following:</p>
<ol class="simple"> <ol class="simple">
<li>creates the step-scope,</li> <li>Creates the step-scope.</li>
<li>realizes local variables, including step-outputs, in the step-scope, and</li> <li>Initializes the local variables including step-outputs, in the step-scope.</li>
<li>runs the step-net, which could use these variables.</li> <li>Runs the step-net, which uses the above mentioned variables.</li>
</ol> </ol>
<p>The RNN operator will compose its output from step outputs in step scopes.</p> <p>The RNN operator will compose its output from step outputs in each of the step scopes.</p>
</div> </div>
<div class="section" id="memory-and-ex-memory"> <div class="section" id="memory-and-ex-memory">
<span id="memory-and-ex-memory"></span><h3>Memory and Ex-memory<a class="headerlink" href="#memory-and-ex-memory" title="Permalink to this headline"></a></h3> <span id="memory-and-ex-memory"></span><h3>Memory and Ex-memory<a class="headerlink" href="#memory-and-ex-memory" title="Permalink to this headline"></a></h3>
<p>Let&#8217;s give more details about memory and ex-memory via a simply example:</p> <p>Let&#8217;s give more details about memory and ex-memory using a simple example:</p>
<p>$$ <p>$$
h_t = U h_{t-1} + W x_t h_t = U h_{t-1} + W x_t
$$,</p> $$,</p>
<p>where $h_t$ and $h_{t-1}$ are the memory and ex-memory of step $t$&#8217;s respectively.</p> <p>where $h_t$ and $h_{t-1}$ are the memory and ex-memory (previous memory) of step $t$ respectively.</p>
<p>In the implementation, we can make an ex-memory variable either &#8220;refers to&#8221; the memory variable of the previous step, <p>In the implementation, we can make an ex-memory variable either &#8220;refer to&#8221; the memory variable of the previous step,
or copy the value of the previous memory value to the current ex-memory variable.</p> or copy the memory value of the previous step to the current ex-memory variable.</p>
</div> </div>
<div class="section" id="usage-in-python"> <div class="section" id="usage-in-python">
<span id="usage-in-python"></span><h3>Usage in Python<a class="headerlink" href="#usage-in-python" title="Permalink to this headline"></a></h3> <span id="usage-in-python"></span><h3>Usage in Python<a class="headerlink" href="#usage-in-python" title="Permalink to this headline"></a></h3>
<p>For more information on Block, please refer to the <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md">design doc</a>.</p> <p>For more information on Block, please refer to the <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md">design doc</a>.</p>
<p>We can define an RNN&#8217;s step-net using Block:</p> <p>We can define an RNN&#8217;s step-net using a Block:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span> <span class="c1"># x is some operator&#39;s output, and is a LoDTensor</span> <span class="n">X</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span> <span class="c1"># x is some operator&#39;s output and is a LoDTensor</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span> <span class="n">a</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span>
<span class="c1"># declare parameters</span> <span class="c1"># declare parameters</span>
...@@ -245,7 +245,7 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -245,7 +245,7 @@ or copy the value of the previous memory value to the current ex-memory variable
<span class="n">x</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_input</span><span class="p">(</span><span class="n">X</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_input</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># declare a memory (rnn&#39;s step)</span> <span class="c1"># declare a memory (rnn&#39;s step)</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_memory</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="n">a</span><span class="p">)</span> <span class="n">h</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_memory</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="c1"># h.pre_state() means previous memory of rnn</span> <span class="c1"># h.pre_state(), the previous memory of rnn</span>
<span class="n">new_state</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">add_two</span><span class="p">(</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span> <span class="n">new_state</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">add_two</span><span class="p">(</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span>
<span class="c1"># update current memory</span> <span class="c1"># update current memory</span>
<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">new_state</span><span class="p">)</span> <span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">new_state</span><span class="p">)</span>
...@@ -257,17 +257,17 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -257,17 +257,17 @@ or copy the value of the previous memory value to the current ex-memory variable
</div> </div>
<p>Python API functions in above example:</p> <p>Python API functions in above example:</p>
<ul class="simple"> <ul class="simple">
<li><code class="docutils literal"><span class="pre">rnn.add_input</span></code> indicates the parameter is a variable that will be segmented into step-inputs.</li> <li><code class="docutils literal"><span class="pre">rnn.add_input</span></code>: indicates that the parameter is a variable that will be segmented into step-inputs.</li>
<li><code class="docutils literal"><span class="pre">rnn.add_memory</span></code> creates a variable used as the memory.</li> <li><code class="docutils literal"><span class="pre">rnn.add_memory</span></code>: creates a variable used as the memory.</li>
<li><code class="docutils literal"><span class="pre">rnn.add_outputs</span></code> mark the variables that will be concatenated across steps into the RNN output.</li> <li><code class="docutils literal"><span class="pre">rnn.add_outputs</span></code>: marks the variables that will be concatenated across steps into the RNN output.</li>
</ul> </ul>
</div> </div>
<div class="section" id="nested-rnn-and-lodtensor"> <div class="section" id="nested-rnn-and-lodtensor">
<span id="nested-rnn-and-lodtensor"></span><h3>Nested RNN and LoDTensor<a class="headerlink" href="#nested-rnn-and-lodtensor" title="Permalink to this headline"></a></h3> <span id="nested-rnn-and-lodtensor"></span><h3>Nested RNN and LoDTensor<a class="headerlink" href="#nested-rnn-and-lodtensor" title="Permalink to this headline"></a></h3>
<p>An RNN whose step-net includes other RNN operators is known as an <em>nested RNN</em>.</p> <p>An RNN whose step-net includes other RNN operators is known as an <em>nested RNN</em>.</p>
<p>For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences.</p> <p>For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences. Each step of the higher level RNN also receives an input from the corresponding step of the lower level, and additionally the output from the previous time step at the same level.</p>
<p>The following figure illustrates the feeding of text into the lower level, one sentence each step, and the feeding of step outputs to the top level. The final top level output is about the whole text.</p> <p>The following figure illustrates feeding in text into the lower level, one sentence at a step, and the feeding in step outputs to the top level. The final top level output is about the whole text.</p>
<p aligh="center"> <p align="center">
<img src="./images/2_level_rnn.png"/> <img src="./images/2_level_rnn.png"/>
</p><div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span> </p><div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span>
...@@ -282,7 +282,7 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -282,7 +282,7 @@ or copy the value of the previous memory value to the current ex-memory variable
<span class="c1"># chapter_data is a set of 128-dim word vectors</span> <span class="c1"># chapter_data is a set of 128-dim word vectors</span>
<span class="c1"># the first level of LoD is sentence</span> <span class="c1"># the first level of LoD is sentence</span>
<span class="c1"># the second level of LoD is chapter</span> <span class="c1"># the second level of LoD is a chapter</span>
<span class="n">chapter_data</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="nb">type</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">lod_tensor</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">chapter_data</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="nb">type</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">lod_tensor</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">lower_level_rnn</span><span class="p">(</span><span class="n">paragraph</span><span class="p">):</span> <span class="k">def</span> <span class="nf">lower_level_rnn</span><span class="p">(</span><span class="n">paragraph</span><span class="p">):</span>
...@@ -310,13 +310,13 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -310,13 +310,13 @@ or copy the value of the previous memory value to the current ex-memory variable
<span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W0</span><span class="p">,</span> <span class="n">paragraph_data</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U0</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W0</span><span class="p">,</span> <span class="n">paragraph_data</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U0</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span>
<span class="n">top_level_rnn</span><span class="o">.</span><span class="n">add_outputs</span><span class="p">(</span><span class="n">h</span><span class="p">)</span> <span class="n">top_level_rnn</span><span class="o">.</span><span class="n">add_outputs</span><span class="p">(</span><span class="n">h</span><span class="p">)</span>
<span class="c1"># just output the last step</span> <span class="c1"># output the last step</span>
<span class="n">chapter_out</span> <span class="o">=</span> <span class="n">top_level_rnn</span><span class="p">(</span><span class="n">output_all_steps</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="n">chapter_out</span> <span class="o">=</span> <span class="n">top_level_rnn</span><span class="p">(</span><span class="n">output_all_steps</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div> </pre></div>
</div> </div>
<p>in above example, the construction of the <code class="docutils literal"><span class="pre">top_level_rnn</span></code> calls <code class="docutils literal"><span class="pre">lower_level_rnn</span></code>. The input is a LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.</p> <p>In the above example, the construction of the <code class="docutils literal"><span class="pre">top_level_rnn</span></code> calls <code class="docutils literal"><span class="pre">lower_level_rnn</span></code>. The input is an LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.</p>
<p>By default, the <code class="docutils literal"><span class="pre">RNNOp</span></code> will concatenate the outputs from all the time steps, <p>By default, the <code class="docutils literal"><span class="pre">RNNOp</span></code> will concatenate the outputs from all the time steps.
if the <code class="docutils literal"><span class="pre">output_all_steps</span></code> set to False, it will only output the final time step.</p> If the <code class="docutils literal"><span class="pre">output_all_steps</span></code> is set to False, it will only output the final time step.</p>
<p align="center"> <p align="center">
<img src="images/rnn_2level_data.png"/> <img src="images/rnn_2level_data.png"/>
</p></div> </p></div>
......
develop/doc/design/ops/sequence_decoder.html
浏览文件 @ c622759a
...@@ -190,31 +190,24 @@ ...@@ -190,31 +190,24 @@
<div class="section" id="design-sequence-decoder-generating-lodtensors"> <div class="section" id="design-sequence-decoder-generating-lodtensors">
<span id="design-sequence-decoder-generating-lodtensors"></span><h1>Design: Sequence Decoder Generating LoDTensors<a class="headerlink" href="#design-sequence-decoder-generating-lodtensors" title="Permalink to this headline"></a></h1> <span id="design-sequence-decoder-generating-lodtensors"></span><h1>Design: Sequence Decoder Generating LoDTensors<a class="headerlink" href="#design-sequence-decoder-generating-lodtensors" title="Permalink to this headline"></a></h1>
<p>In tasks such as machine translation and image to text, <p>In tasks such as machine translation and visual captioning,
a <a class="reference external" href="https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md">sequence decoder</a> is necessary to generate sequences.</p> a <a class="reference external" href="https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md">sequence decoder</a> is necessary to generate sequences, one word at a time.</p>
<p>This documentation describes how to implement the sequence decoder as an operator.</p> <p>This documentation describes how to implement the sequence decoder as an operator.</p>
<div class="section" id="beam-search-based-decoder"> <div class="section" id="beam-search-based-decoder">
<span id="beam-search-based-decoder"></span><h2>Beam Search based Decoder<a class="headerlink" href="#beam-search-based-decoder" title="Permalink to this headline"></a></h2> <span id="beam-search-based-decoder"></span><h2>Beam Search based Decoder<a class="headerlink" href="#beam-search-based-decoder" title="Permalink to this headline"></a></h2>
<p>The <a class="reference external" href="https://en.wikipedia.org/wiki/Beam_search">beam search algorithm</a> is necessary when generating sequences, <p>The <a class="reference external" href="https://en.wikipedia.org/wiki/Beam_search">beam search algorithm</a> is necessary when generating sequences. It is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.</p>
it is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.</p> <p>In the old version of PaddlePaddle, the C++ class <code class="docutils literal"><span class="pre">RecurrentGradientMachine</span></code> implements the general sequence decoder based on beam search, due to the complexity involved, the implementation relies on a lot of special data structures that are quite trivial and hard to be customized by users.</p>
<p>In the old version of PaddlePaddle, a C++ class <code class="docutils literal"><span class="pre">RecurrentGradientMachine</span></code> implements the general sequence decoder based on beam search, <p>There are a lot of heuristic tricks in the sequence generation tasks, so the flexibility of sequence decoder is very important to users.</p>
due to the complexity, the implementation relays on a lot of special data structures, <p>During the refactoring of PaddlePaddle, some new concepts are proposed such as: <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md">LoDTensor</a> and <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md">TensorArray</a> that can better support the sequence usage, and they can also help make the implementation of beam search based sequence decoder <strong>more transparent and modular</strong> .</p>
quite trivial and hard to be customized by users.</p> <p>For example, the RNN states, candidates IDs and probabilities of beam search can be represented all as <code class="docutils literal"><span class="pre">LoDTensors</span></code>;
<p>There are a lot of heuristic tricks in the sequence generation tasks,
so the flexibility of sequence decoder is very important to users.</p>
<p>During PaddlePaddle&#8217;s refactoring work,
some new concept is proposed such as <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md">LoDTensor</a> and <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md">TensorArray</a> that can better support sequence usage,
and they can help to make the implementation of beam search based sequence decoder <strong>more transparent and modular</strong> .</p>
<p>For example, the RNN sates, candidates IDs and probabilities of beam search can be represented as <code class="docutils literal"><span class="pre">LoDTensors</span></code>;
the selected candidate&#8217;s IDs in each time step can be stored in a <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Packed</span></code> to the sentences translated.</p> the selected candidate&#8217;s IDs in each time step can be stored in a <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Packed</span></code> to the sentences translated.</p>
</div> </div>
<div class="section" id="changing-lod-s-absolute-offset-to-relative-offsets"> <div class="section" id="changing-lod-s-absolute-offset-to-relative-offsets">
<span id="changing-lod-s-absolute-offset-to-relative-offsets"></span><h2>Changing LoD&#8217;s absolute offset to relative offsets<a class="headerlink" href="#changing-lod-s-absolute-offset-to-relative-offsets" title="Permalink to this headline"></a></h2> <span id="changing-lod-s-absolute-offset-to-relative-offsets"></span><h2>Changing LoD&#8217;s absolute offset to relative offsets<a class="headerlink" href="#changing-lod-s-absolute-offset-to-relative-offsets" title="Permalink to this headline"></a></h2>
<p>The current <code class="docutils literal"><span class="pre">LoDTensor</span></code> is designed to store levels of variable-length sequences, <p>The current <code class="docutils literal"><span class="pre">LoDTensor</span></code> is designed to store levels of variable-length sequences. It stores several arrays of integers where each represents a level.</p>
it stores several arrays of integers each represents a level.</p> <p>The integers in each level represent the begin and end (not inclusive) offset of a sequence <strong>in the underlying tensor</strong>,
<p>The integers in each level represents the begin and end (not inclusive) offset of a sequence <strong>in the underlying tensor</strong>, let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clarity.</p>
let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clear.</p> <p>The relative-offset LoD can retrieve any sequence very quickly but fails to represent empty sequences, for example, a two-level LoD is as follows</p>
<p>The relative-offset LoD can fast retrieve any sequence but fails to represent empty sequences, for example, a two-level LoD is as follows</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]]</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]]</span>
</pre></div> </pre></div>
...@@ -226,8 +219,7 @@ let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clear. ...@@ -226,8 +219,7 @@ let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clear.
</ul> </ul>
<p>while on the second level, there are several empty sequences that both begin and end at <code class="docutils literal"><span class="pre">3</span></code>. <p>while on the second level, there are several empty sequences that both begin and end at <code class="docutils literal"><span class="pre">3</span></code>.
It is impossible to tell how many empty second-level sequences exist in the first-level sequences.</p> It is impossible to tell how many empty second-level sequences exist in the first-level sequences.</p>
<p>There are many scenarios that relay on empty sequence representation, <p>There are many scenarios that rely on empty sequence representation, for example in machine translation or visual captioning, one instance has no translation or the empty candidate set for a prefix.</p>
such as machine translation or image to text, one instance has no translations or the empty candidate set for a prefix.</p>
<p>So let&#8217;s introduce another format of LoD, <p>So let&#8217;s introduce another format of LoD,
it stores <strong>the offsets of the lower level sequences</strong> and is called <strong>relative-offset</strong> LoD.</p> it stores <strong>the offsets of the lower level sequences</strong> and is called <strong>relative-offset</strong> LoD.</p>
<p>For example, to represent the same sequences of the above data</p> <p>For example, to represent the same sequences of the above data</p>
...@@ -239,13 +231,12 @@ it stores <strong>the offsets of the lower level sequences</strong> and is calle ...@@ -239,13 +231,12 @@ it stores <strong>the offsets of the lower level sequences</strong> and is calle
their offsets in the second-level LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">3)</span></code> and <code class="docutils literal"><span class="pre">[3,</span> <span class="pre">5)</span></code>.</p> their offsets in the second-level LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">3)</span></code> and <code class="docutils literal"><span class="pre">[3,</span> <span class="pre">5)</span></code>.</p>
<p>The second level is the same with the relative offset example because the lower level is a tensor. <p>The second level is the same with the relative offset example because the lower level is a tensor.
It is easy to find out the second sequence in the first-level LoD has two empty sequences.</p> It is easy to find out the second sequence in the first-level LoD has two empty sequences.</p>
<p>The following demos are based on relative-offset LoD.</p> <p>The following examples are based on relative-offset LoD.</p>
</div> </div>
<div class="section" id="usage-in-a-simple-machine-translation-model"> <div class="section" id="usage-in-a-simple-machine-translation-model">
<span id="usage-in-a-simple-machine-translation-model"></span><h2>Usage in a simple machine translation model<a class="headerlink" href="#usage-in-a-simple-machine-translation-model" title="Permalink to this headline"></a></h2> <span id="usage-in-a-simple-machine-translation-model"></span><h2>Usage in a simple machine translation model<a class="headerlink" href="#usage-in-a-simple-machine-translation-model" title="Permalink to this headline"></a></h2>
<p>Let&#8217;s start from a simple machine translation model that is simplified from <a class="reference external" href="https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation">machine translation chapter</a> to draw a simple blueprint of what a sequence decoder can do and how to use it.</p> <p>Let&#8217;s start from a simple machine translation model that is simplified from the <a class="reference external" href="https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation">machine translation chapter</a> to draw a blueprint of what a sequence decoder can do and how to use it.</p>
<p>The model has an encoder that learns the semantic vector from a sequence, <p>The model has an encoder that learns the semantic vector from a sequence, and a decoder which uses the sequence encoder to generate new sentences.</p>
and a decoder which uses the sequence decoder to generate new sentences.</p>
<p><strong>Encoder</strong></p> <p><strong>Encoder</strong></p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span>
...@@ -292,7 +283,7 @@ and a decoder which uses the sequence decoder to generate new sentences.</p> ...@@ -292,7 +283,7 @@ and a decoder which uses the sequence decoder to generate new sentences.</p>
<span class="c1"># which means there are 2 sentences to translate</span> <span class="c1"># which means there are 2 sentences to translate</span>
<span class="c1"># - the first sentence has 1 translation prefixes, the offsets are [0, 1)</span> <span class="c1"># - the first sentence has 1 translation prefixes, the offsets are [0, 1)</span>
<span class="c1"># - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6)</span> <span class="c1"># - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6)</span>
<span class="c1"># the target_word.lod is </span> <span class="c1"># the target_word.lod is</span>
<span class="c1"># [[0, 1, 6]</span> <span class="c1"># [[0, 1, 6]</span>
<span class="c1"># [0, 2, 4, 7, 9 12]]</span> <span class="c1"># [0, 2, 4, 7, 9 12]]</span>
<span class="c1"># which means 2 sentences to translate, each has 1 and 5 prefixes</span> <span class="c1"># which means 2 sentences to translate, each has 1 and 5 prefixes</span>
...@@ -330,30 +321,29 @@ and a decoder which uses the sequence decoder to generate new sentences.</p> ...@@ -330,30 +321,29 @@ and a decoder which uses the sequence decoder to generate new sentences.</p>
<span class="n">translation_ids</span><span class="p">,</span> <span class="n">translation_scores</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">()</span> <span class="n">translation_ids</span><span class="p">,</span> <span class="n">translation_scores</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">()</span>
</pre></div> </pre></div>
</div> </div>
<p>The <code class="docutils literal"><span class="pre">decoder.beam_search</span></code> is a operator that given the candidates and the scores of translations including the candidates, <p>The <code class="docutils literal"><span class="pre">decoder.beam_search</span></code> is an operator that, given the candidates and the scores of translations including the candidates,
return the result of the beam search algorithm.</p> returns the result of the beam search algorithm.</p>
<p>In this way, users can customize anything on the inputs or outputs of beam search, for example, two ways to prune some translation prefixes</p> <p>In this way, users can customize anything on the input or output of beam search, for example:</p>
<ol class="simple"> <ol class="simple">
<li>meke the correspondind elements in <code class="docutils literal"><span class="pre">topk_generated_scores</span></code> zero or some small values, beam_search will discard this candidate.</li> <li>Make the corresponding elements in <code class="docutils literal"><span class="pre">topk_generated_scores</span></code> zero or some small values, beam_search will discard this candidate.</li>
<li>remove some specific candidate in <code class="docutils literal"><span class="pre">selected_ids</span></code></li> <li>Remove some specific candidate in <code class="docutils literal"><span class="pre">selected_ids</span></code>.</li>
<li>get the final <code class="docutils literal"><span class="pre">translation_ids</span></code>, remove the translation sequence in it.</li> <li>Get the final <code class="docutils literal"><span class="pre">translation_ids</span></code>, remove the translation sequence in it.</li>
</ol> </ol>
<p>The implementation of sequence decoder can reuse the C++ class <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30">RNNAlgorithm</a>, <p>The implementation of sequence decoder can reuse the C++ class: <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30">RNNAlgorithm</a>,
so the python syntax is quite similar to a <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop">RNN</a>.</p> so the python syntax is quite similar to that of an <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop">RNN</a>.</p>
<p>Both of them are two-level <code class="docutils literal"><span class="pre">LoDTensors</span></code></p> <p>Both of them are two-level <code class="docutils literal"><span class="pre">LoDTensors</span></code>:</p>
<ul class="simple"> <ul class="simple">
<li>the first level represents <code class="docutils literal"><span class="pre">batch_size</span></code> of (source) sentences;</li> <li>The first level represents <code class="docutils literal"><span class="pre">batch_size</span></code> of (source) sentences.</li>
<li>the second level represents the candidate ID sets for translation prefix.</li> <li>The second level represents the candidate ID sets for translation prefix.</li>
</ul> </ul>
<p>for example, 3 source sentences to translate, and has 2, 3, 1 candidates.</p> <p>For example, 3 source sentences to translate, and has 2, 3, 1 candidates.</p>
<p>Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, <p>Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, and an <code class="docutils literal"><span class="pre">lod_expand</span></code> operator is used to expand the LoD of the previous state to fit the current state.</p>
a <code class="docutils literal"><span class="pre">lod_expand</span></code> operator is used to expand the LoD of the previous state to fit the current state.</p> <p>For example, the previous state:</p>
<p>For example, the previous state</p>
<ul class="simple"> <ul class="simple">
<li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">1,</span> <span class="pre">3][0,</span> <span class="pre">2,</span> <span class="pre">5,</span> <span class="pre">6]</span></code></li> <li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">1,</span> <span class="pre">3][0,</span> <span class="pre">2,</span> <span class="pre">5,</span> <span class="pre">6]</span></code></li>
<li>content of tensor is <code class="docutils literal"><span class="pre">a1</span> <span class="pre">a2</span> <span class="pre">b1</span> <span class="pre">b2</span> <span class="pre">b3</span> <span class="pre">c1</span></code></li> <li>content of tensor is <code class="docutils literal"><span class="pre">a1</span> <span class="pre">a2</span> <span class="pre">b1</span> <span class="pre">b2</span> <span class="pre">b3</span> <span class="pre">c1</span></code></li>
</ul> </ul>
<p>the current state stored in <code class="docutils literal"><span class="pre">encoder_ctx_expanded</span></code></p> <p>the current state is stored in <code class="docutils literal"><span class="pre">encoder_ctx_expanded</span></code>:</p>
<ul class="simple"> <ul class="simple">
<li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">2,</span> <span class="pre">7][0</span> <span class="pre">3</span> <span class="pre">5</span> <span class="pre">8</span> <span class="pre">9</span> <span class="pre">11</span> <span class="pre">11]</span></code></li> <li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">2,</span> <span class="pre">7][0</span> <span class="pre">3</span> <span class="pre">5</span> <span class="pre">8</span> <span class="pre">9</span> <span class="pre">11</span> <span class="pre">11]</span></code></li>
<li>the content is<ul> <li>the content is<ul>
...@@ -366,51 +356,45 @@ a <code class="docutils literal"><span class="pre">lod_expand</span></code> oper ...@@ -366,51 +356,45 @@ a <code class="docutils literal"><span class="pre">lod_expand</span></code> oper
</ul> </ul>
</li> </li>
</ul> </ul>
<p>Benefit from the relative offset LoD, empty candidate set can be represented naturally.</p> <p>The benefit from the relative offset LoD is that the empty candidate set can be represented naturally.</p>
<p>the status in each time step can be stored in <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Pack</span></code>ed to a final LoDTensor, the corresponding syntax is</p> <p>The status in each time step can be stored in <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Pack</span></code>ed to a final LoDTensor. The corresponding syntax is:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_ids</span><span class="p">)</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_ids</span><span class="p">)</span>
<span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_generation_scores</span><span class="p">)</span> <span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_generation_scores</span><span class="p">)</span>
</pre></div> </pre></div>
</div> </div>
<p>the <code class="docutils literal"><span class="pre">selected_ids</span></code> is the candidate ids for the prefixes, <p>The <code class="docutils literal"><span class="pre">selected_ids</span></code> are the candidate ids for the prefixes, and will be <code class="docutils literal"><span class="pre">Packed</span></code> by <code class="docutils literal"><span class="pre">TensorArray</span></code> to a two-level <code class="docutils literal"><span class="pre">LoDTensor</span></code>, where the first level represents the source sequences and the second level represents generated sequences.</p>
it will be <code class="docutils literal"><span class="pre">Packed</span></code> by <code class="docutils literal"><span class="pre">TensorArray</span></code> to a two-level <code class="docutils literal"><span class="pre">LoDTensor</span></code>, <p>Packing the <code class="docutils literal"><span class="pre">selected_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code> that stores scores of each translation candidate.</p>
the first level represents the source sequences, <p>Packing the <code class="docutils literal"><span class="pre">selected_generation_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code>, and each tail is the probability of the translation.</p>
the second level represents generated sequences.</p>
<p>Pack the <code class="docutils literal"><span class="pre">selected_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code> that stores scores of each candidate of translations.</p>
<p>Pack the <code class="docutils literal"><span class="pre">selected_generation_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code>, and each tail is the probability of the translation.</p>
</div> </div>
<div class="section" id="lod-and-shape-changes-during-decoding"> <div class="section" id="lod-and-shape-changes-during-decoding">
<span id="lod-and-shape-changes-during-decoding"></span><h2>LoD and shape changes during decoding<a class="headerlink" href="#lod-and-shape-changes-during-decoding" title="Permalink to this headline"></a></h2> <span id="lod-and-shape-changes-during-decoding"></span><h2>LoD and shape changes during decoding<a class="headerlink" href="#lod-and-shape-changes-during-decoding" title="Permalink to this headline"></a></h2>
<p align="center"> <p align="center">
<img src="./images/LOD-and-shape-changes-during-decoding.jpg"/> <img src="./images/LOD-and-shape-changes-during-decoding.jpg"/>
</p><p>According the image above, the only phrase to change LoD is beam search.</p> </p><p>According to the image above, the only phase that changes the LoD is beam search.</p>
</div> </div>
<div class="section" id="beam-search-design"> <div class="section" id="beam-search-design">
<span id="beam-search-design"></span><h2>Beam search design<a class="headerlink" href="#beam-search-design" title="Permalink to this headline"></a></h2> <span id="beam-search-design"></span><h2>Beam search design<a class="headerlink" href="#beam-search-design" title="Permalink to this headline"></a></h2>
<p>The beam search algorthm will be implemented as one method of the sequence decoder, it has 3 inputs</p> <p>The beam search algorithm will be implemented as one method of the sequence decoder and has 3 inputs:</p>
<ol class="simple"> <ol class="simple">
<li><code class="docutils literal"><span class="pre">topk_ids</span></code>, top K candidate ids for each prefix.</li> <li><code class="docutils literal"><span class="pre">topk_ids</span></code>, the top K candidate ids for each prefix.</li>
<li><code class="docutils literal"><span class="pre">topk_scores</span></code>, the corresponding scores for <code class="docutils literal"><span class="pre">topk_ids</span></code></li> <li><code class="docutils literal"><span class="pre">topk_scores</span></code>, the corresponding scores for <code class="docutils literal"><span class="pre">topk_ids</span></code></li>
<li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the score of the prefixes.</li> <li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the score of the prefixes.</li>
</ol> </ol>
<p>All of the are LoDTensors, so that the sequence affilication is clear. <p>All of these are LoDTensors, so that the sequence affiliation is clear. Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.</p>
Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.</p> <p>It will return three variables:</p>
<p>It will return three variables</p>
<ol class="simple"> <ol class="simple">
<li><code class="docutils literal"><span class="pre">selected_ids</span></code>, the final candidate beam search function selected for the next step.</li> <li><code class="docutils literal"><span class="pre">selected_ids</span></code>, the final candidate beam search function selected for the next step.</li>
<li><code class="docutils literal"><span class="pre">selected_scores</span></code>, the scores for the candidates.</li> <li><code class="docutils literal"><span class="pre">selected_scores</span></code>, the scores for the candidates.</li>
<li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the updated scores for each prefixes (with the new candidates appended).</li> <li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the updated scores for each prefix (with the new candidates appended).</li>
</ol> </ol>
</div> </div>
<div class="section" id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray"> <div class="section" id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray">
<span id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray"></span><h2>Introducing the LoD-based <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">Unpack</span></code> methods in <code class="docutils literal"><span class="pre">TensorArray</span></code><a class="headerlink" href="#introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray" title="Permalink to this headline"></a></h2> <span id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray"></span><h2>Introducing the LoD-based <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">Unpack</span></code> methods in <code class="docutils literal"><span class="pre">TensorArray</span></code><a class="headerlink" href="#introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray" title="Permalink to this headline"></a></h2>
<p>The <code class="docutils literal"><span class="pre">selected_ids</span></code>, <code class="docutils literal"><span class="pre">selected_scores</span></code> and <code class="docutils literal"><span class="pre">generated_scores</span></code> are LoDTensors, <p>The <code class="docutils literal"><span class="pre">selected_ids</span></code>, <code class="docutils literal"><span class="pre">selected_scores</span></code> and <code class="docutils literal"><span class="pre">generated_scores</span></code> are LoDTensors that exist at each time step,
and they exist in each time step,
so it is natural to store them in arrays.</p> so it is natural to store them in arrays.</p>
<p>Currently, PaddlePaddle has a module called <code class="docutils literal"><span class="pre">TensorArray</span></code> which can store an array of tensors, <p>Currently, PaddlePaddle has a module called <code class="docutils literal"><span class="pre">TensorArray</span></code> which can store an array of tensors. It is better to store the results of beam search in a <code class="docutils literal"><span class="pre">TensorArray</span></code>.</p>
the results of beam search are better to store in a <code class="docutils literal"><span class="pre">TensorArray</span></code>.</p> <p>The <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">UnPack</span></code> in <code class="docutils literal"><span class="pre">TensorArray</span></code> are used to pack tensors in the array to an <code class="docutils literal"><span class="pre">LoDTensor</span></code> or split the <code class="docutils literal"><span class="pre">LoDTensor</span></code> to an array of tensors.
<p>The <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">UnPack</span></code> in <code class="docutils literal"><span class="pre">TensorArray</span></code> are used to package tensors in the array to a <code class="docutils literal"><span class="pre">LoDTensor</span></code> or split the <code class="docutils literal"><span class="pre">LoDTensor</span></code> to an array of tensors. It needs some extensions to support the packing or unpacking an array of <code class="docutils literal"><span class="pre">LoDTensors</span></code>.</p>
It needs some extensions to support pack or unpack an array of <code class="docutils literal"><span class="pre">LoDTensors</span></code>.</p>
</div> </div>
</div> </div>
......
develop/doc/searchindex.js
浏览文件 @ c622759a
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
develop/doc_cn/_sources/design/ops/rnn.md.txt
浏览文件 @ c622759a
# RNNOp design # RNNOp design
This document is about an RNN operator which requires that instances in a mini-batch have the same length. We will have a more flexible RNN operator. This document describes the RNN (Recurrent Neural Network) operator and how it is implemented in PaddlePaddle. The RNN op requires that all instances in a mini-batch have the same length. We will have a more flexible dynamic RNN operator in the future.
## RNN Algorithm Implementation ## RNN Algorithm Implementation
<p aligh="center"> <p align="center">
<img src="./images/rnn.jpg"/> <img src="./images/rnn.jpg"/>
</p> </p>
The above diagram shows an RNN unrolled into a full network. The above diagram shows an RNN unrolled into a full network.
There are several important concepts: There are several important concepts here:
- *step-net*: the sub-graph to run at each step, - *step-net*: the sub-graph that runs at each step.
- *memory*, $h_t$, the state of the current step, - *memory*, $h_t$, the state of the current step.
- *ex-memory*, $h_{t-1}$, the state of the previous step, - *ex-memory*, $h_{t-1}$, the state of the previous step.
- *initial memory value*, the ex-memory of the first step. - *initial memory value*, the memory of the first (initial) step.
### Step-scope ### Step-scope
There could be local variables defined in step-nets. PaddlePaddle runtime realizes these variables in *step-scopes* -- scopes created for each step. There could be local variables defined in each step-net. PaddlePaddle runtime realizes these variables in *step-scopes* which are created for each step.
<p aligh="center"> <p align="center">
<img src="./images/rnn.png"/><br/> <img src="./images/rnn.png"/><br/>
Figure 2 the RNN's data flow Figure 2 illustrates the RNN's data flow
</p> </p>
Please be aware that all steps run the same step-net. Each step Please be aware that every step runs the same step-net. Each step does the following:
1. creates the step-scope, 1. Creates the step-scope.
2. realizes local variables, including step-outputs, in the step-scope, and 2. Initializes the local variables including step-outputs, in the step-scope.
3. runs the step-net, which could use these variables. 3. Runs the step-net, which uses the above mentioned variables.
The RNN operator will compose its output from step outputs in step scopes. The RNN operator will compose its output from step outputs in each of the step scopes.
### Memory and Ex-memory ### Memory and Ex-memory
Let's give more details about memory and ex-memory via a simply example: Let's give more details about memory and ex-memory using a simple example:
$$ $$
h_t = U h_{t-1} + W x_t h_t = U h_{t-1} + W x_t
$$, $$,
where $h_t$ and $h_{t-1}$ are the memory and ex-memory of step $t$'s respectively. where $h_t$ and $h_{t-1}$ are the memory and ex-memory (previous memory) of step $t$ respectively.
In the implementation, we can make an ex-memory variable either "refers to" the memory variable of the previous step, In the implementation, we can make an ex-memory variable either "refer to" the memory variable of the previous step,
or copy the value of the previous memory value to the current ex-memory variable. or copy the memory value of the previous step to the current ex-memory variable.
### Usage in Python ### Usage in Python
For more information on Block, please refer to the [design doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md). For more information on Block, please refer to the [design doc](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md).
We can define an RNN's step-net using Block: We can define an RNN's step-net using a Block:
```python ```python
import paddle as pd import paddle as pd
X = some_op() # x is some operator's output, and is a LoDTensor X = some_op() # x is some operator's output and is a LoDTensor
a = some_op() a = some_op()
# declare parameters # declare parameters
...@@ -68,7 +68,7 @@ with rnn.stepnet(): ...@@ -68,7 +68,7 @@ with rnn.stepnet():
x = rnn.add_input(X) x = rnn.add_input(X)
# declare a memory (rnn's step) # declare a memory (rnn's step)
h = rnn.add_memory(init=a) h = rnn.add_memory(init=a)
# h.pre_state() means previous memory of rnn # h.pre_state(), the previous memory of rnn
new_state = pd.add_two( pd.matmul(W, x) + pd.matmul(U, h.pre_state())) new_state = pd.add_two( pd.matmul(W, x) + pd.matmul(U, h.pre_state()))
# update current memory # update current memory
h.update(new_state) h.update(new_state)
...@@ -80,19 +80,19 @@ out = rnn() ...@@ -80,19 +80,19 @@ out = rnn()
Python API functions in above example: Python API functions in above example:
- `rnn.add_input` indicates the parameter is a variable that will be segmented into step-inputs. - `rnn.add_input`: indicates that the parameter is a variable that will be segmented into step-inputs.
- `rnn.add_memory` creates a variable used as the memory. - `rnn.add_memory`: creates a variable used as the memory.
- `rnn.add_outputs` mark the variables that will be concatenated across steps into the RNN output. - `rnn.add_outputs`: marks the variables that will be concatenated across steps into the RNN output.
### Nested RNN and LoDTensor ### Nested RNN and LoDTensor
An RNN whose step-net includes other RNN operators is known as an *nested RNN*. An RNN whose step-net includes other RNN operators is known as an *nested RNN*.
For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences. For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences. Each step of the higher level RNN also receives an input from the corresponding step of the lower level, and additionally the output from the previous time step at the same level.
The following figure illustrates the feeding of text into the lower level, one sentence each step, and the feeding of step outputs to the top level. The final top level output is about the whole text. The following figure illustrates feeding in text into the lower level, one sentence at a step, and the feeding in step outputs to the top level. The final top level output is about the whole text.
<p aligh="center"> <p align="center">
<img src="./images/2_level_rnn.png"/> <img src="./images/2_level_rnn.png"/>
</p> </p>
...@@ -110,7 +110,7 @@ a = some_op() ...@@ -110,7 +110,7 @@ a = some_op()
# chapter_data is a set of 128-dim word vectors # chapter_data is a set of 128-dim word vectors
# the first level of LoD is sentence # the first level of LoD is sentence
# the second level of LoD is chapter # the second level of LoD is a chapter
chapter_data = pd.Variable(shape=[None, 128], type=pd.lod_tensor, level=2) chapter_data = pd.Variable(shape=[None, 128], type=pd.lod_tensor, level=2)
def lower_level_rnn(paragraph): def lower_level_rnn(paragraph):
...@@ -138,14 +138,14 @@ with top_level_rnn.stepnet(): ...@@ -138,14 +138,14 @@ with top_level_rnn.stepnet():
pd.matmul(W0, paragraph_data) + pd.matmul(U0, h.pre_state())) pd.matmul(W0, paragraph_data) + pd.matmul(U0, h.pre_state()))
top_level_rnn.add_outputs(h) top_level_rnn.add_outputs(h)
# just output the last step # output the last step
chapter_out = top_level_rnn(output_all_steps=False) chapter_out = top_level_rnn(output_all_steps=False)
``` ```
in above example, the construction of the `top_level_rnn` calls `lower_level_rnn`. The input is a LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences. In the above example, the construction of the `top_level_rnn` calls `lower_level_rnn`. The input is an LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.
By default, the `RNNOp` will concatenate the outputs from all the time steps, By default, the `RNNOp` will concatenate the outputs from all the time steps.
if the `output_all_steps` set to False, it will only output the final time step. If the `output_all_steps` is set to False, it will only output the final time step.
<p align="center"> <p align="center">
......
develop/doc_cn/_sources/design/ops/sequence_decoder.md.txt
浏览文件 @ c622759a
# Design: Sequence Decoder Generating LoDTensors # Design: Sequence Decoder Generating LoDTensors
In tasks such as machine translation and image to text, In tasks such as machine translation and visual captioning,
a [sequence decoder](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md) is necessary to generate sequences. a [sequence decoder](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md) is necessary to generate sequences, one word at a time.
This documentation describes how to implement the sequence decoder as an operator. This documentation describes how to implement the sequence decoder as an operator.
## Beam Search based Decoder ## Beam Search based Decoder
The [beam search algorithm](https://en.wikipedia.org/wiki/Beam_search) is necessary when generating sequences, The [beam search algorithm](https://en.wikipedia.org/wiki/Beam_search) is necessary when generating sequences. It is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.
it is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.
In the old version of PaddlePaddle, a C++ class `RecurrentGradientMachine` implements the general sequence decoder based on beam search, In the old version of PaddlePaddle, the C++ class `RecurrentGradientMachine` implements the general sequence decoder based on beam search, due to the complexity involved, the implementation relies on a lot of special data structures that are quite trivial and hard to be customized by users.
due to the complexity, the implementation relays on a lot of special data structures,
quite trivial and hard to be customized by users.
There are a lot of heuristic tricks in the sequence generation tasks, There are a lot of heuristic tricks in the sequence generation tasks, so the flexibility of sequence decoder is very important to users.
so the flexibility of sequence decoder is very important to users.
During PaddlePaddle's refactoring work, During the refactoring of PaddlePaddle, some new concepts are proposed such as: [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md) and [TensorArray](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md) that can better support the sequence usage, and they can also help make the implementation of beam search based sequence decoder **more transparent and modular** .
some new concept is proposed such as [LoDTensor](https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md) and [TensorArray](https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md) that can better support sequence usage,
and they can help to make the implementation of beam search based sequence decoder **more transparent and modular** .
For example, the RNN sates, candidates IDs and probabilities of beam search can be represented as `LoDTensors`; For example, the RNN states, candidates IDs and probabilities of beam search can be represented all as `LoDTensors`;
the selected candidate's IDs in each time step can be stored in a `TensorArray`, and `Packed` to the sentences translated. the selected candidate's IDs in each time step can be stored in a `TensorArray`, and `Packed` to the sentences translated.
## Changing LoD's absolute offset to relative offsets ## Changing LoD's absolute offset to relative offsets
The current `LoDTensor` is designed to store levels of variable-length sequences, The current `LoDTensor` is designed to store levels of variable-length sequences. It stores several arrays of integers where each represents a level.
it stores several arrays of integers each represents a level.
The integers in each level represents the begin and end (not inclusive) offset of a sequence **in the underlying tensor**, The integers in each level represent the begin and end (not inclusive) offset of a sequence **in the underlying tensor**,
let's call this format the **absolute-offset LoD** for clear. let's call this format the **absolute-offset LoD** for clarity.
The relative-offset LoD can fast retrieve any sequence but fails to represent empty sequences, for example, a two-level LoD is as follows The relative-offset LoD can retrieve any sequence very quickly but fails to represent empty sequences, for example, a two-level LoD is as follows
```python ```python
[[0, 3, 9] [[0, 3, 9]
[0, 2, 3, 3, 3, 9]] [0, 2, 3, 3, 3, 9]]
...@@ -41,10 +34,9 @@ The first level tells that there are two sequences: ...@@ -41,10 +34,9 @@ The first level tells that there are two sequences:
while on the second level, there are several empty sequences that both begin and end at `3`. while on the second level, there are several empty sequences that both begin and end at `3`.
It is impossible to tell how many empty second-level sequences exist in the first-level sequences. It is impossible to tell how many empty second-level sequences exist in the first-level sequences.
There are many scenarios that relay on empty sequence representation, There are many scenarios that rely on empty sequence representation, for example in machine translation or visual captioning, one instance has no translation or the empty candidate set for a prefix.
such as machine translation or image to text, one instance has no translations or the empty candidate set for a prefix.
So let's introduce another format of LoD, So let's introduce another format of LoD,
it stores **the offsets of the lower level sequences** and is called **relative-offset** LoD. it stores **the offsets of the lower level sequences** and is called **relative-offset** LoD.
For example, to represent the same sequences of the above data For example, to represent the same sequences of the above data
...@@ -54,19 +46,18 @@ For example, to represent the same sequences of the above data ...@@ -54,19 +46,18 @@ For example, to represent the same sequences of the above data
[0, 2, 3, 3, 3, 9]] [0, 2, 3, 3, 3, 9]]
``` ```
the first level represents that there are two sequences, the first level represents that there are two sequences,
their offsets in the second-level LoD is `[0, 3)` and `[3, 5)`. their offsets in the second-level LoD is `[0, 3)` and `[3, 5)`.
The second level is the same with the relative offset example because the lower level is a tensor. The second level is the same with the relative offset example because the lower level is a tensor.
It is easy to find out the second sequence in the first-level LoD has two empty sequences. It is easy to find out the second sequence in the first-level LoD has two empty sequences.
The following demos are based on relative-offset LoD. The following examples are based on relative-offset LoD.
## Usage in a simple machine translation model ## Usage in a simple machine translation model
Let's start from a simple machine translation model that is simplified from [machine translation chapter](https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation) to draw a simple blueprint of what a sequence decoder can do and how to use it. Let's start from a simple machine translation model that is simplified from the [machine translation chapter](https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation) to draw a blueprint of what a sequence decoder can do and how to use it.
The model has an encoder that learns the semantic vector from a sequence, The model has an encoder that learns the semantic vector from a sequence, and a decoder which uses the sequence encoder to generate new sentences.
and a decoder which uses the sequence decoder to generate new sentences.
**Encoder** **Encoder**
```python ```python
...@@ -117,7 +108,7 @@ def generate(): ...@@ -117,7 +108,7 @@ def generate():
# which means there are 2 sentences to translate # which means there are 2 sentences to translate
# - the first sentence has 1 translation prefixes, the offsets are [0, 1) # - the first sentence has 1 translation prefixes, the offsets are [0, 1)
# - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6) # - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6)
# the target_word.lod is # the target_word.lod is
# [[0, 1, 6] # [[0, 1, 6]
# [0, 2, 4, 7, 9 12]] # [0, 2, 4, 7, 9 12]]
# which means 2 sentences to translate, each has 1 and 5 prefixes # which means 2 sentences to translate, each has 1 and 5 prefixes
...@@ -154,37 +145,36 @@ def generate(): ...@@ -154,37 +145,36 @@ def generate():
translation_ids, translation_scores = decoder() translation_ids, translation_scores = decoder()
``` ```
The `decoder.beam_search` is a operator that given the candidates and the scores of translations including the candidates, The `decoder.beam_search` is an operator that, given the candidates and the scores of translations including the candidates,
return the result of the beam search algorithm. returns the result of the beam search algorithm.
In this way, users can customize anything on the inputs or outputs of beam search, for example, two ways to prune some translation prefixes In this way, users can customize anything on the input or output of beam search, for example:
1. meke the correspondind elements in `topk_generated_scores` zero or some small values, beam_search will discard this candidate. 1. Make the corresponding elements in `topk_generated_scores` zero or some small values, beam_search will discard this candidate.
2. remove some specific candidate in `selected_ids` 2. Remove some specific candidate in `selected_ids`.
3. get the final `translation_ids`, remove the translation sequence in it. 3. Get the final `translation_ids`, remove the translation sequence in it.
The implementation of sequence decoder can reuse the C++ class [RNNAlgorithm](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30), The implementation of sequence decoder can reuse the C++ class: [RNNAlgorithm](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30),
so the python syntax is quite similar to a [RNN](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop). so the python syntax is quite similar to that of an [RNN](https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop).
Both of them are two-level `LoDTensors` Both of them are two-level `LoDTensors`:
- the first level represents `batch_size` of (source) sentences; - The first level represents `batch_size` of (source) sentences.
- the second level represents the candidate ID sets for translation prefix. - The second level represents the candidate ID sets for translation prefix.
for example, 3 source sentences to translate, and has 2, 3, 1 candidates. For example, 3 source sentences to translate, and has 2, 3, 1 candidates.
Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, and an `lod_expand` operator is used to expand the LoD of the previous state to fit the current state.
a `lod_expand` operator is used to expand the LoD of the previous state to fit the current state.
For example, the previous state For example, the previous state:
* LoD is `[0, 1, 3][0, 2, 5, 6]` * LoD is `[0, 1, 3][0, 2, 5, 6]`
* content of tensor is `a1 a2 b1 b2 b3 c1` * content of tensor is `a1 a2 b1 b2 b3 c1`
the current state stored in `encoder_ctx_expanded` the current state is stored in `encoder_ctx_expanded`:
* LoD is `[0, 2, 7][0 3 5 8 9 11 11]` * LoD is `[0, 2, 7][0 3 5 8 9 11 11]`
* the content is * the content is
- a1 a1 a1 (a1 has 3 candidates, so the state should be copied 3 times for each candidates) - a1 a1 a1 (a1 has 3 candidates, so the state should be copied 3 times for each candidates)
- a2 a2 - a2 a2
- b1 b1 b1 - b1 b1 b1
...@@ -192,54 +182,48 @@ the current state stored in `encoder_ctx_expanded` ...@@ -192,54 +182,48 @@ the current state stored in `encoder_ctx_expanded`
- b3 b3 - b3 b3
- None (c1 has 0 candidates, so c1 is dropped) - None (c1 has 0 candidates, so c1 is dropped)
Benefit from the relative offset LoD, empty candidate set can be represented naturally. The benefit from the relative offset LoD is that the empty candidate set can be represented naturally.
the status in each time step can be stored in `TensorArray`, and `Pack`ed to a final LoDTensor, the corresponding syntax is The status in each time step can be stored in `TensorArray`, and `Pack`ed to a final LoDTensor. The corresponding syntax is:
```python ```python
decoder.output(selected_ids) decoder.output(selected_ids)
decoder.output(selected_generation_scores) decoder.output(selected_generation_scores)
``` ```
the `selected_ids` is the candidate ids for the prefixes, The `selected_ids` are the candidate ids for the prefixes, and will be `Packed` by `TensorArray` to a two-level `LoDTensor`, where the first level represents the source sequences and the second level represents generated sequences.
it will be `Packed` by `TensorArray` to a two-level `LoDTensor`,
the first level represents the source sequences,
the second level represents generated sequences.
Pack the `selected_scores` will get a `LoDTensor` that stores scores of each candidate of translations. Packing the `selected_scores` will get a `LoDTensor` that stores scores of each translation candidate.
Pack the `selected_generation_scores` will get a `LoDTensor`, and each tail is the probability of the translation. Packing the `selected_generation_scores` will get a `LoDTensor`, and each tail is the probability of the translation.
## LoD and shape changes during decoding ## LoD and shape changes during decoding
<p align="center"> <p align="center">
<img src="./images/LOD-and-shape-changes-during-decoding.jpg"/> <img src="./images/LOD-and-shape-changes-during-decoding.jpg"/>
</p> </p>
According the image above, the only phrase to change LoD is beam search. According to the image above, the only phase that changes the LoD is beam search.
## Beam search design ## Beam search design
The beam search algorthm will be implemented as one method of the sequence decoder, it has 3 inputs The beam search algorithm will be implemented as one method of the sequence decoder and has 3 inputs:
1. `topk_ids`, top K candidate ids for each prefix. 1. `topk_ids`, the top K candidate ids for each prefix.
2. `topk_scores`, the corresponding scores for `topk_ids` 2. `topk_scores`, the corresponding scores for `topk_ids`
3. `generated_scores`, the score of the prefixes. 3. `generated_scores`, the score of the prefixes.
All of the are LoDTensors, so that the sequence affilication is clear. All of these are LoDTensors, so that the sequence affiliation is clear. Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.
Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.
It will return three variables It will return three variables:
1. `selected_ids`, the final candidate beam search function selected for the next step. 1. `selected_ids`, the final candidate beam search function selected for the next step.
2. `selected_scores`, the scores for the candidates. 2. `selected_scores`, the scores for the candidates.
3. `generated_scores`, the updated scores for each prefixes (with the new candidates appended). 3. `generated_scores`, the updated scores for each prefix (with the new candidates appended).
## Introducing the LoD-based `Pack` and `Unpack` methods in `TensorArray` ## Introducing the LoD-based `Pack` and `Unpack` methods in `TensorArray`
The `selected_ids`, `selected_scores` and `generated_scores` are LoDTensors, The `selected_ids`, `selected_scores` and `generated_scores` are LoDTensors that exist at each time step,
and they exist in each time step,
so it is natural to store them in arrays. so it is natural to store them in arrays.
Currently, PaddlePaddle has a module called `TensorArray` which can store an array of tensors, Currently, PaddlePaddle has a module called `TensorArray` which can store an array of tensors. It is better to store the results of beam search in a `TensorArray`.
the results of beam search are better to store in a `TensorArray`.
The `Pack` and `UnPack` in `TensorArray` are used to package tensors in the array to a `LoDTensor` or split the `LoDTensor` to an array of tensors. The `Pack` and `UnPack` in `TensorArray` are used to pack tensors in the array to an `LoDTensor` or split the `LoDTensor` to an array of tensors.
It needs some extensions to support pack or unpack an array of `LoDTensors`. It needs some extensions to support the packing or unpacking an array of `LoDTensors`.
develop/doc_cn/design/ops/rnn.html
浏览文件 @ c622759a
...@@ -204,50 +204,50 @@ ...@@ -204,50 +204,50 @@
<div class="section" id="rnnop-design"> <div class="section" id="rnnop-design">
<span id="rnnop-design"></span><h1>RNNOp design<a class="headerlink" href="#rnnop-design" title="永久链接至标题"></a></h1> <span id="rnnop-design"></span><h1>RNNOp design<a class="headerlink" href="#rnnop-design" title="永久链接至标题"></a></h1>
<p>This document is about an RNN operator which requires that instances in a mini-batch have the same length. We will have a more flexible RNN operator.</p> <p>This document describes the RNN (Recurrent Neural Network) operator and how it is implemented in PaddlePaddle. The RNN op requires that all instances in a mini-batch have the same length. We will have a more flexible dynamic RNN operator in the future.</p>
<div class="section" id="rnn-algorithm-implementation"> <div class="section" id="rnn-algorithm-implementation">
<span id="rnn-algorithm-implementation"></span><h2>RNN Algorithm Implementation<a class="headerlink" href="#rnn-algorithm-implementation" title="永久链接至标题"></a></h2> <span id="rnn-algorithm-implementation"></span><h2>RNN Algorithm Implementation<a class="headerlink" href="#rnn-algorithm-implementation" title="永久链接至标题"></a></h2>
<p aligh="center"> <p align="center">
<img src="./images/rnn.jpg"/> <img src="./images/rnn.jpg"/>
</p><p>The above diagram shows an RNN unrolled into a full network.</p> </p><p>The above diagram shows an RNN unrolled into a full network.</p>
<p>There are several important concepts:</p> <p>There are several important concepts here:</p>
<ul class="simple"> <ul class="simple">
<li><em>step-net</em>: the sub-graph to run at each step,</li> <li><em>step-net</em>: the sub-graph that runs at each step.</li>
<li><em>memory</em>, $h_t$, the state of the current step,</li> <li><em>memory</em>, $h_t$, the state of the current step.</li>
<li><em>ex-memory</em>, $h_{t-1}$, the state of the previous step,</li> <li><em>ex-memory</em>, $h_{t-1}$, the state of the previous step.</li>
<li><em>initial memory value</em>, the ex-memory of the first step.</li> <li><em>initial memory value</em>, the memory of the first (initial) step.</li>
</ul> </ul>
<div class="section" id="step-scope"> <div class="section" id="step-scope">
<span id="step-scope"></span><h3>Step-scope<a class="headerlink" href="#step-scope" title="永久链接至标题"></a></h3> <span id="step-scope"></span><h3>Step-scope<a class="headerlink" href="#step-scope" title="永久链接至标题"></a></h3>
<p>There could be local variables defined in step-nets. PaddlePaddle runtime realizes these variables in <em>step-scopes</em> &#8211; scopes created for each step.</p> <p>There could be local variables defined in each step-net. PaddlePaddle runtime realizes these variables in <em>step-scopes</em> which are created for each step.</p>
<p aligh="center"> <p align="center">
<img src="./images/rnn.png"/><br/> <img src="./images/rnn.png"/><br/>
Figure 2 the RNN's data flow Figure 2 illustrates the RNN's data flow
</p><p>Please be aware that all steps run the same step-net. Each step</p> </p><p>Please be aware that every step runs the same step-net. Each step does the following:</p>
<ol class="simple"> <ol class="simple">
<li>creates the step-scope,</li> <li>Creates the step-scope.</li>
<li>realizes local variables, including step-outputs, in the step-scope, and</li> <li>Initializes the local variables including step-outputs, in the step-scope.</li>
<li>runs the step-net, which could use these variables.</li> <li>Runs the step-net, which uses the above mentioned variables.</li>
</ol> </ol>
<p>The RNN operator will compose its output from step outputs in step scopes.</p> <p>The RNN operator will compose its output from step outputs in each of the step scopes.</p>
</div> </div>
<div class="section" id="memory-and-ex-memory"> <div class="section" id="memory-and-ex-memory">
<span id="memory-and-ex-memory"></span><h3>Memory and Ex-memory<a class="headerlink" href="#memory-and-ex-memory" title="永久链接至标题"></a></h3> <span id="memory-and-ex-memory"></span><h3>Memory and Ex-memory<a class="headerlink" href="#memory-and-ex-memory" title="永久链接至标题"></a></h3>
<p>Let&#8217;s give more details about memory and ex-memory via a simply example:</p> <p>Let&#8217;s give more details about memory and ex-memory using a simple example:</p>
<p>$$ <p>$$
h_t = U h_{t-1} + W x_t h_t = U h_{t-1} + W x_t
$$,</p> $$,</p>
<p>where $h_t$ and $h_{t-1}$ are the memory and ex-memory of step $t$&#8217;s respectively.</p> <p>where $h_t$ and $h_{t-1}$ are the memory and ex-memory (previous memory) of step $t$ respectively.</p>
<p>In the implementation, we can make an ex-memory variable either &#8220;refers to&#8221; the memory variable of the previous step, <p>In the implementation, we can make an ex-memory variable either &#8220;refer to&#8221; the memory variable of the previous step,
or copy the value of the previous memory value to the current ex-memory variable.</p> or copy the memory value of the previous step to the current ex-memory variable.</p>
</div> </div>
<div class="section" id="usage-in-python"> <div class="section" id="usage-in-python">
<span id="usage-in-python"></span><h3>Usage in Python<a class="headerlink" href="#usage-in-python" title="永久链接至标题"></a></h3> <span id="usage-in-python"></span><h3>Usage in Python<a class="headerlink" href="#usage-in-python" title="永久链接至标题"></a></h3>
<p>For more information on Block, please refer to the <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md">design doc</a>.</p> <p>For more information on Block, please refer to the <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/block.md">design doc</a>.</p>
<p>We can define an RNN&#8217;s step-net using Block:</p> <p>We can define an RNN&#8217;s step-net using a Block:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span> <span class="c1"># x is some operator&#39;s output, and is a LoDTensor</span> <span class="n">X</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span> <span class="c1"># x is some operator&#39;s output and is a LoDTensor</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span> <span class="n">a</span> <span class="o">=</span> <span class="n">some_op</span><span class="p">()</span>
<span class="c1"># declare parameters</span> <span class="c1"># declare parameters</span>
...@@ -259,7 +259,7 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -259,7 +259,7 @@ or copy the value of the previous memory value to the current ex-memory variable
<span class="n">x</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_input</span><span class="p">(</span><span class="n">X</span><span class="p">)</span> <span class="n">x</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_input</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="c1"># declare a memory (rnn&#39;s step)</span> <span class="c1"># declare a memory (rnn&#39;s step)</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_memory</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="n">a</span><span class="p">)</span> <span class="n">h</span> <span class="o">=</span> <span class="n">rnn</span><span class="o">.</span><span class="n">add_memory</span><span class="p">(</span><span class="n">init</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="c1"># h.pre_state() means previous memory of rnn</span> <span class="c1"># h.pre_state(), the previous memory of rnn</span>
<span class="n">new_state</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">add_two</span><span class="p">(</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span> <span class="n">new_state</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">add_two</span><span class="p">(</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span>
<span class="c1"># update current memory</span> <span class="c1"># update current memory</span>
<span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">new_state</span><span class="p">)</span> <span class="n">h</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">new_state</span><span class="p">)</span>
...@@ -271,17 +271,17 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -271,17 +271,17 @@ or copy the value of the previous memory value to the current ex-memory variable
</div> </div>
<p>Python API functions in above example:</p> <p>Python API functions in above example:</p>
<ul class="simple"> <ul class="simple">
<li><code class="docutils literal"><span class="pre">rnn.add_input</span></code> indicates the parameter is a variable that will be segmented into step-inputs.</li> <li><code class="docutils literal"><span class="pre">rnn.add_input</span></code>: indicates that the parameter is a variable that will be segmented into step-inputs.</li>
<li><code class="docutils literal"><span class="pre">rnn.add_memory</span></code> creates a variable used as the memory.</li> <li><code class="docutils literal"><span class="pre">rnn.add_memory</span></code>: creates a variable used as the memory.</li>
<li><code class="docutils literal"><span class="pre">rnn.add_outputs</span></code> mark the variables that will be concatenated across steps into the RNN output.</li> <li><code class="docutils literal"><span class="pre">rnn.add_outputs</span></code>: marks the variables that will be concatenated across steps into the RNN output.</li>
</ul> </ul>
</div> </div>
<div class="section" id="nested-rnn-and-lodtensor"> <div class="section" id="nested-rnn-and-lodtensor">
<span id="nested-rnn-and-lodtensor"></span><h3>Nested RNN and LoDTensor<a class="headerlink" href="#nested-rnn-and-lodtensor" title="永久链接至标题"></a></h3> <span id="nested-rnn-and-lodtensor"></span><h3>Nested RNN and LoDTensor<a class="headerlink" href="#nested-rnn-and-lodtensor" title="永久链接至标题"></a></h3>
<p>An RNN whose step-net includes other RNN operators is known as an <em>nested RNN</em>.</p> <p>An RNN whose step-net includes other RNN operators is known as an <em>nested RNN</em>.</p>
<p>For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences.</p> <p>For example, we could have a 2-level RNN, where the top level corresponds to paragraphs, and the lower level corresponds to sentences. Each step of the higher level RNN also receives an input from the corresponding step of the lower level, and additionally the output from the previous time step at the same level.</p>
<p>The following figure illustrates the feeding of text into the lower level, one sentence each step, and the feeding of step outputs to the top level. The final top level output is about the whole text.</p> <p>The following figure illustrates feeding in text into the lower level, one sentence at a step, and the feeding in step outputs to the top level. The final top level output is about the whole text.</p>
<p aligh="center"> <p align="center">
<img src="./images/2_level_rnn.png"/> <img src="./images/2_level_rnn.png"/>
</p><div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span> </p><div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span>
...@@ -296,7 +296,7 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -296,7 +296,7 @@ or copy the value of the previous memory value to the current ex-memory variable
<span class="c1"># chapter_data is a set of 128-dim word vectors</span> <span class="c1"># chapter_data is a set of 128-dim word vectors</span>
<span class="c1"># the first level of LoD is sentence</span> <span class="c1"># the first level of LoD is sentence</span>
<span class="c1"># the second level of LoD is chapter</span> <span class="c1"># the second level of LoD is a chapter</span>
<span class="n">chapter_data</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="nb">type</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">lod_tensor</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">chapter_data</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="mi">128</span><span class="p">],</span> <span class="nb">type</span><span class="o">=</span><span class="n">pd</span><span class="o">.</span><span class="n">lod_tensor</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">lower_level_rnn</span><span class="p">(</span><span class="n">paragraph</span><span class="p">):</span> <span class="k">def</span> <span class="nf">lower_level_rnn</span><span class="p">(</span><span class="n">paragraph</span><span class="p">):</span>
...@@ -324,13 +324,13 @@ or copy the value of the previous memory value to the current ex-memory variable ...@@ -324,13 +324,13 @@ or copy the value of the previous memory value to the current ex-memory variable
<span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W0</span><span class="p">,</span> <span class="n">paragraph_data</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U0</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">W0</span><span class="p">,</span> <span class="n">paragraph_data</span><span class="p">)</span> <span class="o">+</span> <span class="n">pd</span><span class="o">.</span><span class="n">matmul</span><span class="p">(</span><span class="n">U0</span><span class="p">,</span> <span class="n">h</span><span class="o">.</span><span class="n">pre_state</span><span class="p">()))</span>
<span class="n">top_level_rnn</span><span class="o">.</span><span class="n">add_outputs</span><span class="p">(</span><span class="n">h</span><span class="p">)</span> <span class="n">top_level_rnn</span><span class="o">.</span><span class="n">add_outputs</span><span class="p">(</span><span class="n">h</span><span class="p">)</span>
<span class="c1"># just output the last step</span> <span class="c1"># output the last step</span>
<span class="n">chapter_out</span> <span class="o">=</span> <span class="n">top_level_rnn</span><span class="p">(</span><span class="n">output_all_steps</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span> <span class="n">chapter_out</span> <span class="o">=</span> <span class="n">top_level_rnn</span><span class="p">(</span><span class="n">output_all_steps</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div> </pre></div>
</div> </div>
<p>in above example, the construction of the <code class="docutils literal"><span class="pre">top_level_rnn</span></code> calls <code class="docutils literal"><span class="pre">lower_level_rnn</span></code>. The input is a LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.</p> <p>In the above example, the construction of the <code class="docutils literal"><span class="pre">top_level_rnn</span></code> calls <code class="docutils literal"><span class="pre">lower_level_rnn</span></code>. The input is an LoD Tensor. The top level RNN segments input text data into paragraphs, and the lower level RNN segments each paragraph into sentences.</p>
<p>By default, the <code class="docutils literal"><span class="pre">RNNOp</span></code> will concatenate the outputs from all the time steps, <p>By default, the <code class="docutils literal"><span class="pre">RNNOp</span></code> will concatenate the outputs from all the time steps.
if the <code class="docutils literal"><span class="pre">output_all_steps</span></code> set to False, it will only output the final time step.</p> If the <code class="docutils literal"><span class="pre">output_all_steps</span></code> is set to False, it will only output the final time step.</p>
<p align="center"> <p align="center">
<img src="images/rnn_2level_data.png"/> <img src="images/rnn_2level_data.png"/>
</p></div> </p></div>
......
develop/doc_cn/design/ops/sequence_decoder.html
浏览文件 @ c622759a
...@@ -204,31 +204,24 @@ ...@@ -204,31 +204,24 @@
<div class="section" id="design-sequence-decoder-generating-lodtensors"> <div class="section" id="design-sequence-decoder-generating-lodtensors">
<span id="design-sequence-decoder-generating-lodtensors"></span><h1>Design: Sequence Decoder Generating LoDTensors<a class="headerlink" href="#design-sequence-decoder-generating-lodtensors" title="永久链接至标题"></a></h1> <span id="design-sequence-decoder-generating-lodtensors"></span><h1>Design: Sequence Decoder Generating LoDTensors<a class="headerlink" href="#design-sequence-decoder-generating-lodtensors" title="永久链接至标题"></a></h1>
<p>In tasks such as machine translation and image to text, <p>In tasks such as machine translation and visual captioning,
a <a class="reference external" href="https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md">sequence decoder</a> is necessary to generate sequences.</p> a <a class="reference external" href="https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md">sequence decoder</a> is necessary to generate sequences, one word at a time.</p>
<p>This documentation describes how to implement the sequence decoder as an operator.</p> <p>This documentation describes how to implement the sequence decoder as an operator.</p>
<div class="section" id="beam-search-based-decoder"> <div class="section" id="beam-search-based-decoder">
<span id="beam-search-based-decoder"></span><h2>Beam Search based Decoder<a class="headerlink" href="#beam-search-based-decoder" title="永久链接至标题"></a></h2> <span id="beam-search-based-decoder"></span><h2>Beam Search based Decoder<a class="headerlink" href="#beam-search-based-decoder" title="永久链接至标题"></a></h2>
<p>The <a class="reference external" href="https://en.wikipedia.org/wiki/Beam_search">beam search algorithm</a> is necessary when generating sequences, <p>The <a class="reference external" href="https://en.wikipedia.org/wiki/Beam_search">beam search algorithm</a> is necessary when generating sequences. It is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.</p>
it is a heuristic search algorithm that explores the paths by expanding the most promising node in a limited set.</p> <p>In the old version of PaddlePaddle, the C++ class <code class="docutils literal"><span class="pre">RecurrentGradientMachine</span></code> implements the general sequence decoder based on beam search, due to the complexity involved, the implementation relies on a lot of special data structures that are quite trivial and hard to be customized by users.</p>
<p>In the old version of PaddlePaddle, a C++ class <code class="docutils literal"><span class="pre">RecurrentGradientMachine</span></code> implements the general sequence decoder based on beam search, <p>There are a lot of heuristic tricks in the sequence generation tasks, so the flexibility of sequence decoder is very important to users.</p>
due to the complexity, the implementation relays on a lot of special data structures, <p>During the refactoring of PaddlePaddle, some new concepts are proposed such as: <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md">LoDTensor</a> and <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md">TensorArray</a> that can better support the sequence usage, and they can also help make the implementation of beam search based sequence decoder <strong>more transparent and modular</strong> .</p>
quite trivial and hard to be customized by users.</p> <p>For example, the RNN states, candidates IDs and probabilities of beam search can be represented all as <code class="docutils literal"><span class="pre">LoDTensors</span></code>;
<p>There are a lot of heuristic tricks in the sequence generation tasks,
so the flexibility of sequence decoder is very important to users.</p>
<p>During PaddlePaddle&#8217;s refactoring work,
some new concept is proposed such as <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/lod_tensor.md">LoDTensor</a> and <a class="reference external" href="https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/tensor_array.md">TensorArray</a> that can better support sequence usage,
and they can help to make the implementation of beam search based sequence decoder <strong>more transparent and modular</strong> .</p>
<p>For example, the RNN sates, candidates IDs and probabilities of beam search can be represented as <code class="docutils literal"><span class="pre">LoDTensors</span></code>;
the selected candidate&#8217;s IDs in each time step can be stored in a <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Packed</span></code> to the sentences translated.</p> the selected candidate&#8217;s IDs in each time step can be stored in a <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Packed</span></code> to the sentences translated.</p>
</div> </div>
<div class="section" id="changing-lod-s-absolute-offset-to-relative-offsets"> <div class="section" id="changing-lod-s-absolute-offset-to-relative-offsets">
<span id="changing-lod-s-absolute-offset-to-relative-offsets"></span><h2>Changing LoD&#8217;s absolute offset to relative offsets<a class="headerlink" href="#changing-lod-s-absolute-offset-to-relative-offsets" title="永久链接至标题"></a></h2> <span id="changing-lod-s-absolute-offset-to-relative-offsets"></span><h2>Changing LoD&#8217;s absolute offset to relative offsets<a class="headerlink" href="#changing-lod-s-absolute-offset-to-relative-offsets" title="永久链接至标题"></a></h2>
<p>The current <code class="docutils literal"><span class="pre">LoDTensor</span></code> is designed to store levels of variable-length sequences, <p>The current <code class="docutils literal"><span class="pre">LoDTensor</span></code> is designed to store levels of variable-length sequences. It stores several arrays of integers where each represents a level.</p>
it stores several arrays of integers each represents a level.</p> <p>The integers in each level represent the begin and end (not inclusive) offset of a sequence <strong>in the underlying tensor</strong>,
<p>The integers in each level represents the begin and end (not inclusive) offset of a sequence <strong>in the underlying tensor</strong>, let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clarity.</p>
let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clear.</p> <p>The relative-offset LoD can retrieve any sequence very quickly but fails to represent empty sequences, for example, a two-level LoD is as follows</p>
<p>The relative-offset LoD can fast retrieve any sequence but fails to represent empty sequences, for example, a two-level LoD is as follows</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="p">[[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]]</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">9</span><span class="p">]]</span>
</pre></div> </pre></div>
...@@ -240,8 +233,7 @@ let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clear. ...@@ -240,8 +233,7 @@ let&#8217;s call this format the <strong>absolute-offset LoD</strong> for clear.
</ul> </ul>
<p>while on the second level, there are several empty sequences that both begin and end at <code class="docutils literal"><span class="pre">3</span></code>. <p>while on the second level, there are several empty sequences that both begin and end at <code class="docutils literal"><span class="pre">3</span></code>.
It is impossible to tell how many empty second-level sequences exist in the first-level sequences.</p> It is impossible to tell how many empty second-level sequences exist in the first-level sequences.</p>
<p>There are many scenarios that relay on empty sequence representation, <p>There are many scenarios that rely on empty sequence representation, for example in machine translation or visual captioning, one instance has no translation or the empty candidate set for a prefix.</p>
such as machine translation or image to text, one instance has no translations or the empty candidate set for a prefix.</p>
<p>So let&#8217;s introduce another format of LoD, <p>So let&#8217;s introduce another format of LoD,
it stores <strong>the offsets of the lower level sequences</strong> and is called <strong>relative-offset</strong> LoD.</p> it stores <strong>the offsets of the lower level sequences</strong> and is called <strong>relative-offset</strong> LoD.</p>
<p>For example, to represent the same sequences of the above data</p> <p>For example, to represent the same sequences of the above data</p>
...@@ -253,13 +245,12 @@ it stores <strong>the offsets of the lower level sequences</strong> and is calle ...@@ -253,13 +245,12 @@ it stores <strong>the offsets of the lower level sequences</strong> and is calle
their offsets in the second-level LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">3)</span></code> and <code class="docutils literal"><span class="pre">[3,</span> <span class="pre">5)</span></code>.</p> their offsets in the second-level LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">3)</span></code> and <code class="docutils literal"><span class="pre">[3,</span> <span class="pre">5)</span></code>.</p>
<p>The second level is the same with the relative offset example because the lower level is a tensor. <p>The second level is the same with the relative offset example because the lower level is a tensor.
It is easy to find out the second sequence in the first-level LoD has two empty sequences.</p> It is easy to find out the second sequence in the first-level LoD has two empty sequences.</p>
<p>The following demos are based on relative-offset LoD.</p> <p>The following examples are based on relative-offset LoD.</p>
</div> </div>
<div class="section" id="usage-in-a-simple-machine-translation-model"> <div class="section" id="usage-in-a-simple-machine-translation-model">
<span id="usage-in-a-simple-machine-translation-model"></span><h2>Usage in a simple machine translation model<a class="headerlink" href="#usage-in-a-simple-machine-translation-model" title="永久链接至标题"></a></h2> <span id="usage-in-a-simple-machine-translation-model"></span><h2>Usage in a simple machine translation model<a class="headerlink" href="#usage-in-a-simple-machine-translation-model" title="永久链接至标题"></a></h2>
<p>Let&#8217;s start from a simple machine translation model that is simplified from <a class="reference external" href="https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation">machine translation chapter</a> to draw a simple blueprint of what a sequence decoder can do and how to use it.</p> <p>Let&#8217;s start from a simple machine translation model that is simplified from the <a class="reference external" href="https://github.com/PaddlePaddle/book/tree/develop/08.machine_translation">machine translation chapter</a> to draw a blueprint of what a sequence decoder can do and how to use it.</p>
<p>The model has an encoder that learns the semantic vector from a sequence, <p>The model has an encoder that learns the semantic vector from a sequence, and a decoder which uses the sequence encoder to generate new sentences.</p>
and a decoder which uses the sequence decoder to generate new sentences.</p>
<p><strong>Encoder</strong></p> <p><strong>Encoder</strong></p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">paddle</span> <span class="kn">as</span> <span class="nn">pd</span>
...@@ -306,7 +297,7 @@ and a decoder which uses the sequence decoder to generate new sentences.</p> ...@@ -306,7 +297,7 @@ and a decoder which uses the sequence decoder to generate new sentences.</p>
<span class="c1"># which means there are 2 sentences to translate</span> <span class="c1"># which means there are 2 sentences to translate</span>
<span class="c1"># - the first sentence has 1 translation prefixes, the offsets are [0, 1)</span> <span class="c1"># - the first sentence has 1 translation prefixes, the offsets are [0, 1)</span>
<span class="c1"># - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6)</span> <span class="c1"># - the second sentence has 2 translation prefixes, the offsets are [1, 3) and [3, 6)</span>
<span class="c1"># the target_word.lod is </span> <span class="c1"># the target_word.lod is</span>
<span class="c1"># [[0, 1, 6]</span> <span class="c1"># [[0, 1, 6]</span>
<span class="c1"># [0, 2, 4, 7, 9 12]]</span> <span class="c1"># [0, 2, 4, 7, 9 12]]</span>
<span class="c1"># which means 2 sentences to translate, each has 1 and 5 prefixes</span> <span class="c1"># which means 2 sentences to translate, each has 1 and 5 prefixes</span>
...@@ -344,30 +335,29 @@ and a decoder which uses the sequence decoder to generate new sentences.</p> ...@@ -344,30 +335,29 @@ and a decoder which uses the sequence decoder to generate new sentences.</p>
<span class="n">translation_ids</span><span class="p">,</span> <span class="n">translation_scores</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">()</span> <span class="n">translation_ids</span><span class="p">,</span> <span class="n">translation_scores</span> <span class="o">=</span> <span class="n">decoder</span><span class="p">()</span>
</pre></div> </pre></div>
</div> </div>
<p>The <code class="docutils literal"><span class="pre">decoder.beam_search</span></code> is a operator that given the candidates and the scores of translations including the candidates, <p>The <code class="docutils literal"><span class="pre">decoder.beam_search</span></code> is an operator that, given the candidates and the scores of translations including the candidates,
return the result of the beam search algorithm.</p> returns the result of the beam search algorithm.</p>
<p>In this way, users can customize anything on the inputs or outputs of beam search, for example, two ways to prune some translation prefixes</p> <p>In this way, users can customize anything on the input or output of beam search, for example:</p>
<ol class="simple"> <ol class="simple">
<li>meke the correspondind elements in <code class="docutils literal"><span class="pre">topk_generated_scores</span></code> zero or some small values, beam_search will discard this candidate.</li> <li>Make the corresponding elements in <code class="docutils literal"><span class="pre">topk_generated_scores</span></code> zero or some small values, beam_search will discard this candidate.</li>
<li>remove some specific candidate in <code class="docutils literal"><span class="pre">selected_ids</span></code></li> <li>Remove some specific candidate in <code class="docutils literal"><span class="pre">selected_ids</span></code>.</li>
<li>get the final <code class="docutils literal"><span class="pre">translation_ids</span></code>, remove the translation sequence in it.</li> <li>Get the final <code class="docutils literal"><span class="pre">translation_ids</span></code>, remove the translation sequence in it.</li>
</ol> </ol>
<p>The implementation of sequence decoder can reuse the C++ class <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30">RNNAlgorithm</a>, <p>The implementation of sequence decoder can reuse the C++ class: <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/paddle/operators/dynamic_recurrent_op.h#L30">RNNAlgorithm</a>,
so the python syntax is quite similar to a <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop">RNN</a>.</p> so the python syntax is quite similar to that of an <a class="reference external" href="https://github.com/Superjom/Paddle/blob/68cac3c0f8451fe62a4cdf156747d6dc0ee000b3/doc/design/block.md#blocks-with-for-and-rnnop">RNN</a>.</p>
<p>Both of them are two-level <code class="docutils literal"><span class="pre">LoDTensors</span></code></p> <p>Both of them are two-level <code class="docutils literal"><span class="pre">LoDTensors</span></code>:</p>
<ul class="simple"> <ul class="simple">
<li>the first level represents <code class="docutils literal"><span class="pre">batch_size</span></code> of (source) sentences;</li> <li>The first level represents <code class="docutils literal"><span class="pre">batch_size</span></code> of (source) sentences.</li>
<li>the second level represents the candidate ID sets for translation prefix.</li> <li>The second level represents the candidate ID sets for translation prefix.</li>
</ul> </ul>
<p>for example, 3 source sentences to translate, and has 2, 3, 1 candidates.</p> <p>For example, 3 source sentences to translate, and has 2, 3, 1 candidates.</p>
<p>Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, <p>Unlike an RNN, in sequence decoder, the previous state and the current state have different LoD and shape, and an <code class="docutils literal"><span class="pre">lod_expand</span></code> operator is used to expand the LoD of the previous state to fit the current state.</p>
a <code class="docutils literal"><span class="pre">lod_expand</span></code> operator is used to expand the LoD of the previous state to fit the current state.</p> <p>For example, the previous state:</p>
<p>For example, the previous state</p>
<ul class="simple"> <ul class="simple">
<li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">1,</span> <span class="pre">3][0,</span> <span class="pre">2,</span> <span class="pre">5,</span> <span class="pre">6]</span></code></li> <li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">1,</span> <span class="pre">3][0,</span> <span class="pre">2,</span> <span class="pre">5,</span> <span class="pre">6]</span></code></li>
<li>content of tensor is <code class="docutils literal"><span class="pre">a1</span> <span class="pre">a2</span> <span class="pre">b1</span> <span class="pre">b2</span> <span class="pre">b3</span> <span class="pre">c1</span></code></li> <li>content of tensor is <code class="docutils literal"><span class="pre">a1</span> <span class="pre">a2</span> <span class="pre">b1</span> <span class="pre">b2</span> <span class="pre">b3</span> <span class="pre">c1</span></code></li>
</ul> </ul>
<p>the current state stored in <code class="docutils literal"><span class="pre">encoder_ctx_expanded</span></code></p> <p>the current state is stored in <code class="docutils literal"><span class="pre">encoder_ctx_expanded</span></code>:</p>
<ul class="simple"> <ul class="simple">
<li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">2,</span> <span class="pre">7][0</span> <span class="pre">3</span> <span class="pre">5</span> <span class="pre">8</span> <span class="pre">9</span> <span class="pre">11</span> <span class="pre">11]</span></code></li> <li>LoD is <code class="docutils literal"><span class="pre">[0,</span> <span class="pre">2,</span> <span class="pre">7][0</span> <span class="pre">3</span> <span class="pre">5</span> <span class="pre">8</span> <span class="pre">9</span> <span class="pre">11</span> <span class="pre">11]</span></code></li>
<li>the content is<ul> <li>the content is<ul>
...@@ -380,51 +370,45 @@ a <code class="docutils literal"><span class="pre">lod_expand</span></code> oper ...@@ -380,51 +370,45 @@ a <code class="docutils literal"><span class="pre">lod_expand</span></code> oper
</ul> </ul>
</li> </li>
</ul> </ul>
<p>Benefit from the relative offset LoD, empty candidate set can be represented naturally.</p> <p>The benefit from the relative offset LoD is that the empty candidate set can be represented naturally.</p>
<p>the status in each time step can be stored in <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Pack</span></code>ed to a final LoDTensor, the corresponding syntax is</p> <p>The status in each time step can be stored in <code class="docutils literal"><span class="pre">TensorArray</span></code>, and <code class="docutils literal"><span class="pre">Pack</span></code>ed to a final LoDTensor. The corresponding syntax is:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_ids</span><span class="p">)</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_ids</span><span class="p">)</span>
<span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_generation_scores</span><span class="p">)</span> <span class="n">decoder</span><span class="o">.</span><span class="n">output</span><span class="p">(</span><span class="n">selected_generation_scores</span><span class="p">)</span>
</pre></div> </pre></div>
</div> </div>
<p>the <code class="docutils literal"><span class="pre">selected_ids</span></code> is the candidate ids for the prefixes, <p>The <code class="docutils literal"><span class="pre">selected_ids</span></code> are the candidate ids for the prefixes, and will be <code class="docutils literal"><span class="pre">Packed</span></code> by <code class="docutils literal"><span class="pre">TensorArray</span></code> to a two-level <code class="docutils literal"><span class="pre">LoDTensor</span></code>, where the first level represents the source sequences and the second level represents generated sequences.</p>
it will be <code class="docutils literal"><span class="pre">Packed</span></code> by <code class="docutils literal"><span class="pre">TensorArray</span></code> to a two-level <code class="docutils literal"><span class="pre">LoDTensor</span></code>, <p>Packing the <code class="docutils literal"><span class="pre">selected_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code> that stores scores of each translation candidate.</p>
the first level represents the source sequences, <p>Packing the <code class="docutils literal"><span class="pre">selected_generation_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code>, and each tail is the probability of the translation.</p>
the second level represents generated sequences.</p>
<p>Pack the <code class="docutils literal"><span class="pre">selected_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code> that stores scores of each candidate of translations.</p>
<p>Pack the <code class="docutils literal"><span class="pre">selected_generation_scores</span></code> will get a <code class="docutils literal"><span class="pre">LoDTensor</span></code>, and each tail is the probability of the translation.</p>
</div> </div>
<div class="section" id="lod-and-shape-changes-during-decoding"> <div class="section" id="lod-and-shape-changes-during-decoding">
<span id="lod-and-shape-changes-during-decoding"></span><h2>LoD and shape changes during decoding<a class="headerlink" href="#lod-and-shape-changes-during-decoding" title="永久链接至标题"></a></h2> <span id="lod-and-shape-changes-during-decoding"></span><h2>LoD and shape changes during decoding<a class="headerlink" href="#lod-and-shape-changes-during-decoding" title="永久链接至标题"></a></h2>
<p align="center"> <p align="center">
<img src="./images/LOD-and-shape-changes-during-decoding.jpg"/> <img src="./images/LOD-and-shape-changes-during-decoding.jpg"/>
</p><p>According the image above, the only phrase to change LoD is beam search.</p> </p><p>According to the image above, the only phase that changes the LoD is beam search.</p>
</div> </div>
<div class="section" id="beam-search-design"> <div class="section" id="beam-search-design">
<span id="beam-search-design"></span><h2>Beam search design<a class="headerlink" href="#beam-search-design" title="永久链接至标题"></a></h2> <span id="beam-search-design"></span><h2>Beam search design<a class="headerlink" href="#beam-search-design" title="永久链接至标题"></a></h2>
<p>The beam search algorthm will be implemented as one method of the sequence decoder, it has 3 inputs</p> <p>The beam search algorithm will be implemented as one method of the sequence decoder and has 3 inputs:</p>
<ol class="simple"> <ol class="simple">
<li><code class="docutils literal"><span class="pre">topk_ids</span></code>, top K candidate ids for each prefix.</li> <li><code class="docutils literal"><span class="pre">topk_ids</span></code>, the top K candidate ids for each prefix.</li>
<li><code class="docutils literal"><span class="pre">topk_scores</span></code>, the corresponding scores for <code class="docutils literal"><span class="pre">topk_ids</span></code></li> <li><code class="docutils literal"><span class="pre">topk_scores</span></code>, the corresponding scores for <code class="docutils literal"><span class="pre">topk_ids</span></code></li>
<li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the score of the prefixes.</li> <li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the score of the prefixes.</li>
</ol> </ol>
<p>All of the are LoDTensors, so that the sequence affilication is clear. <p>All of these are LoDTensors, so that the sequence affiliation is clear. Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.</p>
Beam search will keep a beam for each prefix and select a smaller candidate set for each prefix.</p> <p>It will return three variables:</p>
<p>It will return three variables</p>
<ol class="simple"> <ol class="simple">
<li><code class="docutils literal"><span class="pre">selected_ids</span></code>, the final candidate beam search function selected for the next step.</li> <li><code class="docutils literal"><span class="pre">selected_ids</span></code>, the final candidate beam search function selected for the next step.</li>
<li><code class="docutils literal"><span class="pre">selected_scores</span></code>, the scores for the candidates.</li> <li><code class="docutils literal"><span class="pre">selected_scores</span></code>, the scores for the candidates.</li>
<li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the updated scores for each prefixes (with the new candidates appended).</li> <li><code class="docutils literal"><span class="pre">generated_scores</span></code>, the updated scores for each prefix (with the new candidates appended).</li>
</ol> </ol>
</div> </div>
<div class="section" id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray"> <div class="section" id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray">
<span id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray"></span><h2>Introducing the LoD-based <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">Unpack</span></code> methods in <code class="docutils literal"><span class="pre">TensorArray</span></code><a class="headerlink" href="#introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray" title="永久链接至标题"></a></h2> <span id="introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray"></span><h2>Introducing the LoD-based <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">Unpack</span></code> methods in <code class="docutils literal"><span class="pre">TensorArray</span></code><a class="headerlink" href="#introducing-the-lod-based-pack-and-unpack-methods-in-tensorarray" title="永久链接至标题"></a></h2>
<p>The <code class="docutils literal"><span class="pre">selected_ids</span></code>, <code class="docutils literal"><span class="pre">selected_scores</span></code> and <code class="docutils literal"><span class="pre">generated_scores</span></code> are LoDTensors, <p>The <code class="docutils literal"><span class="pre">selected_ids</span></code>, <code class="docutils literal"><span class="pre">selected_scores</span></code> and <code class="docutils literal"><span class="pre">generated_scores</span></code> are LoDTensors that exist at each time step,
and they exist in each time step,
so it is natural to store them in arrays.</p> so it is natural to store them in arrays.</p>
<p>Currently, PaddlePaddle has a module called <code class="docutils literal"><span class="pre">TensorArray</span></code> which can store an array of tensors, <p>Currently, PaddlePaddle has a module called <code class="docutils literal"><span class="pre">TensorArray</span></code> which can store an array of tensors. It is better to store the results of beam search in a <code class="docutils literal"><span class="pre">TensorArray</span></code>.</p>
the results of beam search are better to store in a <code class="docutils literal"><span class="pre">TensorArray</span></code>.</p> <p>The <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">UnPack</span></code> in <code class="docutils literal"><span class="pre">TensorArray</span></code> are used to pack tensors in the array to an <code class="docutils literal"><span class="pre">LoDTensor</span></code> or split the <code class="docutils literal"><span class="pre">LoDTensor</span></code> to an array of tensors.
<p>The <code class="docutils literal"><span class="pre">Pack</span></code> and <code class="docutils literal"><span class="pre">UnPack</span></code> in <code class="docutils literal"><span class="pre">TensorArray</span></code> are used to package tensors in the array to a <code class="docutils literal"><span class="pre">LoDTensor</span></code> or split the <code class="docutils literal"><span class="pre">LoDTensor</span></code> to an array of tensors. It needs some extensions to support the packing or unpacking an array of <code class="docutils literal"><span class="pre">LoDTensors</span></code>.</p>
It needs some extensions to support pack or unpack an array of <code class="docutils literal"><span class="pre">LoDTensors</span></code>.</p>
</div> </div>
</div> </div>
......
develop/doc_cn/searchindex.js
浏览文件 @ c622759a
因为 它太大了无法显示 source diff 。你可以改为 查看blob。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或