<p>The defalut implementation is diagonal/peephole connection
(<aclass="reference external"href="https://arxiv.org/pdf/1402.1128.pdf">https://arxiv.org/pdf/1402.1128.pdf</a>), the formula is as follows:</p>
...
...
@@ -368,7 +368,7 @@ with zeros whenever lookup encounters it in <code class="xref py py-attr docutil
the matrix of weights from the input gate to the input), <spanclass="math">\(W_{ic}, W_{fc}, W_{oc}\)</span> are diagonal weight matrices for peephole connections. In
our implementation, we use vectors to reprenset these diagonal weight
matrices. The <spanclass="math">\(b\)</span> terms denote bias vectors (<spanclass="math">\(b_i\)</span> is the input
gate bias vector), <spanclass="math">\(\sigma\)</span> is the non-line activations, such as
gate bias vector), <spanclass="math">\(\sigma\)</span> is the non-linear activations, such as
logistic sigmoid function, and <spanclass="math">\(i, f, o\)</span> and <spanclass="math">\(c\)</span> are the input
gate, forget gate, output gate, and cell activation vectors, respectively,
all of which have the same size as the cell output activation vector <spanclass="math">\(h\)</span>.</p>
...
...
@@ -394,15 +394,15 @@ tensor in this Variable is a matrix with shape
<p>LSTMP (LSTM with recurrent projection) layer has a separate projection
layer after the LSTM layer, projecting the original hidden state to a
lower-dimensional one, which is proposed to reduce the number of total
parameters and furthermore computational complexity for the LSTM,
espeacially for the case that the size of output units is relative
large (<aclass="reference external"href="https://research.google.com/pubs/archive/43905.pdf">https://research.google.com/pubs/archive/43905.pdf</a>).</p>
<p>The formula is as follows:</p>
<divclass="math">
\[ \begin{align}\begin{aligned}i_t & = \sigma(W_{ix}x_{t} + W_{ir}r_{t-1} + W_{ic}c_{t-1} + b_i)\\f_t & = \sigma(W_{fx}x_{t} + W_{fr}r_{t-1} + W_{fc}c_{t-1} + b_f)\\\tilde{c_t} & = act_g(W_{cx}x_t + W_{cr}r_{t-1} + b_c)\\o_t & = \sigma(W_{ox}x_{t} + W_{or}r_{t-1} + W_{oc}c_t + b_o)\\c_t & = f_t \odot c_{t-1} + i_t \odot \tilde{c_t}\\h_t & = o_t \odot act_h(c_t)\\r_t & = \overline{act_h}(W_{rh}h_t)\end{aligned}\end{align} \]</div>
<p>In the above formula:</p>
<ulclass="simple">
<li><spanclass="math">\(W\)</span>: Denotes weight matrices (e.g. <spanclass="math">\(W_{xi}\)</span> is the matrix of weights from the input gate to the input).</li>
<li><spanclass="math">\(W_{ic}\)</span>, <spanclass="math">\(W_{fc}\)</span>, <spanclass="math">\(W_{oc}\)</span>: Diagonal weight matrices for peephole connections. In our implementation, we use vectors to reprenset these diagonal weight matrices.</li>
<li><spanclass="math">\(b\)</span>: Denotes bias vectors (e.g. <spanclass="math">\(b_i\)</span> is the input gate bias vector).</li>
<li><spanclass="math">\(\sigma\)</span>: The activation, such as logistic sigmoid function.</li>
<li><spanclass="math">\(i, f, o\)</span> and <spanclass="math">\(c\)</span>: The input gate, forget gate, output gate, and cell activation vectors, respectively, all of which have the same size as the cell output activation vector <spanclass="math">\(h\)</span>.</li>
<li><spanclass="math">\(h\)</span>: The hidden state.</li>
<li><spanclass="math">\(r\)</span>: The recurrent projection of the hidden state.</li>
<li><spanclass="math">\(\tilde{c_t}\)</span>: The candidate hidden state, whose computation is based on the current input and previous hidden state.</li>
<li><spanclass="math">\(\odot\)</span>: The element-wise product of the vectors.</li>
<li><spanclass="math">\(act_g\)</span> and <spanclass="math">\(act_h\)</span>: The cell input and cell output activation functions and <cite>tanh</cite> is usually used for them.</li>
<li><spanclass="math">\(\overline{act_h}\)</span>: The activation function for the projection output, usually using <cite>identity</cite> or same as <spanclass="math">\(act_h\)</span>.</li>
</ul>
<p>Set <cite>use_peepholes</cite> to <cite>False</cite> to disable peephole connection. The formula
is omitted here, please refer to the paper
<aclass="reference external"href="http://www.bioinf.jku.at/publications/older/2604.pdf">http://www.bioinf.jku.at/publications/older/2604.pdf</a> for details.</p>
<p>Note that these <spanclass="math">\(W_{xi}x_{t}, W_{xf}x_{t}, W_{xc}x_{t}, W_{xo}x_{t}\)</span>
operations on the input <spanclass="math">\(x_{t}\)</span> are NOT included in this operator.
Users can choose to use fully-connected layer before LSTMP layer.</p>
<li><strong>name</strong> (<em>str|None</em>) – A name for this layer(optional). If set None, the layer
will be named automatically.</li>
</ul>
</td>
</tr>
<trclass="field-even field"><thclass="field-name">Returns:</th><tdclass="field-body"><pclass="first">The projection of hidden state, and cell state of LSTMP. The shape of projection is (T x P), for the cell state which is (T x D), and both LoD is the same with the <cite>input</cite>.</p>
"comment":"(string, default: tanh)The activation for candidate hidden state, `tanh` by default.",
"generated":0
}]
},{
"type":"lstmp",
"comment":"\nLong-Short Term Memory with recurrent Projection layer (LSTMP) Operator.\n\nLSTMP has a separate projection layer after the LSTM layer, projecting the \noriginal hidden state to a lower-dimensional one, which is proposed to reduce \nthe number of total parameters and furthermore computational complexity for \nthe LSTM, espeacially for the case that the size of output units is relative \nlarge (https://research.google.com/pubs/archive/43905.pdf). \n\nThe formula is as follows:\n\n$$\ni_t = \\sigma(W_{ix}x_{t} + W_{ir}r_{t-1} + W_{ic}c_{t-1} + b_i) \\\\\n\nf_t = \\sigma(W_{fx}x_{t} + W_{fr}r_{t-1} + W_{fc}c_{t-1} + b_f) \\\\\n\n\\tilde{c_t} = act_g(W_{cx}x_t + W_{cr}r_{t-1} + b_c) \\\\\n\no_t = \\sigma(W_{ox}x_{t} + W_{or}r_{t-1} + W_{oc}c_t + b_o) \\\\\n\nc_t = f_t \\odot c_{t-1} + i_t \\odot \\tilde{c_t} \\\\\n\nh_t = o_t \\odot act_h(c_t) \\\\\n\nr_t = \\overline{act_h}(W_{rh}h_t)\n$$\n\nwhere the W terms denote weight matrices (e.g. $W_{xi}$ is the matrix\nof weights from the input gate to the input), $W_{ic}, W_{fc}, W_{oc}$\nare diagonal weight matrices for peephole connections. In our implementation,\nwe use vectors to reprenset these diagonal weight matrices. The b terms\ndenote bias vectors ($b_i$ is the input gate bias vector), $\\sigma$\nis the activation, such as logistic sigmoid function, and\n$i, f, o$ and $c$ are the input gate, forget gate, output gate,\nand cell activation vectors, respectively, all of which have the same size as\nthe cell output activation vector $h$. Here $h$ is usually called the hidden \nstate and $r$ denotes its recurrent projection. And $\\tilde{c_t}$ is also \ncalled the candidate hidden state, whose computation is based on the current \ninput and previous hidden state.\n\nThe $\\odot$ is the element-wise product of the vectors. $act_g$ and $act_h$\nare the cell input and cell output activation functions and `tanh` is usually\nused for them. $\\overline{act_h}$ is the activation function for the \nprojection output, usually using `identity` or same as $act_h$.\n\nNote that these $W_{xi}x_{t}, W_{xf}x_{t}, W_{xc}x_{t}, W_{xo}x_{t}$\noperations on the input $x_{t}$ are NOT included in this operator.\nUsers can choose to use fully-connected operator before LSTMP operator.\n\n",
"inputs":[
{
"name":"Input",
"comment":"(LoDTensor) the input for sequence data, which supports variable-time length input sequence. The underlying tensor in this LoDTensor is a matrix with shape (T X 4D), where T is the total time steps in this mini-batch, D is the hidden size.",
"duplicable":0,
"intermediate":0
},{
"name":"H0",
"comment":"(Tensor, optional) the initial hidden state is an optional input. This is a tensor with shape (N x D), where N is the batch size and D is the hidden size.",
"duplicable":0,
"intermediate":0
},{
"name":"C0",
"comment":"(Tensor, optional) the initial cell state is an optional input. This is a tensor with shape (N x D), where N is the batch size. `C0` should not be null if `H0` provided.",
"duplicable":0,
"intermediate":0
},{
"name":"Weight",
"comment":"(Tensor) the learnable hidden-hidden weights. - The shape is (P x 4D), where P is the projection layer size and D is the hidden size. - Weight = {W_cr, W_ir, W_fr, W_or}",
"duplicable":0,
"intermediate":0
},{
"name":"ProjWeight",
"comment":"(Tensor) the learnable weight of the projection layer. - The shape is (D x P), where P is the recurrent projection layer size and D is the hidden size. - ProjWeight = {W_rh}",
"duplicable":0,
"intermediate":0
},{
"name":"Bias",
"comment":"(Tensor) the learnable biases, which contains two parts: input-hidden biases and peephole connections weights if setting `use_peepholes` to `True`. 1. `use_peepholes = False` - The shape is (1 x 4D). - Bias = {b_c, b_i, b_f, b_o}.2. `use_peepholes = True` - The shape is (1 x 7D). - Bias = {b_c, b_i, b_f, b_o, W_ic, W_fc, W_oc}.",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Projection",
"comment":"(LoDTensor) the projection of the hidden state of LSTMP operator. The shape is (T x P), and LoD is the same with the `Input`.",
"duplicable":0,
"intermediate":0
},{
"name":"Cell",
"comment":"(LoDTensor) the cell state of LSTMP operator. The shape is (T x D), and lod is the same with the `Input`.",
"duplicable":0,
"intermediate":0
},{
"name":"BatchGate",
"comment":"(LoDTensor) This LoDTensor contains input gate, forget gate and output gate after the activations. This LoDTensor has the same shape as the reorganized input, which is also be called batch input. The LoD size is 2. The first-level LoD is the batch offsets and the second contains the indices, which denotes the position of reorganized sequence in the raw input.",
"duplicable":0,
"intermediate":1
},{
"name":"BatchCellPreAct",
"comment":"(LoDTensor) the pre-activation cell state reorganized in batch. This LoDTensor is obtained in the forward and used in the backward.",
"duplicable":0,
"intermediate":1
},{
"name":"BatchHidden",
"comment":"(LoDTensor) the hidden state reorganized in batch. This LoDTensor is obtained in the forward and used in the backward.",
"duplicable":0,
"intermediate":1
},{
"name":"OrderedP0",
"comment":"(Tensor) the projection of the initial hidden state H0. This is a tensor with shape (N x P), where N is the batch size and P is the hidden size.",
"duplicable":0,
"intermediate":1
}],
"attrs":[
{
"name":"use_peepholes",
"type":"bool",
"comment":"(bool, defalut: True) whether to enable diagonal/peephole connections.",
"generated":0
},{
"name":"is_reverse",
"type":"bool",
"comment":"(bool, defalut: False) whether to compute reversed LSTMP.",
"generated":0
},{
"name":"gate_activation",
"type":"string",
"comment":"(string, default: sigmoid)The activation for input gate, forget gate and output gate, `sigmoid` by default.",
"generated":0
},{
"name":"cell_activation",
"type":"string",
"comment":"(string, default: tanh)The activation for cell output, `tanh` by defalut.",
"generated":0
},{
"name":"candidate_activation",
"type":"string",
"comment":"(string, default: tanh)The activation for candidate hidden state, `tanh` by default.",
"generated":0
},{
"name":"proj_activation",
"type":"string",
"comment":"(string, default: tanh)The activation for projection output, `tanh` by defalut.",
"generated":0
}]
},{
"type":"warpctc",
"comment":"\nAn operator integrating the open-source\n[warp-ctc](https://github.com/baidu-research/warp-ctc) library, which is used in\n[Deep Speech 2: End-toEnd Speech Recognition in English and Mandarin](\nhttps://arxiv.org/pdf/1512.02595v1.pdf),\nto compute Connectionist Temporal Classification (CTC) loss.\nIt can be aliased as softmax with ctc, since a native softmax activation is\ninterated to the warp-ctc library, to to normlize values for each row of the\ninput tensor.\n\nMore detail of CTC loss can be found by refering to\n[Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with\nRecurrent Neural Networks](\nhttp://machinelearning.wustl.edu/mlpapers/paper_files/icml2006_GravesFGS06.pdf).\n",
...
...
@@ -1348,34 +1447,6 @@
"intermediate":0
}],
"attrs":[]
},{
"type":"scatter",
"comment":"\nScatter Operator.\n\nThis operator obtains output by updating the input on selected indices on the first axis:\n\n$$\nOut = Ref \\\\\nOut[Index] = Ref[Index] + Updates\n$$\n\n",
"inputs":[
{
"name":"Ref",
"comment":"The source input of scatter op",
"duplicable":0,
"intermediate":0
},{
"name":"Index",
"comment":"The index input of scatter op where Ref will be updated",
"duplicable":0,
"intermediate":0
},{
"name":"Updates",
"comment":"The updated value of updates op",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"The output of add op",
"duplicable":0,
"intermediate":0
}],
"attrs":[]
},{
"type":"max_sequence_len",
"comment":"Calculate the max sequence length through lod_rank_table.",
...
...
@@ -2134,6 +2205,34 @@
"comment":"(bool, default false) Use Nesterov Momentum",
"generated":0
}]
},{
"type":"scatter",
"comment":"\nScatter Operator.\n\nThis operator obtains output by updating the input on selected indices on the first axis:\n\n$$\nOut = Ref \\\\\nOut[Index] = Ref[Index] + Updates\n$$\n\n",
"inputs":[
{
"name":"Ref",
"comment":"The source input of scatter op",
"duplicable":0,
"intermediate":0
},{
"name":"Index",
"comment":"The index input of scatter op where Ref will be updated",
"duplicable":0,
"intermediate":0
},{
"name":"Updates",
"comment":"The updated value of updates op",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"The output of add op",
"duplicable":0,
"intermediate":0
}],
"attrs":[]
},{
"type":"uniform_random",
"comment":"\nUniform random operator.\n\nThis operator initializes a tensor with random values sampled from a \nuniform distribution.\n\n",
...
...
@@ -2172,6 +2271,29 @@
"comment":"(int, default 5(FP32)) Output tensor data type",
"generated":0
}]
},{
"type":"logical_xor",
"comment":"logical_xor Operator\n\nIt operates element-wise on X and Y, and returns the Out. X, Y and Out are N-dim boolean tensors.\nEach element of Out is calculated by $$Out = (X || Y) \\, \\&\\& \\, !(X \\&\\& Y)$$\n",
"inputs":[
{
"name":"X",
"comment":"(LoDTensor) Left hand operand of logical_xor operator",
"duplicable":0,
"intermediate":0
},{
"name":"Y",
"comment":"(LoDTensor) Right hand operand of logical_xor operator",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"(LoDTensor) n-dim bool tensor. Each element is $$Out = (X || Y) \\, \\&\\& \\, !(X \\&\\& Y)$$",
"duplicable":0,
"intermediate":0
}],
"attrs":[]
},{
"type":"pad",
"comment":"\nPad Operator.\n\nPad input into output, as specified by paddings and pad_value. \nThe input should be a k-D tensor(k > 0 and k < 7). As an example:\n\nGiven:\n\nX = [[1, 2],\n [3, 4]],\n\npaddings = [0, 1, 1, 2],\n\nand\n\npad_value = 0,\n\nwe have:\n\nOut = [[0, 1, 2, 0, 0]\n [0, 3, 4, 0, 0]\n [0, 0, 0, 0, 0]]\n\n",
...
...
@@ -2868,40 +2990,6 @@
"comment":"The exponential factor of Pow",
"generated":0
}]
},{
"type":"lookup_table",
"comment":"\nLookup Table Operator.\n\nThis operator is used to perform lookups on the parameter W,\nthen concatenated into a dense tensor.\n\nThe input Ids can carry the LoD (Level of Details) information,\nor not. And the output only shares the LoD information with input Ids.\n\n",
"inputs":[
{
"name":"W",
"comment":"An input represents embedding tensors, which is a learnable parameter.",
"duplicable":0,
"intermediate":0
},{
"name":"Ids",
"comment":"An input with type int32 or int64 contains the ids to be looked up in W. Ids must be a column vector with rank = 2. The 2nd dimension size must be 1.",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"The lookup results, which have the same type as W.",
"comment":"(int64, default -1) If the value is -1, it makes no effect to lookup. Otherwise the given value indicates padding the output with zeros whenever lookup encounters it in Ids.",
"comment":"(int, default 5 (FP32)) Output data type",
"generated":0
}]
},{
"type":"logical_and",
"comment":"logical_and Operator\n\nIt operates element-wise on X and Y, and returns the Out. X, Y and Out are N-dim boolean tensors.\nEach element of Out is calculated by $$Out = X \\&\\& Y$$\n",
"inputs":[
{
"name":"X",
"comment":"(LoDTensor) Left hand operand of logical_and operator",
"duplicable":0,
"intermediate":0
},{
"name":"Y",
"comment":"(LoDTensor) Right hand operand of logical_and operator",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"(LoDTensor) n-dim bool tensor. Each element is $$Out = X \\&\\& Y$$",
"duplicable":0,
"intermediate":0
}],
"attrs":[]
},{
"type":"logical_not",
"comment":"logical_not Operator\n\nIt operates element-wise on X, and returns the Out. X and Out are N-dim boolean tensors.\nEach element of Out is calculated by $$Out = !X$$\n",
"inputs":[
{
"name":"X",
"comment":"(LoDTensor) Operand of logical_not operator",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"(LoDTensor) n-dim bool tensor. Each element is $$Out = !X$$",
"comment":"(float, default 1.0)The scaling factor of the scale operator.",
"generated":0
}]
},{
"type":"lstmp",
"comment":"\nLong-Short Term Memory with recurrent Projection layer (LSTMP) Operator.\n\nLSTMP has a separate projection layer after the LSTM layer, projecting the \noriginal hidden state to a lower-dimensional one, which is proposed to reduce \nthe number of total parameters and furthermore computational complexity for \nthe LSTM, espeacially for the case that the size of output units is relative \nlarge (https://research.google.com/pubs/archive/43905.pdf). \n\nThe formula is as follows:\n\n$$\ni_t = \\sigma(W_{ix}x_{t} + W_{ir}r_{t-1} + W_{ic}c_{t-1} + b_i) \\\\\n\nf_t = \\sigma(W_{fx}x_{t} + W_{fr}r_{t-1} + W_{fc}c_{t-1} + b_f) \\\\\n\n\\tilde{c_t} = act_g(W_{cx}x_t + W_{cr}r_{t-1} + b_c) \\\\\n\no_t = \\sigma(W_{ox}x_{t} + W_{or}r_{t-1} + W_{oc}c_t + b_o) \\\\\n\nc_t = f_t \\odot c_{t-1} + i_t \\odot \\tilde{c_t} \\\\\n\nh_t = o_t \\odot act_h(c_t) \\\\\n\nr_t = \\overline{act_h}(W_{rh}h_t)\n$$\n\nwhere the W terms denote weight matrices (e.g. $W_{xi}$ is the matrix\nof weights from the input gate to the input), $W_{ic}, W_{fc}, W_{oc}$\nare diagonal weight matrices for peephole connections. In our implementation,\nwe use vectors to reprenset these diagonal weight matrices. The b terms\ndenote bias vectors ($b_i$ is the input gate bias vector), $\\sigma$\nis the activation, such as logistic sigmoid function, and\n$i, f, o$ and $c$ are the input gate, forget gate, output gate,\nand cell activation vectors, respectively, all of which have the same size as\nthe cell output activation vector $h$. Here $h$ is usually called the hidden \nstate and $r$ denotes its recurrent projection. And $\\tilde{c_t}$ is also \ncalled the candidate hidden state, whose computation is based on the current \ninput and previous hidden state.\n\nThe $\\odot$ is the element-wise product of the vectors. $act_g$ and $act_h$\nare the cell input and cell output activation functions and `tanh` is usually\nused for them. $\\overline{act_h}$ is the activation function for the \nprojection output, usually using `identity` or same as $act_h$.\n\nNote that these $W_{xi}x_{t}, W_{xf}x_{t}, W_{xc}x_{t}, W_{xo}x_{t}$\noperations on the input $x_{t}$ are NOT included in this operator.\nUsers can choose to use fully-connected operator before LSTMP operator.\n\n",
"inputs":[
{
"name":"Input",
"comment":"(LoDTensor) the input for sequence data, which supports variable-time length input sequence. The underlying tensor in this LoDTensor is a matrix with shape (T X 4D), where T is the total time steps in this mini-batch, D is the hidden size.",
"duplicable":0,
"intermediate":0
},{
"name":"H0",
"comment":"(Tensor, optional) the initial hidden state is an optional input. This is a tensor with shape (N x D), where N is the batch size and D is the hidden size.",
"duplicable":0,
"intermediate":0
},{
"name":"C0",
"comment":"(Tensor, optional) the initial cell state is an optional input. This is a tensor with shape (N x D), where N is the batch size. `C0` should not be null if `H0` provided.",
"duplicable":0,
"intermediate":0
},{
"name":"Weight",
"comment":"(Tensor) the learnable hidden-hidden weights. - The shape is (P x 4D), where P is the projection layer size and D is the hidden size. - Weight = {W_cr, W_ir, W_fr, W_or}",
"duplicable":0,
"intermediate":0
},{
"name":"ProjWeight",
"comment":"(Tensor) the learnable weight of the projection layer. - The shape is (D x P), where P is the recurrent projection layer size and D is the hidden size. - ProjWeight = {W_rh}",
"duplicable":0,
"intermediate":0
},{
"name":"Bias",
"comment":"(Tensor) the learnable biases, which contains two parts: input-hidden biases and peephole connections weights if setting `use_peepholes` to `True`. 1. `use_peepholes = False` - The shape is (1 x 4D). - Bias = {b_c, b_i, b_f, b_o}.2. `use_peepholes = True` - The shape is (1 x 7D). - Bias = {b_c, b_i, b_f, b_o, W_ic, W_fc, W_oc}.",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Projection",
"comment":"(LoDTensor) the projection of the hidden state of LSTMP operator. The shape is (T x P), and LoD is the same with the `Input`.",
"duplicable":0,
"intermediate":0
},{
"name":"Cell",
"comment":"(LoDTensor) the cell state of LSTMP operator. The shape is (T x D), and lod is the same with the `Input`.",
"duplicable":0,
"intermediate":0
},{
"name":"BatchGate",
"comment":"(LoDTensor) This LoDTensor contains input gate, forget gate and output gate after the activations. This LoDTensor has the same shape as the reorganized input, which is also be called batch input. The LoD size is 2. The first-level LoD is the batch offsets and the second contains the indices, which denotes the position of reorganized sequence in the raw input.",
"duplicable":0,
"intermediate":1
},{
"name":"BatchCellPreAct",
"comment":"(LoDTensor) the pre-activation cell state reorganized in batch. This LoDTensor is obtained in the forward and used in the backward.",
"duplicable":0,
"intermediate":1
},{
"name":"BatchHidden",
"comment":"(LoDTensor) the hidden state reorganized in batch. This LoDTensor is obtained in the forward and used in the backward.",
"duplicable":0,
"intermediate":1
},{
"name":"OrderedP0",
"comment":"(Tensor) the projection of the initial hidden state H0. This is a tensor with shape (N x P), where N is the batch size and P is the hidden size.",
"duplicable":0,
"intermediate":1
}],
"attrs":[
{
"name":"use_peepholes",
"type":"bool",
"comment":"(bool, defalut: True) whether to enable diagonal/peephole connections.",
"generated":0
},{
"name":"is_reverse",
"type":"bool",
"comment":"(bool, defalut: False) whether to compute reversed LSTMP.",
"generated":0
},{
"name":"gate_activation",
"type":"string",
"comment":"(string, default: sigmoid)The activation for input gate, forget gate and output gate, `sigmoid` by default.",
"generated":0
},{
"name":"cell_activation",
"type":"string",
"comment":"(string, default: tanh)The activation for cell output, `tanh` by defalut.",
"generated":0
},{
"name":"candidate_activation",
"type":"string",
"comment":"(string, default: tanh)The activation for candidate hidden state, `tanh` by default.",
"generated":0
},{
"name":"proj_activation",
"type":"string",
"comment":"(string, default: tanh)The activation for projection output, `tanh` by defalut.",
"generated":0
}]
},{
"type":"mean",
"comment":"\nMean Operator.\n\nOut is a scalar which is the mean of all elements in X. \n\n",
...
...
@@ -3511,106 +3559,58 @@
}],
"attrs":[]
},{
"type":"lod_tensor_to_array",
"comment":"",
"type":"lookup_table",
"comment":"\nLookup Table Operator.\n\nThis operator is used to perform lookups on the parameter W,\nthen concatenated into a dense tensor.\n\nThe input Ids can carry the LoD (Level of Details) information,\nor not. And the output only shares the LoD information with input Ids.\n\n",
"inputs":[
{
"name":"X",
"comment":"",
"name":"W",
"comment":"An input represents embedding tensors, which is a learnable parameter.",
"duplicable":0,
"intermediate":0
},{
"name":"RankTable",
"comment":"",
"name":"Ids",
"comment":"An input with type int32 or int64 contains the ids to be looked up in W. Ids must be a column vector with rank = 2. The 2nd dimension size must be 1.",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"",
"comment":"The lookup results, which have the same type as W.",
"duplicable":0,
"intermediate":0
}],
"attrs":[]
},{
"type":"logical_and",
"comment":"logical_and Operator\n\nIt operates element-wise on X and Y, and returns the Out. X, Y and Out are N-dim boolean tensors.\nEach element of Out is calculated by $$Out = X \\&\\& Y$$\n",
"inputs":[
"attrs":[
{
"name":"X",
"comment":"(LoDTensor) Left hand operand of logical_and operator",
"comment":"(LoDTensor) Right hand operand of logical_and operator",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"(LoDTensor) n-dim bool tensor. Each element is $$Out = X \\&\\& Y$$",
"duplicable":0,
"intermediate":0
}],
"attrs":[]
},{
"type":"logical_not",
"comment":"logical_not Operator\n\nIt operates element-wise on X, and returns the Out. X and Out are N-dim boolean tensors.\nEach element of Out is calculated by $$Out = !X$$\n",
"inputs":[
{
"name":"X",
"comment":"(LoDTensor) Operand of logical_not operator",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"(LoDTensor) n-dim bool tensor. Each element is $$Out = !X$$",
"comment":"(int64, default -1) If the value is -1, it makes no effect to lookup. Otherwise the given value indicates padding the output with zeros whenever lookup encounters it in Ids.",
"generated":0
}]
},{
"type":"logical_xor",
"comment":"logical_xor Operator\n\nIt operates element-wise on X and Y, and returns the Out. X, Y and Out are N-dim boolean tensors.\nEach element of Out is calculated by $$Out = (X || Y) \\, \\&\\& \\, !(X \\&\\& Y)$$\n",
"type":"lod_tensor_to_array",
"comment":"",
"inputs":[
{
"name":"X",
"comment":"(LoDTensor) Left hand operand of logical_xor operator",
"comment":"",
"duplicable":0,
"intermediate":0
},{
"name":"Y",
"comment":"(LoDTensor) Right hand operand of logical_xor operator",
"name":"RankTable",
"comment":"",
"duplicable":0,
"intermediate":0
}],
"outputs":[
{
"name":"Out",
"comment":"(LoDTensor) n-dim bool tensor. Each element is $$Out = (X || Y) \\, \\&\\& \\, !(X \\&\\& Y)$$",
"comment":"\nThe sequence_concat operator concatenates multiple LoDTensors.\nIt only supports sequence (LoD Tensor with level number is 1)\nor a nested sequence (LoD tensor with level number is 2) as its input.\n- Case1:\n If the axis is other than 0(here, axis is 1 and level is 1),\n each input should have the same LoD information and the LoD\n information of the output keeps the same as the input.\n\n LoD(x0) = {{0,2,4}, {0,1,2,3,4}}; Dims(x0) = (4,3,4)\n LoD(x1) = {{0,2,4}, {0,1,2,3,4}}; Dims(x1) = (4,4,4)\n LoD(Out) = {{0,2,4}, {0,1,2,3,4}}; Dims(Out) = (4,7,4)\n\n- Case2:\n If the axis is 0(here, leve is 0), the inputs are concatenated along\n time steps, the LoD information of the output need to re-compute.\n The LoD information of level-1 should be same.\n\n LoD(x0) = {{0,2,4}, {0,1,2,3,4}}; Dims(x0) = (4,3,4)\n LoD(x1) = {{0,2,4}, {0,1,3,5,7}}; Dims(x1) = (7,3,4)\n LoD(Out) = {{0,2,4}, {0,2,5,8,11}}; Dims(Out) = (11,3,4)\n\n- Case3:\n If the axis is 0(here, level is 1).\n\n LoD(x0) = {{0,2,4}, {0,1,2,3,4}}; Dims(x0) = (4,3,4)\n LoD(x1) = {{0,3,4}, {0,1,3,5,7}}; Dims(x1) = (7,3,4)\n LoD(Out) = {{0,5,8}, {0,1,2,3,5,7,8,9,11}}; Dims(Out) = (11,3,4)\n\n- Case4:\n If the LoD number is 1, axis is 0, level is 0\n\n LoD(x0) = {{0,1,2,3,4}}; Dims(x0) = (4,3,4)\n LoD(x1) = {{0,1,3,5,7}}; Dims(x1) = (7,3,4)\n LoD(Out) = {{0,2,5,8,11}}; Dims(Out) = (11,3,4)\n\nNOTE: The levels of all the inputs should be the same.\n ",
...
...
@@ -5209,6 +5191,24 @@
"comment":"(int, default 0) The level at which the inputs will be joined. If the level is 0, the inputs will be joined at the nested sequence level. If the level is 1, the inputs will be joined at the sequence level. The level should be less than the level number of inputs.",
<p>The defalut implementation is diagonal/peephole connection
(<aclass="reference external"href="https://arxiv.org/pdf/1402.1128.pdf">https://arxiv.org/pdf/1402.1128.pdf</a>), the formula is as follows:</p>
...
...
@@ -387,7 +387,7 @@ with zeros whenever lookup encounters it in <code class="xref py py-attr docutil
the matrix of weights from the input gate to the input), <spanclass="math">\(W_{ic}, W_{fc}, W_{oc}\)</span> are diagonal weight matrices for peephole connections. In
our implementation, we use vectors to reprenset these diagonal weight
matrices. The <spanclass="math">\(b\)</span> terms denote bias vectors (<spanclass="math">\(b_i\)</span> is the input
gate bias vector), <spanclass="math">\(\sigma\)</span> is the non-line activations, such as
gate bias vector), <spanclass="math">\(\sigma\)</span> is the non-linear activations, such as
logistic sigmoid function, and <spanclass="math">\(i, f, o\)</span> and <spanclass="math">\(c\)</span> are the input
gate, forget gate, output gate, and cell activation vectors, respectively,
all of which have the same size as the cell output activation vector <spanclass="math">\(h\)</span>.</p>
...
...
@@ -413,15 +413,15 @@ tensor in this Variable is a matrix with shape
<p>LSTMP (LSTM with recurrent projection) layer has a separate projection
layer after the LSTM layer, projecting the original hidden state to a
lower-dimensional one, which is proposed to reduce the number of total
parameters and furthermore computational complexity for the LSTM,
espeacially for the case that the size of output units is relative
large (<aclass="reference external"href="https://research.google.com/pubs/archive/43905.pdf">https://research.google.com/pubs/archive/43905.pdf</a>).</p>
<p>The formula is as follows:</p>
<divclass="math">
\[ \begin{align}\begin{aligned}i_t & = \sigma(W_{ix}x_{t} + W_{ir}r_{t-1} + W_{ic}c_{t-1} + b_i)\\f_t & = \sigma(W_{fx}x_{t} + W_{fr}r_{t-1} + W_{fc}c_{t-1} + b_f)\\\tilde{c_t} & = act_g(W_{cx}x_t + W_{cr}r_{t-1} + b_c)\\o_t & = \sigma(W_{ox}x_{t} + W_{or}r_{t-1} + W_{oc}c_t + b_o)\\c_t & = f_t \odot c_{t-1} + i_t \odot \tilde{c_t}\\h_t & = o_t \odot act_h(c_t)\\r_t & = \overline{act_h}(W_{rh}h_t)\end{aligned}\end{align} \]</div>
<p>In the above formula:</p>
<ulclass="simple">
<li><spanclass="math">\(W\)</span>: Denotes weight matrices (e.g. <spanclass="math">\(W_{xi}\)</span> is the matrix of weights from the input gate to the input).</li>
<li><spanclass="math">\(W_{ic}\)</span>, <spanclass="math">\(W_{fc}\)</span>, <spanclass="math">\(W_{oc}\)</span>: Diagonal weight matrices for peephole connections. In our implementation, we use vectors to reprenset these diagonal weight matrices.</li>
<li><spanclass="math">\(b\)</span>: Denotes bias vectors (e.g. <spanclass="math">\(b_i\)</span> is the input gate bias vector).</li>
<li><spanclass="math">\(\sigma\)</span>: The activation, such as logistic sigmoid function.</li>
<li><spanclass="math">\(i, f, o\)</span> and <spanclass="math">\(c\)</span>: The input gate, forget gate, output gate, and cell activation vectors, respectively, all of which have the same size as the cell output activation vector <spanclass="math">\(h\)</span>.</li>
<li><spanclass="math">\(h\)</span>: The hidden state.</li>
<li><spanclass="math">\(r\)</span>: The recurrent projection of the hidden state.</li>
<li><spanclass="math">\(\tilde{c_t}\)</span>: The candidate hidden state, whose computation is based on the current input and previous hidden state.</li>
<li><spanclass="math">\(\odot\)</span>: The element-wise product of the vectors.</li>
<li><spanclass="math">\(act_g\)</span> and <spanclass="math">\(act_h\)</span>: The cell input and cell output activation functions and <cite>tanh</cite> is usually used for them.</li>
<li><spanclass="math">\(\overline{act_h}\)</span>: The activation function for the projection output, usually using <cite>identity</cite> or same as <spanclass="math">\(act_h\)</span>.</li>
</ul>
<p>Set <cite>use_peepholes</cite> to <cite>False</cite> to disable peephole connection. The formula
is omitted here, please refer to the paper
<aclass="reference external"href="http://www.bioinf.jku.at/publications/older/2604.pdf">http://www.bioinf.jku.at/publications/older/2604.pdf</a> for details.</p>
<p>Note that these <spanclass="math">\(W_{xi}x_{t}, W_{xf}x_{t}, W_{xc}x_{t}, W_{xo}x_{t}\)</span>
operations on the input <spanclass="math">\(x_{t}\)</span> are NOT included in this operator.
Users can choose to use fully-connected layer before LSTMP layer.</p>
<li><strong>name</strong> (<em>str|None</em>) – A name for this layer(optional). If set None, the layer
will be named automatically.</li>
</ul>
</td>
</tr>
<trclass="field-even field"><thclass="field-name">返回:</th><tdclass="field-body"><pclass="first">The projection of hidden state, and cell state of LSTMP. The shape of projection is (T x P), for the cell state which is (T x D), and both LoD is the same with the <cite>input</cite>.</p>
