提交 4d6f4df7 编写于 作者: T Travis CI

Deploy to GitHub Pages: 4615c517

上级 f444865b
...@@ -202,14 +202,17 @@ After training the model, we can use it to generate sequences. A common practice ...@@ -202,14 +202,17 @@ After training the model, we can use it to generate sequences. A common practice
* use :code:`GeneratedInput` for trg_embedding. :code:`GeneratedInput` computes the embedding of the generated token at the last time step for the input at the current time step. * use :code:`GeneratedInput` for trg_embedding. :code:`GeneratedInput` computes the embedding of the generated token at the last time step for the input at the current time step.
* use :code:`beam_search` function. This function needs to set: * use :code:`beam_search` function. This function needs to set:
- :code:`id_input`: the integer ID of the data, used to identify the corresponding output in the generated files.
- :code:`dict_file`: the dictionary file for converting word id to word.
- :code:`bos_id`: the start token. Every sentence starts with the start token. - :code:`bos_id`: the start token. Every sentence starts with the start token.
- :code:`eos_id`: the end token. Every sentence ends with the end token. - :code:`eos_id`: the end token. Every sentence ends with the end token.
- :code:`beam_size`: the beam size used in beam search. - :code:`beam_size`: the beam size used in beam search.
- :code:`max_length`: the maximum length of the generated sentences. - :code:`max_length`: the maximum length of the generated sentences.
- :code:`result_file`: the path of the generation result file.
* use :code:`seqtext_printer_evaluator` to print text according to index matrix and dictionary. This function needs to set:
- :code:`id_input`: the integer ID of the data, used to identify the corresponding output in the generated files.
- :code:`dict_file`: the dictionary file for converting word id to word.
- :code:`result_file`: the path of the generation result file.
The code is listed below: The code is listed below:
.. code-block:: python .. code-block:: python
...@@ -230,14 +233,15 @@ The code is listed below: ...@@ -230,14 +233,15 @@ The code is listed below:
beam_gen = beam_search(name=decoder_group_name, beam_gen = beam_search(name=decoder_group_name,
step=gru_decoder_with_attention, step=gru_decoder_with_attention,
input=group_inputs, input=group_inputs,
id_input=data_layer(name="sent_id",
size=1),
dict_file=trg_dict_path,
bos_id=0, # Beginnning token. bos_id=0, # Beginnning token.
eos_id=1, # End of sentence token. eos_id=1, # End of sentence token.
beam_size=beam_size, beam_size=beam_size,
max_length=max_length, max_length=max_length)
result_file=gen_trans_file)
seqtext_printer_evaluator(input=beam_gen,
id_input=data_layer(name="sent_id", size=1),
dict_file=trg_dict_path,
result_file=gen_trans_file)
outputs(beam_gen) outputs(beam_gen)
......
...@@ -200,12 +200,15 @@ Its <strong>output function</strong> simply takes <span class="math">\(x_t\)</sp ...@@ -200,12 +200,15 @@ Its <strong>output function</strong> simply takes <span class="math">\(x_t\)</sp
<ul class="simple"> <ul class="simple">
<li>use <code class="code docutils literal"><span class="pre">GeneratedInput</span></code> for trg_embedding. <code class="code docutils literal"><span class="pre">GeneratedInput</span></code> computes the embedding of the generated token at the last time step for the input at the current time step.</li> <li>use <code class="code docutils literal"><span class="pre">GeneratedInput</span></code> for trg_embedding. <code class="code docutils literal"><span class="pre">GeneratedInput</span></code> computes the embedding of the generated token at the last time step for the input at the current time step.</li>
<li>use <code class="code docutils literal"><span class="pre">beam_search</span></code> function. This function needs to set:<ul> <li>use <code class="code docutils literal"><span class="pre">beam_search</span></code> function. This function needs to set:<ul>
<li><code class="code docutils literal"><span class="pre">id_input</span></code>: the integer ID of the data, used to identify the corresponding output in the generated files.</li>
<li><code class="code docutils literal"><span class="pre">dict_file</span></code>: the dictionary file for converting word id to word.</li>
<li><code class="code docutils literal"><span class="pre">bos_id</span></code>: the start token. Every sentence starts with the start token.</li> <li><code class="code docutils literal"><span class="pre">bos_id</span></code>: the start token. Every sentence starts with the start token.</li>
<li><code class="code docutils literal"><span class="pre">eos_id</span></code>: the end token. Every sentence ends with the end token.</li> <li><code class="code docutils literal"><span class="pre">eos_id</span></code>: the end token. Every sentence ends with the end token.</li>
<li><code class="code docutils literal"><span class="pre">beam_size</span></code>: the beam size used in beam search.</li> <li><code class="code docutils literal"><span class="pre">beam_size</span></code>: the beam size used in beam search.</li>
<li><code class="code docutils literal"><span class="pre">max_length</span></code>: the maximum length of the generated sentences.</li> <li><code class="code docutils literal"><span class="pre">max_length</span></code>: the maximum length of the generated sentences.</li>
</ul>
</li>
<li>use <code class="code docutils literal"><span class="pre">seqtext_printer_evaluator</span></code> to print text according to index matrix and dictionary. This function needs to set:<ul>
<li><code class="code docutils literal"><span class="pre">id_input</span></code>: the integer ID of the data, used to identify the corresponding output in the generated files.</li>
<li><code class="code docutils literal"><span class="pre">dict_file</span></code>: the dictionary file for converting word id to word.</li>
<li><code class="code docutils literal"><span class="pre">result_file</span></code>: the path of the generation result file.</li> <li><code class="code docutils literal"><span class="pre">result_file</span></code>: the path of the generation result file.</li>
</ul> </ul>
</li> </li>
...@@ -227,14 +230,15 @@ Its <strong>output function</strong> simply takes <span class="math">\(x_t\)</sp ...@@ -227,14 +230,15 @@ Its <strong>output function</strong> simply takes <span class="math">\(x_t\)</sp
<span class="n">beam_gen</span> <span class="o">=</span> <span class="n">beam_search</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">decoder_group_name</span><span class="p">,</span> <span class="n">beam_gen</span> <span class="o">=</span> <span class="n">beam_search</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">decoder_group_name</span><span class="p">,</span>
<span class="n">step</span><span class="o">=</span><span class="n">gru_decoder_with_attention</span><span class="p">,</span> <span class="n">step</span><span class="o">=</span><span class="n">gru_decoder_with_attention</span><span class="p">,</span>
<span class="nb">input</span><span class="o">=</span><span class="n">group_inputs</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">group_inputs</span><span class="p">,</span>
<span class="n">id_input</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;sent_id&quot;</span><span class="p">,</span>
<span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span>
<span class="n">dict_file</span><span class="o">=</span><span class="n">trg_dict_path</span><span class="p">,</span>
<span class="n">bos_id</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="c1"># Beginnning token.</span> <span class="n">bos_id</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="c1"># Beginnning token.</span>
<span class="n">eos_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="c1"># End of sentence token.</span> <span class="n">eos_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="c1"># End of sentence token.</span>
<span class="n">beam_size</span><span class="o">=</span><span class="n">beam_size</span><span class="p">,</span> <span class="n">beam_size</span><span class="o">=</span><span class="n">beam_size</span><span class="p">,</span>
<span class="n">max_length</span><span class="o">=</span><span class="n">max_length</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="n">max_length</span><span class="p">)</span>
<span class="n">result_file</span><span class="o">=</span><span class="n">gen_trans_file</span><span class="p">)</span>
<span class="n">seqtext_printer_evaluator</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">beam_gen</span><span class="p">,</span>
<span class="n">id_input</span><span class="o">=</span><span class="n">data_layer</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;sent_id&quot;</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span>
<span class="n">dict_file</span><span class="o">=</span><span class="n">trg_dict_path</span><span class="p">,</span>
<span class="n">result_file</span><span class="o">=</span><span class="n">gen_trans_file</span><span class="p">)</span>
<span class="n">outputs</span><span class="p">(</span><span class="n">beam_gen</span><span class="p">)</span> <span class="n">outputs</span><span class="p">(</span><span class="n">beam_gen</span><span class="p">)</span>
</pre></div> </pre></div>
</div> </div>
......
因为 它太大了无法显示 source diff 。你可以改为 查看blob
...@@ -499,10 +499,11 @@ It contains one or more input layers.</p> ...@@ -499,10 +499,11 @@ It contains one or more input layers.</p>
<code class="descclassname">paddle.trainer_config_helpers.evaluators.</code><code class="descname">seqtext_printer_evaluator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span></dt> <code class="descclassname">paddle.trainer_config_helpers.evaluators.</code><code class="descname">seqtext_printer_evaluator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span></dt>
<dd><p>Sequence text printer will print text according to index matrix and a <dd><p>Sequence text printer will print text according to index matrix and a
dictionary. There can be multiple input to this layer:</p> dictionary. There can be multiple input to this layer:</p>
<p>1. If there is only one input, the input must be a matrix containing <p>1. If there is no id_input, the input must be a matrix containing
the sequence of indices;</p> the sequence of indices;</p>
<p>2. If there are more than one input, the first input should be ids, <ol class="arabic simple" start="2">
and are interpreted as sample ids.</p> <li>If there is id_input, it should be ids, and interpreted as sample ids.</li>
</ol>
<p>The output format will be:</p> <p>The output format will be:</p>
<ol class="arabic simple"> <ol class="arabic simple">
<li>sequence without sub-sequence, and there is probability.</li> <li>sequence without sub-sequence, and there is probability.</li>
...@@ -527,7 +528,8 @@ and are interpreted as sample ids.</p> ...@@ -527,7 +528,8 @@ and are interpreted as sample ids.</p>
<p>Typically SequenceTextPrinter layer takes output of maxid or RecurrentGroup <p>Typically SequenceTextPrinter layer takes output of maxid or RecurrentGroup
with maxid (when generating) as an input.</p> with maxid (when generating) as an input.</p>
<p>The simple usage is:</p> <p>The simple usage is:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="nb">eval</span> <span class="o">=</span> <span class="n">seqtext_printer_evaluator</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <div class="highlight-python"><div class="highlight"><pre><span></span><span class="nb">eval</span> <span class="o">=</span> <span class="n">seqtext_printer_evaluator</span><span class="p">(</span><span class="nb">input</span><span class="o">=</span><span class="n">maxid_layer</span><span class="p">,</span>
<span class="n">id_input</span><span class="o">=</span><span class="n">sample_id</span><span class="p">,</span>
<span class="n">dict_file</span><span class="o">=</span><span class="n">dict_file</span><span class="p">,</span> <span class="n">dict_file</span><span class="o">=</span><span class="n">dict_file</span><span class="p">,</span>
<span class="n">result_file</span><span class="o">=</span><span class="n">result_file</span><span class="p">)</span> <span class="n">result_file</span><span class="o">=</span><span class="n">result_file</span><span class="p">)</span>
</pre></div> </pre></div>
...@@ -536,16 +538,29 @@ with maxid (when generating) as an input.</p> ...@@ -536,16 +538,29 @@ with maxid (when generating) as an input.</p>
<col class="field-name" /> <col class="field-name" />
<col class="field-body" /> <col class="field-body" />
<tbody valign="top"> <tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>input</strong> (<em>LayerOutput|list</em>) &#8211; Input Layer name.</li> <li><strong>input</strong> (<em>LayerOutput|list</em>) &#8211; Input Layer name.</li>
<li><strong>dict_file</strong> (<em>basestring</em>) &#8211; The input dictionary which contains a list of tokens.</li> <li><strong>result_file</strong> (<em>basestring</em>) &#8211; Path of the file to store the generated results.</li>
<li><strong>result_file</strong> (<em>basestring</em>) &#8211; The file is to save the results.</li> <li><strong>id_input</strong> (<em>LayerOutput</em>) &#8211; Index of the input sequence, and the specified index will
be prited in the gereated results. This an optional
parameter.</li>
<li><strong>dict_file</strong> (<em>basestring</em>) &#8211; Path of dictionary. This is an optional parameter.
Every line is a word in the dictionary with
(line number - 1) as the word index.
If this parameter is set to None, or to an empty string,
only word index are printed in the generated results.</li>
<li><strong>delimited</strong> (<em>bool</em>) &#8211; Whether to use space to separate output tokens. <li><strong>delimited</strong> (<em>bool</em>) &#8211; Whether to use space to separate output tokens.
Default is True. No space is added if set to False.</li> Default is True. No space is added if set to False.</li>
<li><strong>name</strong> (<em>None|basestring</em>) &#8211; Evaluator name.</li> <li><strong>name</strong> (<em>None|basestring</em>) &#8211; Evaluator name.</li>
</ul> </ul>
</td> </td>
</tr> </tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The seq_text_printer that prints the generated sequence to a file.</p>
</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last">evaluator</p>
</td>
</tr>
</tbody> </tbody>
</table> </table>
</dd></dl> </dd></dl>
......
...@@ -978,8 +978,7 @@ to maintain tractability.</p> ...@@ -978,8 +978,7 @@ to maintain tractability.</p>
<span class="nb">input</span><span class="o">=</span><span class="p">[</span><span class="n">StaticInput</span><span class="p">(</span><span class="n">encoder_last</span><span class="p">)],</span> <span class="nb">input</span><span class="o">=</span><span class="p">[</span><span class="n">StaticInput</span><span class="p">(</span><span class="n">encoder_last</span><span class="p">)],</span>
<span class="n">bos_id</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">bos_id</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
<span class="n">eos_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">eos_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
<span class="n">beam_size</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">beam_size</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">result_file</span><span class="o">=</span><span class="s2">&quot;./generated_sequences.txt&quot;</span><span class="p">)</span>
</pre></div> </pre></div>
</div> </div>
<p>Please see the following demo for more details:</p> <p>Please see the following demo for more details:</p>
...@@ -1015,25 +1014,16 @@ algorithm. To maintain tractability, every iteration only ...@@ -1015,25 +1014,16 @@ algorithm. To maintain tractability, every iteration only
only stores a predetermined number, called the beam_size, only stores a predetermined number, called the beam_size,
of the most promising next words. The greater the beam of the most promising next words. The greater the beam
size, the fewer candidate words are pruned.</li> size, the fewer candidate words are pruned.</li>
<li><strong>result_file</strong> (<em>basestring</em>) &#8211; Path of the file to store the generated results.</li>
<li><strong>dict_file</strong> (<em>basestring</em>) &#8211; Path of dictionary. This is an optional parameter.
Every line is a word in the dictionary with
(line number - 1) as the word index.
If this parameter is set to None, or to an empty string,
only word index are printed in the generated results.</li>
<li><strong>num_results_per_sample</strong> (<em>int</em>) &#8211; Number of the generated results per input <li><strong>num_results_per_sample</strong> (<em>int</em>) &#8211; Number of the generated results per input
sequence. This number must always be less than sequence. This number must always be less than
beam size.</li> beam size.</li>
<li><strong>id_input</strong> (<em>LayerOutput</em>) &#8211; Index of the input sequence, and the specified index will
be prited in the gereated results. This an optional
parameter.</li>
</ul> </ul>
</td> </td>
</tr> </tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The seq_text_printer that prints the generated sequence to a file.</p> <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The generated word index.</p>
</td> </td>
</tr> </tr>
<tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last">evaluator</p> <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last">LayerOutput</p>
</td> </td>
</tr> </tr>
</tbody> </tbody>
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册