Skip to content

A
Annotated Deep Learning Paper Implementations

Greenplum / Annotated Deep Learning Paper Implementations
10 个月 前同步成功

通知 6
Star 0
Fork 0
A
Annotated Deep Learning Paper Implementations

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

切换分支/标签
查找文件 普通视图历史永久链接Permalink
index.html 29.8 KB
Newer Older
V
translations  
Varuna Jayasiri 已提交
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 
<!DOCTYPE html>
<html lang="zh">
<head>
    <meta http-equiv="content-type" content="text/html;charset=utf-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <meta name="description" content="记录了实现，并解释了 Transformer-XL 模型。"/>

    <meta name="twitter:card" content="summary"/>
    <meta name="twitter:image:src" content="https://avatars1.githubusercontent.com/u/64068543?s=400&amp;v=4"/>
    <meta name="twitter:title" content="变压器 XL"/>
    <meta name="twitter:description" content="记录了实现，并解释了 Transformer-XL 模型。"/>
    <meta name="twitter:site" content="@labmlai"/>
    <meta name="twitter:creator" content="@labmlai"/>

    <meta property="og:url" content="https://nn.labml.ai/transformers/xl/index.html"/>
    <meta property="og:title" content="变压器 XL"/>
    <meta property="og:image" content="https://avatars1.githubusercontent.com/u/64068543?s=400&amp;v=4"/>
    <meta property="og:site_name" content="变压器 XL"/>
    <meta property="og:type" content="object"/>
    <meta property="og:title" content="变压器 XL"/>
    <meta property="og:description" content="记录了实现，并解释了 Transformer-XL 模型。"/>

    <title>变压器 XL</title>
    <link rel="shortcut icon" href="/icon.png"/>
    <link rel="stylesheet" href="../../pylit.css?v=1">
    <link rel="canonical" href="https://nn.labml.ai/transformers/xl/index.html"/>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.13.18/dist/katex.min.css" integrity="sha384-zTROYFVGOfTw7JV7KUu8udsvW2fx4lWOsCEDqhBreBwlHI4ioVRtmIvEThzJHGET" crossorigin="anonymous">

    <!-- Global site tag (gtag.js) - Google Analytics -->
    <script async src="https://www.googletagmanager.com/gtag/js?id=G-4V3HC8HBLH"></script>
    <script>
        window.dataLayer = window.dataLayer || [];

        function gtag() {
            dataLayer.push(arguments);
        }

        gtag('js', new Date());

        gtag('config', 'G-4V3HC8HBLH');
    </script>
</head>
<body>
<div id='container'>
    <div id="background"></div>
    <div class='section'>
        <div class='docs'>
            <p>
                <a class="parent" href="/">home</a>
                <a class="parent" href="../index.html">transformers</a>
                <a class="parent" href="index.html">xl</a>
            </p>
            <p>
                <a href="https://github.com/labmlai/annotated_deep_learning_paper_implementations" target="_blank">
                    <img alt="Github"
                         src="https://img.shields.io/github/stars/labmlai/annotated_deep_learning_paper_implementations?style=social"
                         style="max-width:100%;"/></a>
                <a href="https://twitter.com/labmlai" rel="nofollow" target="_blank">
                    <img alt="Twitter"
                         src="https://img.shields.io/twitter/follow/labmlai?style=social"
                         style="max-width:100%;"/></a>
            </p>
            <p>
                <a href="https://github.com/labmlai/annotated_deep_learning_paper_implementations/tree/master/labml_nn/transformers/xl/__init__.py" target="_blank">
                    View code on Github</a>
            </p>
        </div>
    </div>
    <div class='section' id='section-0'>
        <div class='docs doc-strings'>
            <div class='section-link'>
                <a href='#section-0'>#</a>
            </div>
            <h1>变压器 XL</h1>
V
zh translation  
Varuna Jayasiri 已提交
75 76 77 78 79 80 
<p>这是 <a href="https://pytorch.org">PyTorch 中 Transfor</a> <a href="https://papers.labml.ai/paper/1901.02860">mer-XL：超越固定长度上下文的专心语言模型</a>的实现。</p>
<p>Transformer 的注意力跨度有限，等于并行训练序列的长度。所有这些位置都有固定的位置编码。Transformer XL 通过让每个位置关注过去预先计算的嵌入次数，从而延长了这种注意力跨度。例如，如果上下文长度为<span ><span class="katex"><span aria-hidden="true" class="katex-html"><span class="base"><span class="strut" style="height:0.69444em;vertical-align:0em;"></span><span class="mord coloredeq eqa" style=""><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span></span></span>，它将保留前一批长度的所有层的嵌入<span ><span class="katex"><span aria-hidden="true" class="katex-html"><span class="base"><span class="strut" style="height:0.69444em;vertical-align:0em;"></span><span class="mord coloredeq eqa" style=""><span class="mord mathnormal" style="margin-right:0.01968em">l</span></span></span></span></span></span>并将其馈送到当前步骤。如果我们使用固定位置编码，这些预先计算的嵌入将与当前上下文具有相同的位置。它们引入了相对位置编码，其中位置编码是在注意力计算时引入的。</p>
<p>相对多头注意力的带注释的实现已经开始<a href="relative_mha.html"><code  class="highlight"><span></span><span class="n">relative_mha</span><span class="o">.</span><span class="n">py</span></code>
</a>了。</p>
<p>这是用于<a href="experiment.html">在 Tiny Shakespeare 数据集上训练 transformer XL 模型的训练代码</a>和笔记本。</p>
<p><a href="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/xl/experiment.ipynb"><img alt="Open In Colab" src="https://colab.research.google.com/assets/colab-badge.svg"></a></p>
V
translations  
Varuna Jayasiri 已提交
81 82 83 

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
84 85 86 87 88 89 90 91 92 
            <div class="highlight"><pre><span class="lineno">35</span><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span><span class="p">,</span> <span class="n">Optional</span>
<span class="lineno">36</span>
<span class="lineno">37</span><span class="kn">import</span> <span class="nn">torch</span>
<span class="lineno">38</span><span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
<span class="lineno">39</span>
<span class="lineno">40</span><span class="kn">from</span> <span class="nn">labml_helpers.module</span> <span class="kn">import</span> <span class="n">Module</span>
<span class="lineno">41</span><span class="kn">from</span> <span class="nn">labml_nn.utils</span> <span class="kn">import</span> <span class="n">clone_module_list</span>
<span class="lineno">42</span><span class="kn">from</span> <span class="nn">.relative_mha</span> <span class="kn">import</span> <span class="n">RelativeMultiHeadAttention</span>
<span class="lineno">43</span><span class="kn">from</span> <span class="nn">..feed_forward</span> <span class="kn">import</span> <span class="n">FeedForward</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
93 94 95 96 97 98 99 100 101 102 103 104 
        </div>
    </div>
    <div class='section' id='section-1'>
        <div class='docs doc-strings'>
            <div class='section-link'>
                <a href='#section-1'>#</a>
            </div>
            <h2>变压器 XL 层</h2>
<p>变压器 XL 模型由许多这样的层组成。</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
105 
            <div class="highlight"><pre><span class="lineno">46</span><span class="k">class</span> <span class="nc">TransformerXLLayer</span><span class="p">(</span><span class="n">Module</span><span class="p">):</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 
        </div>
    </div>
    <div class='section' id='section-2'>
        <div class='docs doc-strings'>
            <div class='section-link'>
                <a href='#section-2'>#</a>
            </div>
            <ul><li><code  class="highlight"><span></span><span class="n">d_model</span></code>
是令牌嵌入的大小</li>
<li><code  class="highlight"><span></span><span class="n">self_attn</span></code>
是<a href="relative_mha.html">自我关注模块</a></li>
<li><code  class="highlight"><span></span><span class="n">feed_forward</span></code>
是前馈模块</li>
<li><code  class="highlight"><span></span><span class="n">dropout_prob</span></code>
是自我关注和 FFN 后退学的概率</li></ul>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
124 125 126 127 128 
            <div class="highlight"><pre><span class="lineno">52</span>    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span>
<span class="lineno">53</span>                 <span class="n">d_model</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
<span class="lineno">54</span>                 <span class="n">self_attn</span><span class="p">:</span> <span class="n">RelativeMultiHeadAttention</span><span class="p">,</span>
<span class="lineno">55</span>                 <span class="n">feed_forward</span><span class="p">:</span> <span class="n">FeedForward</span><span class="p">,</span>
<span class="lineno">56</span>                 <span class="n">dropout_prob</span><span class="p">:</span> <span class="nb">float</span><span class="p">):</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
129 130 131 132 133 134 135 136 137 138 
        </div>
    </div>
    <div class='section' id='section-3'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-3'>#</a>
            </div>
            
        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
139 140 141 142 143 144 145 
            <div class="highlight"><pre><span class="lineno">63</span>        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
<span class="lineno">64</span>        <span class="bp">self</span><span class="o">.</span><span class="n">size</span> <span class="o">=</span> <span class="n">d_model</span>
<span class="lineno">65</span>        <span class="bp">self</span><span class="o">.</span><span class="n">self_attn</span> <span class="o">=</span> <span class="n">self_attn</span>
<span class="lineno">66</span>        <span class="bp">self</span><span class="o">.</span><span class="n">feed_forward</span> <span class="o">=</span> <span class="n">feed_forward</span>
<span class="lineno">67</span>        <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="n">dropout_prob</span><span class="p">)</span>
<span class="lineno">68</span>        <span class="bp">self</span><span class="o">.</span><span class="n">norm_self_attn</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">LayerNorm</span><span class="p">([</span><span class="n">d_model</span><span class="p">])</span>
<span class="lineno">69</span>        <span class="bp">self</span><span class="o">.</span><span class="n">norm_ff</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">LayerNorm</span><span class="p">([</span><span class="n">d_model</span><span class="p">])</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 
        </div>
    </div>
    <div class='section' id='section-4'>
        <div class='docs doc-strings'>
            <div class='section-link'>
                <a href='#section-4'>#</a>
            </div>
            <ul><li><code  class="highlight"><span></span><span class="n">x</span></code>
是令牌级特征形状向量的张量<code  class="highlight"><span></span><span class="p">[</span><span class="n">seq_len</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">d_model</span><span class="p">]</span></code>
</li>
<li><code  class="highlight"><span></span><span class="n">mem</span></code>
是过去令牌级别特征形状向量的张量<code  class="highlight"><span></span><span class="p">[</span><span class="n">mem_len</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">d_model</span><span class="p">]</span></code>
</li>
<li><code  class="highlight"><span></span><span class="n">mask</span></code>
是形状的矩阵<code  class="highlight"><span></span><span class="p">[</span><span class="n">seq_len</span><span class="p">,</span> <span class="n">mem_len</span> <span class="o">+</span> <span class="n">seq_len</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">]</span></code>
或<code  class="highlight"><span></span><span class="p">[</span><span class="n">seq_len</span><span class="p">,</span> <span class="n">mem_len</span> <span class="o">+</span> <span class="n">seq_len</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span></code>
。<code  class="highlight"><span></span><span class="n">mask</span><span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">]</span></code>
如果 token<code  class="highlight"><span></span><span class="n">i</span></code>
 可以在处看到令牌，则为 true<code  class="highlight"><span></span><span class="n">j</span></code>
。</li></ul>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
169 170 171 172 
            <div class="highlight"><pre><span class="lineno">71</span>    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span>
<span class="lineno">72</span>                <span class="n">x</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span>
<span class="lineno">73</span>                <span class="n">mem</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">],</span>
<span class="lineno">74</span>                <span class="n">mask</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">):</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
173 174 175 176 177 178 179 180 181 182 183 
        </div>
    </div>
    <div class='section' id='section-5'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-5'>#</a>
            </div>
            <p>在进行自我注意之前对向量进行归一化</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
184 
            <div class="highlight"><pre><span class="lineno">82</span>        <span class="n">z</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">norm_self_attn</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
185 186 187 188 189 190 191 192 193 194 195 
        </div>
    </div>
    <div class='section' id='section-6'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-6'>#</a>
            </div>
            <p>如果有记忆</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
196 
            <div class="highlight"><pre><span class="lineno">84</span>        <span class="k">if</span> <span class="n">mem</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
197 198 199 200 201 202 203 204 205 206 207 
        </div>
    </div>
    <div class='section' id='section-7'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-7'>#</a>
            </div>
            <p>规范化它</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
208 
            <div class="highlight"><pre><span class="lineno">86</span>            <span class="n">mem</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">norm_self_attn</span><span class="p">(</span><span class="n">mem</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
209 210 211 212 213 214 215 216 217 218 219 220 
        </div>
    </div>
    <div class='section' id='section-8'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-8'>#</a>
            </div>
            <p>连接与<code  class="highlight"><span></span><span class="n">z</span></code>
</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
221 
            <div class="highlight"><pre><span class="lineno">88</span>            <span class="n">m_z</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">((</span><span class="n">mem</span><span class="p">,</span> <span class="n">z</span><span class="p">),</span> <span class="n">dim</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
222 223 224 225 226 227 228 229 230 231 232 
        </div>
    </div>
    <div class='section' id='section-9'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-9'>#</a>
            </div>
            <p>如果没有内存，则忽略</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
233 234 
            <div class="highlight"><pre><span class="lineno">90</span>        <span class="k">else</span><span class="p">:</span>
<span class="lineno">91</span>            <span class="n">m_z</span> <span class="o">=</span> <span class="n">z</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
235 236 237 238 239 240 241 242 243 244 245 
        </div>
    </div>
    <div class='section' id='section-10'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-10'>#</a>
            </div>
            <p>注意</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
246 
            <div class="highlight"><pre><span class="lineno">93</span>        <span class="n">self_attn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">self_attn</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">z</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">m_z</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">m_z</span><span class="p">,</span> <span class="n">mask</span><span class="o">=</span><span class="n">mask</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
247 248 249 250 251 252 253 254 255 256 257 
        </div>
    </div>
    <div class='section' id='section-11'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-11'>#</a>
            </div>
            <p>添加关注结果</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
258 
            <div class="highlight"><pre><span class="lineno">95</span>        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span><span class="p">(</span><span class="n">self_attn</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
259 260 261 262 263 264 265 266 267 268 269 
        </div>
    </div>
    <div class='section' id='section-12'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-12'>#</a>
            </div>
            <p>标准化以进行前馈</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
270 
            <div class="highlight"><pre><span class="lineno">98</span>        <span class="n">z</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">norm_ff</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
271 272 273 274 275 276 277 278 279 280 281 
        </div>
    </div>
    <div class='section' id='section-13'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-13'>#</a>
            </div>
            <p>通过前馈网络</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
282 
            <div class="highlight"><pre><span class="lineno">100</span>        <span class="n">ff</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">feed_forward</span><span class="p">(</span><span class="n">z</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
283 284 285 286 287 288 289 290 291 292 293 
        </div>
    </div>
    <div class='section' id='section-14'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-14'>#</a>
            </div>
            <p>将前馈结果添加回来</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
294 
            <div class="highlight"><pre><span class="lineno">102</span>        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">dropout</span><span class="p">(</span><span class="n">ff</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
295 296 297 298 299 300 301 302 303 304 305 
        </div>
    </div>
    <div class='section' id='section-15'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-15'>#</a>
            </div>
            <p></p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
306 
            <div class="highlight"><pre><span class="lineno">105</span>        <span class="k">return</span> <span class="n">x</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
307 308 309 310 311 312 313 314 315 316 317 318 
        </div>
    </div>
    <div class='section' id='section-16'>
        <div class='docs doc-strings'>
            <div class='section-link'>
                <a href='#section-16'>#</a>
            </div>
            <h2>变压器 XL 型号</h2>
<p>它由多个变压器 XL 层组成</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
319 
            <div class="highlight"><pre><span class="lineno">108</span><span class="k">class</span> <span class="nc">TransformerXL</span><span class="p">(</span><span class="n">Module</span><span class="p">):</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
320 321 322 323 324 325 326 327 328 329 
        </div>
    </div>
    <div class='section' id='section-17'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-17'>#</a>
            </div>
            
        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
330 331 
            <div class="highlight"><pre><span class="lineno">115</span>    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">layer</span><span class="p">:</span> <span class="n">TransformerXLLayer</span><span class="p">,</span> <span class="n">n_layers</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="lineno">116</span>        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
332 333 334 335 336 337 338 339 340 341 342 
        </div>
    </div>
    <div class='section' id='section-18'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-18'>#</a>
            </div>
            <p>制作变压器层的副本</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
343 
            <div class="highlight"><pre><span class="lineno">118</span>        <span class="bp">self</span><span class="o">.</span><span class="n">layers</span> <span class="o">=</span> <span class="n">clone_module_list</span><span class="p">(</span><span class="n">layer</span><span class="p">,</span> <span class="n">n_layers</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
344 345 346 347 348 349 350 351 352 353 354 
        </div>
    </div>
    <div class='section' id='section-19'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-19'>#</a>
            </div>
            <p>最终归一化层</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
355 
            <div class="highlight"><pre><span class="lineno">120</span>        <span class="bp">self</span><span class="o">.</span><span class="n">norm</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">LayerNorm</span><span class="p">([</span><span class="n">layer</span><span class="o">.</span><span class="n">size</span><span class="p">])</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 
        </div>
    </div>
    <div class='section' id='section-20'>
        <div class='docs doc-strings'>
            <div class='section-link'>
                <a href='#section-20'>#</a>
            </div>
            <ul><li><code  class="highlight"><span></span><span class="n">x</span></code>
是嵌入形状向量的令牌的张量<code  class="highlight"><span></span><span class="p">[</span><span class="n">seq_len</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">d_model</span><span class="p">]</span></code>
</li>
<li><code  class="highlight"><span></span><span class="n">mem</span></code>
是过去令牌级别的张量列表，每个层的形状<code  class="highlight"><span></span><span class="p">[</span><span class="n">mem_len</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">,</span> <span class="n">d_model</span><span class="p">]</span></code>
向量特征</li>
<li><code  class="highlight"><span></span><span class="n">mask</span></code>
是掩码矩阵</li></ul>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
374 
            <div class="highlight"><pre><span class="lineno">122</span>    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">mem</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">],</span> <span class="n">mask</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">):</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
375 376 377 378 379 380 381 382 383 384 385 
        </div>
    </div>
    <div class='section' id='section-21'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-21'>#</a>
            </div>
            <p>用于存储令牌级特征向量的列表，这些向量将成为下一个连续批次的记忆。</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
386 
            <div class="highlight"><pre><span class="lineno">131</span>        <span class="n">new_mem</span> <span class="o">=</span> <span class="p">[]</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
387 388 389 390 391 392 393 394 395 396 397 
        </div>
    </div>
    <div class='section' id='section-22'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-22'>#</a>
            </div>
            <p>穿过每个变压器层</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
398 
            <div class="highlight"><pre><span class="lineno">133</span>        <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">layer</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layers</span><span class="p">):</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
399 400 401 402 403 404 405 406 407 408 409 
        </div>
    </div>
    <div class='section' id='section-23'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-23'>#</a>
            </div>
            <p>添加到特征向量列表中</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
410 
            <div class="highlight"><pre><span class="lineno">135</span>            <span class="n">new_mem</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">detach</span><span class="p">())</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
411 412 413 414 415 416 417 418 419 420 421 
        </div>
    </div>
    <div class='section' id='section-24'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-24'>#</a>
            </div>
            <p>记忆</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
422 
            <div class="highlight"><pre><span class="lineno">137</span>            <span class="n">m</span> <span class="o">=</span> <span class="n">mem</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">if</span> <span class="n">mem</span> <span class="k">else</span> <span class="kc">None</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
423 424 425 426 427 428 429 430 431 432 433 
        </div>
    </div>
    <div class='section' id='section-25'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-25'>#</a>
            </div>
            <p>穿过变压器 XL 层</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
434 
            <div class="highlight"><pre><span class="lineno">139</span>            <span class="n">x</span> <span class="o">=</span> <span class="n">layer</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="n">x</span><span class="p">,</span> <span class="n">mem</span><span class="o">=</span><span class="n">m</span><span class="p">,</span> <span class="n">mask</span><span class="o">=</span><span class="n">mask</span><span class="p">)</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
435 436 437 438 439 440 441 442 443 444 445 
        </div>
    </div>
    <div class='section' id='section-26'>
        <div class='docs'>
            <div class='section-link'>
                <a href='#section-26'>#</a>
            </div>
            <p>最后，对向量进行归一化</p>

        </div>
        <div class='code'>
V
zh translation  
Varuna Jayasiri 已提交
446 
            <div class="highlight"><pre><span class="lineno">141</span>        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">norm</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">new_mem</span></pre></div>
V
translations  
Varuna Jayasiri 已提交
447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 
        </div>
    </div>
    <div class='footer'>
        <a href="https://papers.labml.ai">Trending Research Papers</a>
        <a href="https://labml.ai">labml.ai</a>
    </div>
</div>
<script src=../../interactive.js?v=1"></script>
<script>
    function handleImages() {
        var images = document.querySelectorAll('p>img')

        for (var i = 0; i < images.length; ++i) {
            handleImage(images[i])
        }
    }

    function handleImage(img) {
        img.parentElement.style.textAlign = 'center'

        var modal = document.createElement('div')
        modal.id = 'modal'

        var modalContent = document.createElement('div')
        modal.appendChild(modalContent)

        var modalImage = document.createElement('img')
        modalContent.appendChild(modalImage)

        var span = document.createElement('span')
        span.classList.add('close')
        span.textContent = 'x'
        modal.appendChild(span)

        img.onclick = function () {
            console.log('clicked')
            document.body.appendChild(modal)
            modalImage.src = img.src
        }

        span.onclick = function () {
            document.body.removeChild(modal)
        }
    }

    handleImages()
</script>
</body>
</html>