rnn gluon with multilayer plot

chapter_recurrent-neural-networks/rnn-gluon.md

@@ -139,6 +139,18 @@ class RNNModel(gluon.Block):
 
 对于一个多层循环神经网络，当前时刻隐含层的输入来自同一时刻输入层（如果有）或上一隐含层的输出。每一层的隐含状态只沿着同一层传递。
 
+把[单层循环神经网络](rnn-scratch.md)中隐含层的每个单元当做一个函数$f$，这个函数在$t$时刻的输入是$\mathbf{X}_t, \mathbf{H}_{t-1}$，输出是$\mathbf{H}_t$：
+
+$$f(\mathbf{X}_t, \mathbf{H}_{t-1}) = \mathbf{H}_t$$
+
+假设输入为第0层，输出为第$L+1$层，在一共$L$个隐含层的循环神经网络中，上式中可以拓展成以下的函数:
+
+$$f(\mathbf{H}_t^{(l-1)}, \mathbf{H}_{t-1}^{(l)}) = \mathbf{H}_t$$
+
+如下图所示。
+
+![](../img/multi-layer-rnn.svg)
+
 ```{.python .input}
 model_name = 'rnn_relu'
 

img/multi-layer-rnn.svg

+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="409px" height="496px" version="1.1" content="&lt;mxfile userAgent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36&quot; version=&quot;7.9.4&quot; editor=&quot;www.draw.io&quot; type=&quot;device&quot;&gt;&lt;diagram id=&quot;8989ad3f-6651-7bfb-3f0a-49771e91a30c&quot; name=&quot;Page-1&quot;&gt;7VzLcqM4FP0aV80sJgUS+LGMk3T3IlM1VVnMWjYyphsjBuTYnq8fgcRDPGLituS2hiwSuFwkuOfoHnGlygQ+7Y5fExRv/yQeDifA8o4T+DwBYAEs9jsznLhh7ky5wU8Cj5vsyvAW/IuFUdzn7wMPp5IjJSSkQSwb1ySK8JpKNpQk5CC7bUgo9xojH7cMb2sUtq1/Bx7dirdwrcr+DQf+tujZtsSVFVr/8BOyj0R/EwA3+Q+/vENFW8I/3SKPHGom+DKBTwkhlB/tjk84zEJbhI3f96XnavncCY7okBsELO8o3OPiifPnoqciFthjoRGnJKFb4pMIhS+VdZm/L85atNjZlu5CdmizQxx5jxka2Z0xjrjlSxCGwvU7pvQk0Ed7Spip6uGVkFi0k9KE/CiRcJil/Z4FTVDiY2FyuSl7gZqPCMRXTHaYJifmkOAQ0eBdxh4JCvmlX3nrXyRgvQLrWKLPbxFkt6eW3ERK9skai7sqMNhB7TEqUw5RD76uyXgt9OAFZvrwMnp82bYewByoDzDLNMB4bKSEqHTMScH/KNLz/0GkdUsPdBojg0N9jZExQHkO24DitxjlQTiwiaEMCUpjPlfbBMcMsqUfojTtjfOGgfVEQpKw84hE+KPQv+OE4uOHkRZXG7m/HPCH2jRPmLa1Gd7c6odmKOOBYzLj7dltKO+6rirKD+H8/QMmKfrt5GBmZH5xCg5pyC82MJmuiw62Xn3+OTTUCzPICuAN1dC4TyMpuXbQFdyMrsDoufb1RWtY3UDhzMM2NMFolUPbjBjCxe2SNDS6HHL9hDysfqXyk8VQ0utMHLArhtOQ9bBcSZGc/rPP1mmWGxLRP9Kcpo/MAczjY3WRHfnib1g4X97Kt6KZdB8Xtt+s3wsre7Xahcp3VZhsyXPV9GRW/nwtc821QSeGKJUZxEnSIEUHT1AY+GzQP68ZTzCzLzN+BGsUPooLu8Dz8gzSRVk5q9RJegX+Nehng3bOdTroB65BP2gu/cBIv2HzJveG/OsqUxrCPzjybxD/nOaStk7+dVVdf2X+vU7A0h4lWLEEu7P2DFAZBbtqUwZRcJThy2RYKwe7FkQM4uAoxZdJsU4OOv0cHEogOO0i0MPDQy/QZ0FNMGserXKHLOBZZYh3yOsXg7GMs4JLHh53OXGfu6nRSSGliM9dCXAI23MvtwNweAXAi3bNqfdp34Eza9T2evamnW2orHidKRKygKJTzU2QevADO5b7ueeS/dkBf4KL9xUNqFiOlPtoHaqcKX+Wcs2GVO4eA6OS6FcS4NxQSrrKmOO4/sS4vlhKejchXVlKWv2ckZIz/j8vJcZtsNS9MnmxlDQbUikl/fXBUUrUfYZaN5QS47aK6R7XF0tJsyFVUtLq54yUnPH/aSmZdn0I/8rVt3EBQvUCRFll01B4m97bFpTXkX6K6ee4+uq+03vbgjKufSlf+9Ka/u5tC8q47qV83Usr/+5uC8qY/1TnP636e3f7T8b8pzr/KeQfO63+4xD/WK7+qxN8+Q8=&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g transform="translate(0.5,0.5)"><path d="M 44 50 L 44 91.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 39 85.53 L 44 95.53 L 49 85.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 204 50 L 204 91.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 199 85.53 L 204 95.53 L 209 85.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 364 50 L 364 91.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 359 85.53 L 364 95.53 L 369 85.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 84 140 L 155.06 140" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 149.53 145 L 159.53 140 L 149.53 135" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 44 180 L 44 221.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 39 215.53 L 44 225.53 L 49 215.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><rect x="4" y="100" width="80" height="80" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><path d="M 44 395 L 44 436.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 39 430.53 L 44 440.53 L 49 430.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 84 355 L 155.06 355" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 149.53 360 L 159.53 355 L 149.53 350" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><rect x="4" y="315" width="80" height="80" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><path d="M 244 140 L 315.06 140" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 309.53 145 L 319.53 140 L 309.53 135" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><rect x="164" y="100" width="80" height="80" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><path d="M 244 355 L 315.06 355" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 309.53 360 L 319.53 355 L 309.53 350" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 204 395 L 204 436.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 199 430.53 L 204 440.53 L 209 430.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><rect x="164" y="315" width="80" height="80" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><rect x="324" y="100" width="80" height="80" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><path d="M 364 395 L 364 436.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 359 430.53 L 364 440.53 L 369 430.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><rect x="324" y="315" width="80" height="80" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><g transform="translate(13.5,-0.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 63px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(0)</sup><sub>1</sub></font></b></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(173.5,-0.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 63px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(0)</sup><sub>2</sub></font></b></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(333.5,-0.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 63px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(0)</sup><sub>3</sub></font></b></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(-0.5,454.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="89" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 91px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(L+1)</sup><sub>1</sub></font></b></div></div></foreignObject><text x="45" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(159.5,454.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="89" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 91px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(L+1)</sup><sub>2</sub></font></b></div></div></foreignObject><text x="45" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(319.5,454.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="89" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 91px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(L+1)</sup><sub>3</sub></font></b></div></div></foreignObject><text x="45" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(28.5,214.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="30" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 36px">...</font></div></div></foreignObject><text x="15" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 44 265 L 44 295 L 44 306.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 39 300.53 L 44 310.53 L 49 300.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 204 180 L 204 221.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 199 215.53 L 204 225.53 L 209 215.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(188.5,214.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="30" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 36px">...</font></div></div></foreignObject><text x="15" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 204 265 L 204 295 L 204 306.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 199 300.53 L 204 310.53 L 209 300.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 364 180 L 364 221.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 359 215.53 L 364 225.53 L 369 215.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(348.5,214.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="30" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 36px">...</font></div></div></foreignObject><text x="15" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 364 265 L 364 295 L 364 306.06" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 359 300.53 L 364 310.53 L 369 300.53" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(13.5,119.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 63px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(1)</sup><sub>1</sub></font></b></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(11.5,334.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="64" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 64px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(L)</sup><sub>1</sub></font></b></div></div></foreignObject><text x="32" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(173.5,119.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 63px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(1)</sup><sub>2</sub></font></b></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(333.5,119.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 63px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(1)</sup><sub>3</sub></font></b></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(171.5,334.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="64" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 64px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(L)</sup><sub>2</sub></font></b></div></div></foreignObject><text x="32" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(331.5,334.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="64" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 64px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><b style="font-size: 28px"><font style="font-size: 28px">H<sup>(L)</sup><sub>3</sub></font></b></div></div></foreignObject><text x="32" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g></g></svg>
\ No newline at end of file