word2vec done

chapter_natural-language-processing/word2vec.md

@@ -128,19 +128,54 @@ $$ \text{log} \mathbb{P} (w_o \mid w_c) = \text{log} \frac{1}{1+\text{exp}(-\mat
 $$ - \text{log} \mathbb{P} (w_o \mid w_c) = -\text{log} \frac{1}{1+\text{exp}(-\mathbf{u}_o^\top \mathbf{v}_c)}  - \sum_{k=1, w_k \sim \mathbb{P}(w)}^K \text{log} \frac{1}{1+\text{exp}(\mathbf{u}_{i_k}^\top \mathbf{v}_c)} $$
 
 
-当我们把$K$取较小值时，每次随机梯度下降的梯度计算开销将较小。
+当我们把$K$取较小值时，每次随机梯度下降的梯度计算开销将由$\mathcal{O}(|\mathcal{V}|)$降为$\mathcal{O}(K)$。
 
+我们也可以对连续词袋模型进行负采样。有关背景词$w^{(t-m)}, \ldots,  w^{(t-1)},  w^{(t+1)}, \ldots,  w^{(t+m)}$生成中心词$w_c$的损失函数
+
+$$-\text{log} \mathbb{P}(w^{(t)} \mid  w^{(t-m)}, \ldots,  w^{(t-1)},  w^{(t+1)}, \ldots,  w^{(t+m)})$$
+
+在负采样中可以近似为
+
+$$-\text{log} \frac{1}{1+\text{exp}[-\mathbf{u}_c^\top (\mathbf{v}_{o_1} + \ldots + \mathbf{v}_{o_{2m}}) /(2m)]}  - \sum_{k=1, w_k \sim \mathbb{P}(w)}^K \text{log} \frac{1}{1+\text{exp}[(\mathbf{u}_{i_k}^\top (\mathbf{v}_{o_1} + \ldots + \mathbf{v}_{o_{2m}}) /(2m)]}$$
+
+同样地，当我们把$K$取较小值时，每次随机梯度下降的梯度计算开销将由$\mathcal{O}(|\mathcal{V}|)$降为$\mathcal{O}(K)$。
+
+
+## 层序softmax
+
+层序softmax利用了二叉树。树的每个叶子节点代表着词典$\mathcal{V}$中的每个词。每个词$w_i$相应的词向量为$\mathbf{v}_i$。我们以下图为例，来描述层序softmax的工作机制。
+
+
+![](../img/hierarchical_softmax.svg)
+
+
+假设$L(w)$为从二叉树的根到代表词$w$的叶子节点的路径上的节点数，并设$n(w,i)$为该路径上第$i$个节点，该节点的向量为$\mathbf{u}_{n(w,i)}$。以上图为例，$L(w_3) = 4$。那么，跳字模型和连续词袋模型所需要计算的任意词$w_i$生成词$w$的概率为：
+
+$$\mathbb{P}(w \mid w_i) = \prod_{j=1}^{L(w)-1} \sigma([n(w, j+1) = \text{left_child}(n(w,j))] \cdot \mathbf{u}_{n(w,j)}^\top \mathbf{v}_i)$$
+
+其中$\sigma(x) = 1/(1+\text{exp}(-x))$，如果$x$为真，$[x] = 1$；反之$[x] = -1$。
+
+由于$\sigma(x)+\sigma(-x) = 1$，$w_i$生成词典中任何词的概率之和为1：
+
+$$\sum_{w=1}^{\mathcal{V}} \mathbb{P}(w \mid w_i) = 1$$
+
+
+让我们计算$w_i$生成$w_3$的概率，由于在二叉树中由根到$w_3$的路径上需要向左、向右、再向左地遍历，我们得到
+
+$$\mathbb{P}(w_3 \mid w_i) = \sigma(\mathbf{u}_{n(w_3,1)}^\top \mathbf{v}_i)) \cdot \sigma(-\mathbf{u}_{n(w_3,2)}^\top \mathbf{v}_i)) \cdot \sigma(\mathbf{u}_{n(w_3,3)}^\top \mathbf{v}_i))$$
+
+我们可以使用随机梯度下降在跳字模型和连续词袋模型中不断迭代计算字典中所有词向量$\mathbf{v}$和非叶子节点的向量$\mathbf{u}$。每次迭代的计算开销由$\mathcal{O}(|\mathcal{V}|)$降为二叉树的高度$\mathcal{O}(\text{log}|\mathcal{V}|)$。
 
 
 
 ## 结论
 
-word2vec工具主要包含两个模型：跳字模型和连续词袋模型，以及两种高效训练的方法：负采样和层序softmax。
+word2vec工具中的跳字模型和连续词袋模型可以使用两种负采样和层序softmax减小训练开销。
 
 
 ## 练习
 
-* 噪声词$\mathbb{P}(w)$在实际中被建议设为$w$的单字概率的3/4次方。为什么？（想想0.99^(3/4)和0.01^(3/4)的大小）
+* 噪声词$\mathbb{P}(w)$在实际中被建议设为$w$的单字概率的3/4次方。为什么？（想想$0.99^{3/4}$和$0.01^{3/4}$的大小）
 * 一些"the"和"a"之类的英文高频词会对结果产生什么影响？该如何处理？（可参考[word2vec论文](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)第2.3节）
 * 如何训练包括例如"new york"在内的词组向量？（可参考[word2vec论文](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)第4节）。
 

img/hierarchical_softmax.svg

+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="553px" height="280px" version="1.1" content="&lt;mxfile userAgent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36&quot; version=&quot;8.0.1&quot; editor=&quot;www.draw.io&quot; type=&quot;device&quot;&gt;&lt;diagram id=&quot;675fec1c-a3c6-69fd-dd79-093af1883ec1&quot; name=&quot;Page-1&quot;&gt;7Ztbb5swGIZ/TaTtYpEPHC/btNtuJk2qtF0TcAgqwRk4TbpfPxNsjo5CmgRYQqW28PnI+3w2PjHBs9XuW+yslz+oR8IJAt5ugp8mCNk64H9Tw3tmMDHODH4ceJkJFoaX4C8RRpHO3wQeSSoRGaUhC9ZVo0ujiLisYnPimG6r0RY0rJa6dnzSMLy4Tti0/g48tsyslnys1P6dBP5SlgyBCJk77qsf000kypsgvNj/ZMErR+Yl4idLx6Pbkgk/T/AsppRlV6vdjISptFK2LN3XA6F5vWMSsTYJUJbgzQk3RNZ4Xy/2LrXgCbjs/OZxuwwYeVk7bhqy5eC5bclWIb+D/NJJ1hmLRbAjPP9HP3SSRAQmLKavuZgatyyCMJzRkMb7YrCnE8vT8pilEAvNsWHwEFFXEjOyO/i8MFeROyehK8Lidx5FJMBYz5IIx9QEh21BWZqWJcDS5gi/8vOMC235hZBXLTW+M6mhDSpS522kA621e9Ma1bQ2utNaH4bWEY3IZcS0qlrmfXUHWhq3piXU+xPTvDUxEepPTOvmxLT7E9MehpidvZ6Q3t/rSU4p7kZszaoOcbscd8GbGwzoNTW77CbgQIYDnbmurtVct8t+os0bLvIe0tWFwsNKAh/VcGG5xHVVGs4t/uQg15B4jdWJmoK8VnQTuyKWmFkyJ/aJiIXUOpd01BU6SltMQocFb9VKqMQVJfykAa/eoQE0xjU8Wd1FovLiRC0fiGuvaKuWUfbEjYz2qPOnbkW/1cj7NPofY6k1WeKR5Wks2wz8u2CpN1lqI8vTWF6+V/4YS2NkeTbLNtOeYb5h7bFXPpe+LOk/pG826dsj/dPow4H049bI8myWaCAs5dLOOPE5A2abbclrwSyDg6BPcvz1ME2XzgzNALbJLyr6Q8OcAgwME1iWBSAC1vC5ttkC7auRZgtbYyttT7PPKe1wyOkantqahkxoA5v/wKr+CE0hhJpuQAsAAC00fK5Dmd5CxVrF2EpPpama4BohS6cqdF+nAqvxZ0NlwJdkfwDugUfAYL0rAvmVn/7fymySzVzaoLTxWpXMJWtWpjTXvIqRHVO5Um0HRbGp4oSBH/Fbl/sT4fbHdMk/cJ3wQQSsAs9Li1HuSOzPxqX7D0+gtgchtLvkyQ15f2T7AIHDztn6xJFqensae13Jfjqd3jVTTatBbbmdeRGoqlnrtRo0Ght08/hQHX6XLVo1zb0WfDzCP37cqUv4qmnxteBrI/zjx7O6hK+aO18L/sSc/eK/owscP3rTpQuoDjJdwgWiTyd1/zP4+a59ANVmb4rvNexruYDqSEgfLoDu2wVafEZyNRdQLbv14QL4vl0Ag5oPKM4FXsgH+G3x9V22BlR84Yif/wE=&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g transform="translate(0.5,0.5)"><ellipse cx="277" cy="30" rx="20" ry="20" fill="#d5e8d4" stroke="#82b366" stroke-width="4" pointer-events="none"/><ellipse cx="132" cy="90" rx="20" ry="20" fill="#d5e8d4" stroke="#82b366" stroke-width="4" pointer-events="none"/><ellipse cx="62" cy="150" rx="20" ry="20" fill="#d5e8d4" stroke="#82b366" stroke-width="4" pointer-events="none"/><ellipse cx="22" cy="210" rx="20" ry="20" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><ellipse cx="92" cy="210" rx="20" ry="20" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><ellipse cx="162" cy="210" rx="20" ry="20" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><ellipse cx="232" cy="210" rx="20" ry="20" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><ellipse cx="192" cy="150" rx="20" ry="20" fill="#d5e8d4" stroke="#82b366" stroke-width="4" pointer-events="none"/><ellipse cx="427" cy="90" rx="20" ry="20" fill="#d5e8d4" stroke="#82b366" stroke-width="4" pointer-events="none"/><ellipse cx="527" cy="210" rx="20" ry="20" fill="none" stroke="#000000" stroke-width="4" pointer-events="none"/><ellipse cx="487" cy="150" rx="20" ry="20" fill="#d5e8d4" stroke="#82b366" stroke-width="4" pointer-events="none"/><path d="M 150.49 82.37 L 258.52 37.65" fill="none" stroke="#b85450" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 77.22 137.03 L 116.81 103.02" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 32.91 193.24 L 50.91 166.64" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 83.06 192.11 L 70.94 167.89" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 177.86 135.86 L 146.14 104.14" fill="none" stroke="#b85450" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 170.94 192.11 L 183.06 167.89" fill="none" stroke="#b85450" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 221.09 193.24 L 203.09 166.64" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 408.41 82.62 L 295.57 37.43" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 372.22 137.03 L 411.91 103.13" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 441.14 104.14 L 472.86 135.86" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 465.94 192.11 L 478.06 167.89" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><path d="M 516.09 193.24 L 498.09 166.64" fill="none" stroke="#000000" stroke-width="4" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(3.5,239.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="36" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 36px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">w<sub>1</sub></font></div></div></foreignObject><text x="18" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(361.5,150.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="41" height="58" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 43px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 50px">...</font></div></div></foreignObject><text x="21" y="35" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(63.5,239.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="36" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 36px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">w<sub>2</sub></font></div></div></foreignObject><text x="18" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(143.5,239.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="36" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 36px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">w<sub>3</sub></font></div></div></foreignObject><text x="18" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(213.5,239.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="36" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 36px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">w<sub>4</sub></font></div></div></foreignObject><text x="18" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(500.5,239.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="52" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 52px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">w<sub>|V|</sub></font></div></div></foreignObject><text x="26" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(147.5,-0.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="98" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 98px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">n(w<sub>3</sub>,1)</font></div></div></foreignObject><text x="49" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(7.5,59.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="98" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 98px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">n(w<sub>3</sub>,2)</font></div></div></foreignObject><text x="49" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><g transform="translate(217.5,119.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="98" height="40" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 98px; white-space: nowrap; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;"><font style="font-size: 30px">n(w<sub>3</sub>,3)</font></div></div></foreignObject><text x="49" y="26" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g></g></svg>
\ No newline at end of file