`
+
+
+
+## 模型配置说明
+
+### 数据定义
+
+通过`define_py_data_sources2`函数从dataprovider中读入数据，其中args指定了训练文本(srcText)和词汇表(dictfile)。
+
+```python
+from paddle.trainer_config_helpers import *
+import math
+
+args = {'srcText': 'data/simple-examples/data/ptb.train.txt',
+ 'dictfile': 'data/vocabulary.txt'}
+
+define_py_data_sources2(
+ train_list="data/train.list",
+ test_list="data/test.list",
+ module="dataprovider",
+ obj="process",
+ args=args)
+```
+
+### 算法配置
+
+在这里，我们指定了模型的训练参数, L2正则项系数、学习率和batch size。
+
+```python
+settings(
+ batch_size=100, regularization=L2Regularization(8e-4), learning_rate=3e-3)
+```
+
+### 模型结构
+
+本配置的模型结构如下图所示：
+
+
+ 
+ 图5. 模型配置中的N-gram神经网络模型
+
+
+1. 定义参数维度和和数据输入。
+
+ ```python
+ dictsize = 1953 # 字典大小
+ embsize = 32 # 词向量维度
+ hiddensize = 256 # 隐层维度
+
+ firstword = data_layer(name = "firstw", size = dictsize)
+ secondword = data_layer(name = "secondw", size = dictsize)
+ thirdword = data_layer(name = "thirdw", size = dictsize)
+ fourthword = data_layer(name = "fourthw", size = dictsize)
+ nextword = data_layer(name = "fifthw", size = dictsize)
+ ```
+
+2. 将$w_t$之前的$n-1$个词 $w_{t-n+1},...w_{t-1}$，通过$|V|\times D$的矩阵映射到D维词向量（本例中取D=32）。
+
+ ```python
+ def wordemb(inlayer):
+ wordemb = table_projection(
+ input = inlayer,
+ size = embsize,
+ param_attr=ParamAttr(name = "_proj",
+ initial_std=0.001, # 参数初始化标准差
+ l2_rate= 0,)) # 词向量不需要稀疏化，因此其l2_rate设为0
+ return wordemb
+
+ Efirst = wordemb(firstword)
+ Esecond = wordemb(secondword)
+ Ethird = wordemb(thirdword)
+ Efourth = wordemb(fourthword)
+ ```
+
+3. 接着，将这n-1个词向量经过concat_layer连接成一个大向量作为历史文本特征。
+
+ ```python
+ contextemb = concat_layer(input = [Efirst, Esecond, Ethird, Efourth])
+ ```
+4. 然后，将历史文本特征经过一个全连接得到文本隐层特征。
+
+ ```python
+ hidden1 = fc_layer(
+ input = contextemb,
+ size = hiddensize,
+ act = SigmoidActivation(),
+ layer_attr = ExtraAttr(drop_rate=0.5),
+ bias_attr = ParamAttr(learning_rate = 2),
+ param_attr = ParamAttr(
+ initial_std = 1./math.sqrt(embsize*8),
+ learning_rate = 1))
+ ```
+
+5. 最后，将文本隐层特征，再经过一个全连接，映射成一个$|V|$维向量，同时通过softmax归一化得到这`|V|`个词的生成概率。
+
+ ```python
+ # use context embedding to predict nextword
+ predictword = fc_layer(
+ input = hidden1,
+ size = dictsize,
+ bias_attr = ParamAttr(learning_rate = 2),
+ act = SoftmaxActivation())
+ ```
+
+6. 网络的损失函数为多分类交叉熵，可直接调用`classification_cost`函数。
+
+ ```python
+ cost = classification_cost(
+ input = predictword,
+ label = nextword)
+ # network input and output
+ outputs(cost)
+ ```
+
+##训练模型
+
+模型训练命令为`./train.sh`。脚本内容如下，其中指定了总共需要执行30个pass。
+
+```bash
+paddle train \
+ --config ngram.py \
+ --use_gpu=1 \
+ --dot_period=100 \
+ --log_period=3000 \
+ --test_period=0 \
+ --save_dir=model \
+ --num_passes=30
+```
+
+一个pass的训练日志如下所示：
+
+```text
+.............................
+I1222 09:27:16.477841 12590 TrainerInternal.cpp:162] Batch=3000 samples=300000 AvgCost=5.36135 CurrentCost=5.36135 Eval: classification_error_evaluator=0.818653 CurrentEval: class
+ification_error_evaluator=0.818653
+.............................
+I1222 09:27:22.416700 12590 TrainerInternal.cpp:162] Batch=6000 samples=600000 AvgCost=5.29301 CurrentCost=5.22467 Eval: classification_error_evaluator=0.814542 CurrentEval: class
+ification_error_evaluator=0.81043
+.............................
+I1222 09:27:28.343756 12590 TrainerInternal.cpp:162] Batch=9000 samples=900000 AvgCost=5.22494 CurrentCost=5.08876 Eval: classification_error_evaluator=0.810088 CurrentEval: class
+ification_error_evaluator=0.80118
+..I1222 09:27:29.128582 12590 TrainerInternal.cpp:179] Pass=0 Batch=9296 samples=929600 AvgCost=5.21786 Eval: classification_error_evaluator=0.809647
+I1222 09:27:29.627616 12590 Tester.cpp:111] Test samples=73760 cost=4.9594 Eval: classification_error_evaluator=0.79676
+I1222 09:27:29.627713 12590 GradientMachine.cpp:112] Saving parameters to model/pass-00000
+```
+经过30个pass，我们将得到平均错误率为classification_error_evaluator=0.735611。
+
+
+## 应用模型
+训练模型后，我们可以加载模型参数，用训练出来的词向量初始化其他模型，也可以将模型参数从二进制格式转换成文本格式进行后续应用。
+
+### 初始化其他模型
+
+训练好的模型参数可以用来初始化其他模型。具体方法如下：
+在PaddlePaddle 训练命令行中，用`--init_model_path` 来定义初始化模型的位置，用`--load_missing_parameter_strategy`指定除了词向量以外的新模型其他参数的初始化策略。注意，新模型需要和原模型共享被初始化参数的参数名。
+
+### 查看词向量
+PaddlePaddle训练出来的参数为二进制格式，存储在对应训练pass的文件夹下。这里我们提供了文件`format_convert.py`用来互转PaddlePaddle训练结果的二进制文件和文本格式特征文件。
+
+```bash
+python format_convert.py --b2t -i INPUT -o OUTPUT -d DIM
+```
+其中，INPUT是输入的（二进制）词向量模型名称，OUTPUT是输出的文本模型名称，DIM是词向量参数维度。
+
+用法如：
+
+```bash
+python format_convert.py --b2t -i model/pass-00029/_proj -o model/pass-00029/_proj.txt -d 32
+```
+转换后得到的文本文件如下：
+
+```text
+0,4,62496
+-0.7444070,-0.1846171,-1.5771370,0.7070392,2.1963732,-0.0091410, ......
+-0.0721337,-0.2429973,-0.0606297,0.1882059,-0.2072131,-0.7661019, ......
+......
+```
+
+其中，第一行是PaddlePaddle 输出文件的格式说明，包含3个属性：
+1) PaddlePaddle的版本号，本例中为0;
+2) 浮点数占用的字节数，本例中为4;
+3) 总计的参数个数, 本例中为62496（即1953*32）;
+第二行及之后的每一行都按顺序表示字典里一个词的特征，用逗号分隔。
+
+### 修改词向量
+
+我们可以对词向量进行修改，并转换成PaddlePaddle参数二进制格式，方法：
+
+```bash
+python format_convert.py --t2b -i INPUT -o OUTPUT
+```
+
+其中，INPUT是输入的输入的文本词向量模型名称，OUTPUT是输出的二进制词向量模型名称
+
+输入的文本格式如下（注意，不包含上面二进制转文本后第一行的格式说明）：
+
+```text
+-0.7444070,-0.1846171,-1.5771370,0.7070392,2.1963732,-0.0091410, ......
+-0.0721337,-0.2429973,-0.0606297,0.1882059,-0.2072131,-0.7661019, ......
+......
+```
+
+
+
+### 计算词语之间的余弦距离
+
+两个向量之间的距离可以用余弦值来表示，余弦值在$[-1,1]$的区间内，向量间余弦值越大，其距离越近。这里我们在`calculate_dis.py`中实现不同词语的距离度量。
+用法如下：
+
+```bash
+python calculate_dis.py VOCABULARY EMBEDDINGLAYER`
+```
+
+其中，`VOCABULARY`是字典，`EMBEDDINGLAYER`是词向量模型，示例如下：
+
+```bash
+python calculate_dis.py data/vocabulary.txt model/pass-00029/_proj.txt
+```
+
+
+## 总结
+本章中，我们介绍了词向量、语言模型和词向量的关系、以及如何通过训练神经网络模型获得词向量。在信息检索中，我们可以根据向量间的余弦夹角，来判断query和文档关键词这二者间的相关性。在句法分析和语义分析中，训练好的词向量可以用来初始化模型，以得到更好的效果。在文档分类中，有了词向量之后，可以用聚类的方法将文档中同义词进行分组。希望大家在本章后能够自行运用词向量进行相关领域的研究。
+
+
+## 参考文献
+1. Bengio Y, Ducharme R, Vincent P, et al. [A neural probabilistic language model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)[J]. journal of machine learning research, 2003, 3(Feb): 1137-1155.
+2. Mikolov T, Kombrink S, Deoras A, et al. [Rnnlm-recurrent neural network language modeling toolkit](http://www.fit.vutbr.cz/~imikolov/rnnlm/rnnlm-demo.pdf)[C]//Proc. of the 2011 ASRU Workshop. 2011: 196-201.
+3. Mikolov T, Chen K, Corrado G, et al. [Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf)[J]. arXiv preprint arXiv:1301.3781, 2013.
+4. Maaten L, Hinton G. [Visualizing data using t-SNE](https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf)[J]. Journal of Machine Learning Research, 2008, 9(Nov): 2579-2605.
+5. https://en.wikipedia.org/wiki/Singular_value_decomposition
diff --git a/word2vec/calculate_dis.py b/word2vec/calculate_dis.py
new file mode 100755
index 0000000000000000000000000000000000000000..a18e5ffee11a3178526255bc27ab4479583d408e
--- /dev/null
+++ b/word2vec/calculate_dis.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Example:
+ python calculate_dis.py DICTIONARYTXT FEATURETXT
+
+Required arguments:
+ DICTIONARYTXT the dictionary generated in dataprovider
+ FEATURETXT the text format word feature, one line for one word
+"""
+
+import numpy as np
+from argparse import ArgumentParser
+
+
+def load_dict(fdict):
+ words = [line.strip() for line in fdict.readlines()]
+ dictionary = dict(zip(words, xrange(len(words))))
+ return dictionary
+
+
+def load_emb(femb):
+ feaBank = []
+ flag_firstline = True
+ for line in femb:
+ if flag_firstline:
+ flag_firstline = False
+ continue
+ fea = np.array([float(x) for x in line.strip().split(',')])
+ normfea = fea * 1.0 / np.linalg.norm(fea)
+ feaBank.append(normfea)
+ return feaBank
+
+
+def calcos(id1, id2, Fea):
+ f1 = Fea[id1]
+ f2 = Fea[id2]
+ return np.dot(f1.transpose(), f2)
+
+
+def get_wordidx(w, Dict):
+ if w not in Dict:
+ print 'ERROR: %s not in the dictionary' % w
+ return -1
+ return Dict[w]
+
+
+if __name__ == '__main__':
+ parser = ArgumentParser()
+ parser.add_argument('dict', help='dictionary file')
+ parser.add_argument('fea', help='feature file')
+ args = parser.parse_args()
+
+ with open(args.dict) as fdict:
+ word_dict = load_dict(fdict)
+
+ with open(args.fea) as ffea:
+ word_fea = load_emb(ffea)
+
+ while True:
+ w1, w2 = raw_input("please input two words: ").split()
+ w1_id = get_wordidx(w1, word_dict)
+ w2_id = get_wordidx(w2, word_dict)
+ if w1_id == -1 or w2_id == -1:
+ continue
+ print 'similarity: %s' % (calcos(w1_id, w2_id, word_fea))
diff --git a/word2vec/data/getdata.sh b/word2vec/data/getdata.sh
new file mode 100755
index 0000000000000000000000000000000000000000..7b9e938640add251df4b8f1c61277b1c1eed61c6
--- /dev/null
+++ b/word2vec/data/getdata.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+set -e
+
+wget http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz
+tar -zxf simple-examples.tgz
+echo `pwd`/simple-examples/data/ptb.train.txt > train.list
+echo `pwd`/simple-examples/data/ptb.valid.txt > test.list
diff --git a/word2vec/dataprovider.py b/word2vec/dataprovider.py
new file mode 100644
index 0000000000000000000000000000000000000000..2f48d4f0fb17b84696f85f1df4cc558082ea9eed
--- /dev/null
+++ b/word2vec/dataprovider.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle.trainer.PyDataProvider2 import *
+import collections
+import logging
+import pdb
+
+logging.basicConfig(
+ format='[%(levelname)s %(asctime)s %(filename)s:%(lineno)s] %(message)s', )
+logger = logging.getLogger('paddle')
+logger.setLevel(logging.INFO)
+
+N = 5 # Ngram
+cutoff = 50 # select words with frequency > cutoff to dictionary
+
+
+def build_dict(ftrain, fdict):
+ sentences = []
+ with open(ftrain) as fin:
+ for line in fin:
+ line = [''] + line.strip().split() + ['']
+ sentences += line
+ wordfreq = collections.Counter(sentences)
+ wordfreq = filter(lambda x: x[1] > cutoff, wordfreq.items())
+ dictionary = sorted(wordfreq, key=lambda x: (-x[1], x[0]))
+ words, _ = list(zip(*dictionary))
+ for word in words:
+ print >> fdict, word
+ word_idx = dict(zip(words, xrange(len(words))))
+ logger.info("Dictionary size=%s" % len(words))
+ return word_idx
+
+
+def initializer(settings, srcText, dictfile, **xargs):
+ with open(dictfile, 'w') as fdict:
+ settings.dicts = build_dict(srcText, fdict)
+ input_types = []
+ for i in xrange(N):
+ input_types.append(integer_value(len(settings.dicts)))
+ settings.input_types = input_types
+
+
+@provider(init_hook=initializer)
+def process(settings, filename):
+ UNKID = settings.dicts['']
+ with open(filename) as fin:
+ for line in fin:
+ line = [''] * (N - 1) + line.strip().split() + ['']
+ line = [settings.dicts.get(w, UNKID) for w in line]
+ for i in range(N, len(line) + 1):
+ yield line[i - N:i]
diff --git a/word2vec/format_convert.py b/word2vec/format_convert.py
new file mode 100755
index 0000000000000000000000000000000000000000..f12ad81c0aa0d532d6f337d41479228f5b04ebc9
--- /dev/null
+++ b/word2vec/format_convert.py
@@ -0,0 +1,158 @@
+# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Example:
+ python format_convert.py --b2t -i INPUT -o OUTPUT -d DIM
+ python format_convert.py --t2b -i INPUT -o OUTPUT
+
+Options:
+ -h, --help show this help message and exit
+ --b2t convert parameter file of embedding model from binary to text
+ --t2b convert parameter file of embedding model from text to binary
+ -i INPUT input parameter file name
+ -o OUTPUT output parameter file name
+ -d DIM dimension of parameter
+"""
+from optparse import OptionParser
+import struct
+
+
+def binary2text(input, output, paraDim):
+ """
+ Convert a binary parameter file of embedding model to be a text file.
+ input: the name of input binary parameter file, the format is:
+ 1) the first 16 bytes is filehead:
+ version(4 bytes): version of paddle, default = 0
+ floatSize(4 bytes): sizeof(float) = 4
+ paraCount(8 bytes): total number of parameter
+ 2) the next (paraCount * 4) bytes is parameters, each has 4 bytes
+ output: the name of output text parameter file, for example:
+ 0,4,32156096
+ -0.7845433,1.1937413,-0.1704215,...
+ 0.0000909,0.0009465,-0.0008813,...
+ ...
+ the format is:
+ 1) the first line is filehead:
+ version=0, floatSize=4, paraCount=32156096
+ 2) other lines print the paramters
+ a) each line prints paraDim paramters splitted by ','
+ b) there is paraCount/paraDim lines (embedding words)
+ paraDim: dimension of parameters
+ """
+ fi = open(input, "rb")
+ fo = open(output, "w")
+ """
+ """
+ version, floatSize, paraCount = struct.unpack("iil", fi.read(16))
+ newHead = ','.join([str(version), str(floatSize), str(paraCount)])
+ print >> fo, newHead
+
+ bytes = 4 * int(paraDim)
+ format = "%df" % int(paraDim)
+ context = fi.read(bytes)
+ line = 0
+
+ while context:
+ numbers = struct.unpack(format, context)
+ lst = []
+ for i in numbers:
+ lst.append('%8.7f' % i)
+ print >> fo, ','.join(lst)
+ context = fi.read(bytes)
+ line += 1
+ fi.close()
+ fo.close()
+ print "binary2text finish, total", line, "lines"
+
+
+def get_para_count(input):
+ """
+ Compute the total number of embedding parameters in input text file.
+ input: the name of input text file
+ """
+ numRows = 1
+ paraDim = 0
+ with open(input) as f:
+ line = f.readline()
+ paraDim = len(line.split(","))
+ for line in f:
+ numRows += 1
+ return numRows * paraDim
+
+
+def text2binary(input, output, paddle_head=True):
+ """
+ Convert a text parameter file of embedding model to be a binary file.
+ input: the name of input text parameter file, for example:
+ -0.7845433,1.1937413,-0.1704215,...
+ 0.0000909,0.0009465,-0.0008813,...
+ ...
+ the format is:
+ 1) it doesn't have filehead
+ 2) each line stores the same dimension of parameters,
+ the separator is commas ','
+ output: the name of output binary parameter file, the format is:
+ 1) the first 16 bytes is filehead:
+ version(4 bytes), floatSize(4 bytes), paraCount(8 bytes)
+ 2) the next (paraCount * 4) bytes is parameters, each has 4 bytes
+ """
+ fi = open(input, "r")
+ fo = open(output, "wb")
+
+ newHead = struct.pack("iil", 0, 4, get_para_count(input))
+ fo.write(newHead)
+
+ count = 0
+ for line in fi:
+ line = line.strip().split(",")
+ for i in range(0, len(line)):
+ binary_data = struct.pack("f", float(line[i]))
+ fo.write(binary_data)
+ count += 1
+ fi.close()
+ fo.close()
+ print "text2binary finish, total", count, "lines"
+
+
+def main():
+ """
+ Main entry for running format_convert.py
+ """
+ usage = "usage: \n" \
+ "python %prog --b2t -i INPUT -o OUTPUT -d DIM \n" \
+ "python %prog --t2b -i INPUT -o OUTPUT"
+ parser = OptionParser(usage)
+ parser.add_option(
+ "--b2t",
+ action="store_true",
+ help="convert parameter file of embedding model from binary to text")
+ parser.add_option(
+ "--t2b",
+ action="store_true",
+ help="convert parameter file of embedding model from text to binary")
+ parser.add_option(
+ "-i", action="store", dest="input", help="input parameter file name")
+ parser.add_option(
+ "-o", action="store", dest="output", help="output parameter file name")
+ parser.add_option(
+ "-d", action="store", dest="dim", help="dimension of parameter")
+ (options, args) = parser.parse_args()
+ if options.b2t:
+ binary2text(options.input, options.output, options.dim)
+ if options.t2b:
+ text2binary(options.input, options.output)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/word2vec/image/2d_similarity.png b/word2vec/image/2d_similarity.png
new file mode 100644
index 0000000000000000000000000000000000000000..e67356dcf2156a9220780eb4c4ef98f4185a8294
Binary files /dev/null and b/word2vec/image/2d_similarity.png differ
diff --git a/word2vec/image/cbow.png b/word2vec/image/cbow.png
new file mode 100644
index 0000000000000000000000000000000000000000..413ce7ccb241acb1bfe1c3d3e91a93c047440d23
Binary files /dev/null and b/word2vec/image/cbow.png differ
diff --git a/word2vec/image/ngram.png b/word2vec/image/ngram.png
new file mode 100644
index 0000000000000000000000000000000000000000..d137d1d8f8c3940d93ed758cd8a4c0937c495871
Binary files /dev/null and b/word2vec/image/ngram.png differ
diff --git a/word2vec/image/nnlm.png b/word2vec/image/nnlm.png
new file mode 100644
index 0000000000000000000000000000000000000000..65dcc344ebfed7a793e25129f80829f4c2c22a19
Binary files /dev/null and b/word2vec/image/nnlm.png differ
diff --git a/word2vec/image/sentence_emb.png b/word2vec/image/sentence_emb.png
new file mode 100644
index 0000000000000000000000000000000000000000..b6f08fa8a3ca4c891a8c55532d9324c2d238c780
Binary files /dev/null and b/word2vec/image/sentence_emb.png differ
diff --git a/word2vec/image/skipgram.png b/word2vec/image/skipgram.png
new file mode 100644
index 0000000000000000000000000000000000000000..92f7c370383175538b62c28618c723c8916074bb
Binary files /dev/null and b/word2vec/image/skipgram.png differ
diff --git a/word2vec/ngram.py b/word2vec/ngram.py
new file mode 100644
index 0000000000000000000000000000000000000000..13d6291dc5b3886c42c18e80af413b36bdb724e1
--- /dev/null
+++ b/word2vec/ngram.py
@@ -0,0 +1,84 @@
+# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from paddle.trainer_config_helpers import *
+
+import math
+
+#################### Data Configure ####################
+args = {
+ 'srcText': 'data/simple-examples/data/ptb.train.txt',
+ 'dictfile': 'data/vocabulary.txt'
+}
+define_py_data_sources2(
+ train_list="data/train.list",
+ test_list="data/test.list",
+ module="dataprovider",
+ obj="process",
+ args=args)
+
+settings(
+ batch_size=100, regularization=L2Regularization(8e-4), learning_rate=3e-3)
+
+dictsize = 1953
+embsize = 32
+hiddensize = 256
+
+firstword = data_layer(name="firstw", size=dictsize)
+secondword = data_layer(name="secondw", size=dictsize)
+thirdword = data_layer(name="thirdw", size=dictsize)
+fourthword = data_layer(name="fourthw", size=dictsize)
+nextword = data_layer(name="fifthw", size=dictsize)
+
+
+# construct word embedding for each datalayer
+def wordemb(inlayer):
+ wordemb = table_projection(
+ input=inlayer,
+ size=embsize,
+ param_attr=ParamAttr(
+ name="_proj",
+ initial_std=0.001,
+ learning_rate=1,
+ l2_rate=0, ))
+ return wordemb
+
+
+Efirst = wordemb(firstword)
+Esecond = wordemb(secondword)
+Ethird = wordemb(thirdword)
+Efourth = wordemb(fourthword)
+
+# concatentate Ngram embeddings into context embedding
+contextemb = concat_layer(input=[Efirst, Esecond, Ethird, Efourth])
+hidden1 = fc_layer(
+ input=contextemb,
+ size=hiddensize,
+ act=SigmoidActivation(),
+ layer_attr=ExtraAttr(drop_rate=0.5),
+ bias_attr=ParamAttr(learning_rate=2),
+ param_attr=ParamAttr(
+ initial_std=1. / math.sqrt(embsize * 8), learning_rate=1))
+
+# use context embedding to predict nextword
+predictword = fc_layer(
+ input=hidden1,
+ size=dictsize,
+ bias_attr=ParamAttr(learning_rate=2),
+ act=SoftmaxActivation())
+
+cost = classification_cost(input=predictword, label=nextword)
+
+# network input and output
+outputs(cost)
diff --git a/word2vec/train.sh b/word2vec/train.sh
new file mode 100755
index 0000000000000000000000000000000000000000..1e7a7753aeed45e34165539ab34c2792ec8e8196
--- /dev/null
+++ b/word2vec/train.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+set -e
+
+paddle train \
+ --config ngram.py \
+ --use_gpu=1 \
+ --dot_period=100 \
+ --log_period=3000 \
+ --test_period=0 \
+ --save_dir=model \
+ --num_passes=30
