Merge branch 'develop' of https://github.com/PaddlePaddle/models into ds2_pcloud

.travis.yml

-group: deprecated-2017Q2
 language: cpp
 cache: ccache
 sudo: required

README.md

@@ -47,29 +47,37 @@ PaddlePaddle提供了丰富的运算单元，帮助大家以模块化的方式
 
 - 5.1 [基于 Pairwise 和 Listwise 的排序学习](https://github.com/PaddlePaddle/models/tree/develop/ltr)
 
-## 6. 序列标注
+## 6. 深度结构化语义模型
+ 深度结构化语义模型使用DNN模型在一个连续的语义空间中学习文本低纬的向量表示，最终建模两个句子间的语义相似度。
+
+本例中我们演示如何使用 PaddlePaddle实现一个通用的深度结构化语义模型来建模两个字符串间的语义相似度。
+模型支持CNN(卷积网络)、FC(全连接网络)、RNN(递归神经网络)等不同的网络结构，以及分类、回归、排序等不同损失函数，采用了比较通用的数据格式，用户替换数据便可以在真实场景中使用。
+
+- 6.1 [深度结构化语义模型](https://github.com/PaddlePaddle/models/tree/develop/dssm)
+
+## 7. 序列标注
 
 给定输入序列，序列标注模型为序列中每一个元素贴上一个类别标签，是自然语言处理领域最基础的任务之一。随着深度学习的不断探索和发展，利用循环神经网络学习输入序列的特征表示，条件随机场（Conditional Random Field, CRF）在特征基础上完成序列标注任务，逐渐成为解决序列标注问题的标配解决方案。
 
 在序列标注的例子中，我们以命名实体识别（Named Entity Recognition，NER）任务为例，介绍如何训练一个端到端的序列标注模型。
 
-- 6.1 [命名实体识别](https://github.com/PaddlePaddle/models/tree/develop/sequence_tagging_for_ner)
+- 7.1 [命名实体识别](https://github.com/PaddlePaddle/models/tree/develop/sequence_tagging_for_ner)
 
-## 7. 序列到序列学习
+## 8. 序列到序列学习
 
 序列到序列学习实现两个甚至是多个不定长模型之间的映射，有着广泛的应用，包括：机器翻译、智能对话与问答、广告创意语料生成、自动编码（如金融画像编码）、判断多个文本串之间的语义相关性等。
 
 在序列到序列学习的例子中，我们以机器翻译任务为例，提供了多种改进模型，供大家学习和使用。包括：不带注意力机制的序列到序列映射模型，这一模型是所有序列到序列学习模型的基础；使用 scheduled sampling 改善 RNN 模型在生成任务中的错误累积问题；带外部记忆机制的神经机器翻译，通过增强神经网络的记忆能力，来完成复杂的序列到序列学习任务。
 
-- 7.1 [无注意力机制的编码器解码器模型](https://github.com/PaddlePaddle/models/tree/develop/nmt_without_attention)
+- 8.1 [无注意力机制的编码器解码器模型](https://github.com/PaddlePaddle/models/tree/develop/nmt_without_attention)
 
-## 8. 图像分类
+## 9. 图像分类
 图像相比文字能够提供更加生动、容易理解及更具艺术感的信息，是人们转递与交换信息的重要来源。在图像分类的例子中，我们向大家介绍如何在PaddlePaddle中训练AlexNet、VGG、GoogLeNet和ResNet模型。同时还提供了一个模型转换工具，能够将Caffe训练好的模型文件，转换为PaddlePaddle的模型文件。
 
-- 8.1 [将Caffe模型文件转换为PaddlePaddle模型文件](https://github.com/PaddlePaddle/models/tree/develop/image_classification/caffe2paddle)
-- 8.2 [AlexNet](https://github.com/PaddlePaddle/models/tree/develop/image_classification)
-- 8.3 [VGG](https://github.com/PaddlePaddle/models/tree/develop/image_classification)
-- 8.4 [Residual Network](https://github.com/PaddlePaddle/models/tree/develop/image_classification)
+- 9.1 [将Caffe模型文件转换为PaddlePaddle模型文件](https://github.com/PaddlePaddle/models/tree/develop/image_classification/caffe2paddle)
+- 9.2 [AlexNet](https://github.com/PaddlePaddle/models/tree/develop/image_classification)
+- 9.3 [VGG](https://github.com/PaddlePaddle/models/tree/develop/image_classification)
+- 9.4 [Residual Network](https://github.com/PaddlePaddle/models/tree/develop/image_classification)
 
 
 ## Copyright and License

ctr/README.md

 # 点击率预估
 
+以下是本例目录包含的文件以及对应说明:
+
+```
+├── README.md               # 本教程markdown 文档
+├── dataset.md              # 数据集处理教程
+├── images                  # 本教程图片目录
+│   ├── lr_vs_dnn.jpg
+│   └── wide_deep.png
+├── infer.py                # 预测脚本
+├── network_conf.py         # 模型网络配置
+├── reader.py               # data reader
+├── train.py                # 训练脚本
+└── utils.py                # helper functions
+└── avazu_data_processer.py # 示例数据预处理脚本
+```
+
 ## 背景介绍
 
-CTR(Click-Through Rate，点击率预估)\[[1](https://en.wikipedia.org/wiki/Click-through_rate)\] 是用来表示用户点击一个特定链接的概率，
-通常被用来衡量一个在线广告系统的有效性。
+CTR(Click-Through Rate，点击率预估)\[[1](https://en.wikipedia.org/wiki/Click-through_rate)\]
+是对用户点击一个特定链接的概率做出预测，是广告投放过程中的一个重要环节。精准的点击率预估对在线广告系统收益最大化具有重要意义。
 
-当有多个广告位时，CTR 预估一般会作为排序的基准。
-比如在搜索引擎的广告系统里，当用户输入一个带商业价值的搜索词（query）时，系统大体上会执行下列步骤来展示广告：
+当有多个广告位时，CTR 预估一般会作为排序的基准，比如在搜索引擎的广告系统里，当用户输入一个带商业价值的搜索词（query）时，系统大体上会执行下列步骤来展示广告：
 
-1.  召回满足 query 的广告集合
+1.  获取与用户搜索词相关的广告集合
 2.  业务规则和相关性过滤
 3.  根据拍卖机制和 CTR 排序
 4.  展出广告
@@ -36,13 +51,11 @@ Figure 1. LR 和 DNN 模型结构对比
 </p>
 
 LR 的蓝色箭头部分可以直接类比到 DNN 中对应的结构，可以看到 LR 和 DNN 有一些共通之处（比如权重累加），
-但前者的模型复杂度在相同输入维度下比后者可能低很多（从某方面讲，模型越复杂，越有潜力学习到更复杂的信息）。
-
+但前者的模型复杂度在相同输入维度下比后者可能低很多（从某方面讲，模型越复杂，越有潜力学习到更复杂的信息）；
 如果 LR 要达到匹敌 DNN 的学习能力，必须增加输入的维度，也就是增加特征的数量，
 这也就是为何 LR 和大规模的特征工程必须绑定在一起的原因。
 
-LR 对于 DNN 模型的优势是对大规模稀疏特征的容纳能力，包括内存和计算量等方面，工业界都有非常成熟的优化方法。
-
+LR 对于 DNN 模型的优势是对大规模稀疏特征的容纳能力，包括内存和计算量等方面，工业界都有非常成熟的优化方法；
 而 DNN 模型具有自己学习新特征的能力，一定程度上能够提升特征使用的效率，
 这使得 DNN 模型在同样规模特征的情况下，更有可能达到更好的学习效果。
 
@@ -59,10 +72,62 @@ LR 对于 DNN 模型的优势是对大规模稀疏特征的容纳能力，包括
 
 我们直接使用第一种方法做分类任务。
 
-我们使用 Kaggle 上 `Click-through rate prediction` 任务的数据集\[[2](https://www.kaggle.com/c/avazu-ctr-prediction/data)\] 来演示模型。
+我们使用 Kaggle 上 `Click-through rate prediction` 任务的数据集\[[2](https://www.kaggle.com/c/avazu-ctr-prediction/data)\] 来演示本例中的模型。
+
+具体的特征处理方法参看 [data process](./dataset.md)。
+
+本教程中演示模型的输入格式如下：
+
+```
+# <dnn input ids> \t <lr input sparse values> \t click
+1 23 190 \t 230:0.12 3421:0.9 23451:0.12 \t 0
+23 231 \t 1230:0.12 13421:0.9 \t 1
+```
+
+详细的格式描述如下：
+
+- `dnn input ids` 采用 one-hot 表示，只需要填写值为1的ID（注意这里不是变长输入）
+- `lr input sparse values` 使用了 `ID:VALUE` 的表示，值部分最好规约到值域 `[-1, 1]`。
+
+此外，模型训练时需要传入一个文件描述 dnn 和 lr两个子模型的输入维度，文件的格式如下：
 
-具体的特征处理方法参看 [data process](./dataset.md)
+```
+dnn_input_dim: <int>
+lr_input_dim: <int>
+```
+
+其中， `<int>` 表示一个整型数值。
+
+本目录下的 `avazu_data_processor.py` 可以对下载的演示数据集\[[2](#参考文档)\] 进行处理，具体使用方法参考如下说明：
+
+```
+usage: avazu_data_processer.py [-h] --data_path DATA_PATH --output_dir
+                               OUTPUT_DIR
+                               [--num_lines_to_detect NUM_LINES_TO_DETECT]
+                               [--test_set_size TEST_SET_SIZE]
+                               [--train_size TRAIN_SIZE]
+
+PaddlePaddle CTR example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --data_path DATA_PATH
+                        path of the Avazu dataset
+  --output_dir OUTPUT_DIR
+                        directory to output
+  --num_lines_to_detect NUM_LINES_TO_DETECT
+                        number of records to detect dataset's meta info
+  --test_set_size TEST_SET_SIZE
+                        size of the validation dataset(default: 10000)
+  --train_size TRAIN_SIZE
+                        size of the trainset (default: 100000)
+```
 
+- `data_path` 是待处理的数据路径
+- `output_dir` 生成数据的输出路径
+- `num_lines_to_detect` 预先扫描数据生成ID的个数，这里是扫描的文件行数
+- `test_set_size` 生成测试集的行数
+- `train_size` 生成训练姐的行数
 
 ## Wide & Deep Learning Model
 
@@ -201,18 +266,20 @@ trainer.train(
 ## 运行训练和测试
 训练模型需要如下步骤：
 
-1. 下载训练数据，可以使用 Kaggle 上 CTR 比赛的数据\[[2](#参考文献)\]
+1. 准备训练数据
     1. 从 [Kaggle CTR](https://www.kaggle.com/c/avazu-ctr-prediction/data) 下载 train.gz
     2. 解压 train.gz 得到 train.txt
-2. 执行 `python train.py --train_data_path train.txt` ，开始训练
+    3. `mkdir -p output; python avazu_data_processer.py --data_path train.txt --output_dir output --num_lines_to_detect 1000 --test_set_size 100` 生成演示数据
+2. 执行 `python train.py --train_data_path ./output/train.txt --test_data_path ./output/test.txt --data_meta_file ./output/data.meta.txt --model_type=0` 开始训练
 
 上面第2个步骤可以为 `train.py` 填充命令行参数来定制模型的训练过程，具体的命令行参数及用法如下
 
 ```
 usage: train.py [-h] --train_data_path TRAIN_DATA_PATH
-                [--batch_size BATCH_SIZE] [--test_set_size TEST_SET_SIZE]
+                [--test_data_path TEST_DATA_PATH] [--batch_size BATCH_SIZE]
                 [--num_passes NUM_PASSES]
-                [--num_lines_to_detact NUM_LINES_TO_DETACT]
+                [--model_output_prefix MODEL_OUTPUT_PREFIX] --data_meta_file
+                DATA_META_FILE --model_type MODEL_TYPE
 
 PaddlePaddle CTR example
 
@@ -220,16 +287,78 @@ optional arguments:
   -h, --help            show this help message and exit
   --train_data_path TRAIN_DATA_PATH
                         path of training dataset
+  --test_data_path TEST_DATA_PATH
+                        path of testing dataset
   --batch_size BATCH_SIZE
                         size of mini-batch (default:10000)
-  --test_set_size TEST_SET_SIZE
-                        size of the validation dataset(default: 10000)
   --num_passes NUM_PASSES
                         number of passes to train
-  --num_lines_to_detact NUM_LINES_TO_DETACT
-                        number of records to detect dataset's meta info
+  --model_output_prefix MODEL_OUTPUT_PREFIX
+                        prefix of path for model to store (default:
+                        ./ctr_models)
+  --data_meta_file DATA_META_FILE
+                        path of data meta info file
+  --model_type MODEL_TYPE
+                        model type, classification: 0, regression 1 (default
+                        classification)
+```
+
+- `train_data_path` ： 训练集的路径
+- `test_data_path` : 测试集的路径
+- `num_passes`: 模型训练多少轮
+- `data_meta_file`: 参考[数据和任务抽象](### 数据和任务抽象)的描述。
+- `model_type`: 模型分类或回归
+
+
+## 用训好的模型做预测
+训好的模型可以用来预测新的数据， 预测数据的格式为
+
+```
+# <dnn input ids> \t <lr input sparse values>
+1 23 190 \t 230:0.12 3421:0.9 23451:0.12
+23 231 \t 1230:0.12 13421:0.9
 ```
 
+这里与训练数据的格式唯一不同的地方，就是没有标签，也就是训练数据中第3列 `click` 对应的数值。
+
+`infer.py` 的使用方法如下
+
+```
+usage: infer.py [-h] --model_gz_path MODEL_GZ_PATH --data_path DATA_PATH
+                --prediction_output_path PREDICTION_OUTPUT_PATH
+                [--data_meta_path DATA_META_PATH] --model_type MODEL_TYPE
+
+PaddlePaddle CTR example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --model_gz_path MODEL_GZ_PATH
+                        path of model parameters gz file
+  --data_path DATA_PATH
+                        path of the dataset to infer
+  --prediction_output_path PREDICTION_OUTPUT_PATH
+                        path to output the prediction
+  --data_meta_path DATA_META_PATH
+                        path of trainset's meta info, default is ./data.meta
+  --model_type MODEL_TYPE
+                        model type, classification: 0, regression 1 (default
+                        classification)
+```
+
+- `model_gz_path_model`：用 `gz` 压缩过的模型路径
+- `data_path` ： 需要预测的数据路径
+- `prediction_output_paht`：预测输出的路径
+- `data_meta_file` ：参考[数据和任务抽象](### 数据和任务抽象)的描述。
+- `model_type` ：分类或回归
+
+示例数据可以用如下命令预测
+
+```
+python infer.py --model_gz_path <model_path> --data_path output/infer.txt --prediction_output_path predictions.txt --data_meta_path data.meta.txt
+```
+
+最终的预测结果位于 `predictions.txt`。
+
 ## 参考文献
 1. <https://en.wikipedia.org/wiki/Click-through_rate>
 2. <https://www.kaggle.com/c/avazu-ctr-prediction/data>

ctr/data_provider.py → ctr/avazu_data_processer.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-import os
 import sys
 import csv
+import cPickle
+import argparse
 import numpy as np
+
+from utils import logger, TaskMode
+
+parser = argparse.ArgumentParser(description="PaddlePaddle CTR example")
+parser.add_argument(
+    '--data_path', type=str, required=True, help="path of the Avazu dataset")
+parser.add_argument(
+    '--output_dir', type=str, required=True, help="directory to output")
+parser.add_argument(
+    '--num_lines_to_detect',
+    type=int,
+    default=500000,
+    help="number of records to detect dataset's meta info")
+parser.add_argument(
+    '--test_set_size',
+    type=int,
+    default=10000,
+    help="size of the validation dataset(default: 10000)")
+parser.add_argument(
+    '--train_size',
+    type=int,
+    default=100000,
+    help="size of the trainset (default: 100000)")
+args = parser.parse_args()
 '''
 The fields of the dataset are:
 
@@ -22,7 +50,7 @@ The fields of the dataset are:
     15. device_conn_type
     16. C14-C21 -- anonymized categorical variables
 
-We will treat following fields as categorical features:
+We will treat the following fields as categorical features:
 
     - C1
     - banner_pos
@@ -40,6 +68,14 @@ and some other features as id features:
 
 The `hour` field will be treated as a continuous feature and will be transformed
 to one-hot representation which has 24 bits.
+
+This script will output 3 files:
+
+1. train.txt
+2. test.txt
+3. infer.txt
+
+all the files are for demo.
 '''
 
 feature_dims = {}
@@ -161,6 +197,7 @@ def detect_dataset(path, topn, id_fea_space=10000):
     NOTE the records should be randomly shuffled first.
     '''
     # create categorical statis objects.
+    logger.warning('detecting dataset')
 
     with open(path, 'rb') as csvfile:
         reader = csv.DictReader(csvfile)
@@ -174,9 +211,6 @@ def detect_dataset(path, topn, id_fea_space=10000):
     for key, item in fields.items():
         feature_dims[key] = item.size()
 
-    #for key in id_features:
-    #feature_dims[key] = id_fea_space
-
     feature_dims['hour'] = 24
     feature_dims['click'] = 1
 
@@ -184,10 +218,17 @@ def detect_dataset(path, topn, id_fea_space=10000):
         feature_dims[key] for key in categorial_features + ['hour']) + 1
     feature_dims['lr_input'] = np.sum(feature_dims[key]
                                       for key in id_features) + 1
-
     return feature_dims
 
 
+def load_data_meta(meta_path):
+    '''
+    Load dataset's meta infomation.
+    '''
+    feature_dims, fields = cPickle.load(open(meta_path, 'rb'))
+    return feature_dims, fields
+
+
 def concat_sparse_vectors(inputs, dims):
     '''
     Concaterate more than one sparse vectors into one.
@@ -211,67 +252,162 @@ class AvazuDataset(object):
     '''
     Load AVAZU dataset as train set.
     '''
-    TRAIN_MODE = 0
-    TEST_MODE = 1
 
-    def __init__(self, train_path, n_records_as_test=-1):
+    def __init__(self,
+                 train_path,
+                 n_records_as_test=-1,
+                 fields=None,
+                 feature_dims=None):
         self.train_path = train_path
         self.n_records_as_test = n_records_as_test
-        # task model: 0 train, 1 test
-        self.mode = 0
+        self.fields = fields
+        # default is train mode.
+        self.mode = TaskMode.create_train()
 
-    def train(self):
-        self.mode = self.TRAIN_MODE
-        return self._parse(self.train_path, skip_n_lines=self.n_records_as_test)
+        self.categorial_dims = [
+            feature_dims[key] for key in categorial_features + ['hour']
+        ]
+        self.id_dims = [feature_dims[key] for key in id_features]
 
-    def test(self):
-        self.mode = self.TEST_MODE
-        return self._parse(self.train_path, top_n_lines=self.n_records_as_test)
-
-    def _parse(self, path, skip_n_lines=-1, top_n_lines=-1):
-        with open(path, 'rb') as csvfile:
-            reader = csv.DictReader(csvfile)
-
-            categorial_dims = [
-                feature_dims[key] for key in categorial_features + ['hour']
-            ]
-            id_dims = [feature_dims[key] for key in id_features]
+    def train(self):
+        '''
+        Load trainset.
+        '''
+        logger.info("load trainset from %s" % self.train_path)
+        self.mode = TaskMode.create_train()
+        with open(self.train_path) as f:
+            reader = csv.DictReader(f)
 
             for row_id, row in enumerate(reader):
-                if skip_n_lines > 0 and row_id < skip_n_lines:
+                # skip top n lines
+                if self.n_records_as_test > 0 and row_id < self.n_records_as_test:
                     continue
-                if top_n_lines > 0 and row_id > top_n_lines:
-                    break
-
-                record = []
-                for key in categorial_features:
-                    record.append(fields[key].gen(row[key]))
-                record.append([int(row['hour'][-2:])])
-                dense_input = concat_sparse_vectors(record, categorial_dims)
 
-                record = []
-                for key in id_features:
-                    if 'cross' not in key:
-                        record.append(fields[key].gen(row[key]))
-                    else:
-                        fea0 = fields[key].cross_fea0
-                        fea1 = fields[key].cross_fea1
-                        record.append(
-                            fields[key].gen_cross_fea(row[fea0], row[fea1]))
+                rcd = self._parse_record(row)
+                if rcd:
+                    yield rcd
 
-                sparse_input = concat_sparse_vectors(record, id_dims)
+    def test(self):
+        '''
+        Load testset.
+        '''
+        logger.info("load testset from %s" % self.train_path)
+        self.mode = TaskMode.create_test()
+        with open(self.train_path) as f:
+            reader = csv.DictReader(f)
 
-                record = [dense_input, sparse_input]
+            for row_id, row in enumerate(reader):
+                # skip top n lines
+                if self.n_records_as_test > 0 and row_id > self.n_records_as_test:
+                    break
 
-                record.append(list((int(row['click']), )))
-                yield record
+                rcd = self._parse_record(row)
+                if rcd:
+                    yield rcd
 
+    def infer(self):
+        '''
+        Load inferset.
+        '''
+        logger.info("load inferset from %s" % self.train_path)
+        self.mode = TaskMode.create_infer()
+        with open(self.train_path) as f:
+            reader = csv.DictReader(f)
 
-if __name__ == '__main__':
-    path = 'train.txt'
-    print detect_dataset(path, 400000)
+            for row_id, row in enumerate(reader):
+                rcd = self._parse_record(row)
+                if rcd:
+                    yield rcd
 
-    filereader = AvazuDataset(path)
-    for no, rcd in enumerate(filereader.train()):
-        print no, rcd
-        if no > 1000: break
+    def _parse_record(self, row):
+        '''
+        Parse a CSV row and get a record.
+        '''
+        record = []
+        for key in categorial_features:
+            record.append(self.fields[key].gen(row[key]))
+        record.append([int(row['hour'][-2:])])
+        dense_input = concat_sparse_vectors(record, self.categorial_dims)
+
+        record = []
+        for key in id_features:
+            if 'cross' not in key:
+                record.append(self.fields[key].gen(row[key]))
+            else:
+                fea0 = self.fields[key].cross_fea0
+                fea1 = self.fields[key].cross_fea1
+                record.append(
+                    self.fields[key].gen_cross_fea(row[fea0], row[fea1]))
+
+        sparse_input = concat_sparse_vectors(record, self.id_dims)
+
+        record = [dense_input, sparse_input]
+
+        if not self.mode.is_infer():
+            record.append(list((int(row['click']), )))
+        return record
+
+
+def ids2dense(vec, dim):
+    return vec
+
+
+def ids2sparse(vec):
+    return ["%d:1" % x for x in vec]
+
+
+detect_dataset(args.data_path, args.num_lines_to_detect)
+dataset = AvazuDataset(
+    args.data_path,
+    args.test_set_size,
+    fields=fields,
+    feature_dims=feature_dims)
+
+output_trainset_path = os.path.join(args.output_dir, 'train.txt')
+output_testset_path = os.path.join(args.output_dir, 'test.txt')
+output_infer_path = os.path.join(args.output_dir, 'infer.txt')
+output_meta_path = os.path.join(args.output_dir, 'data.meta.txt')
+
+with open(output_trainset_path, 'w') as f:
+    for id, record in enumerate(dataset.train()):
+        if id and id % 10000 == 0:
+            logger.info("load %d records" % id)
+        if id > args.train_size:
+            break
+        dnn_input, lr_input, click = record
+        dnn_input = ids2dense(dnn_input, feature_dims['dnn_input'])
+        lr_input = ids2sparse(lr_input)
+        line = "%s\t%s\t%d\n" % (' '.join(map(str, dnn_input)),
+                                 ' '.join(map(str, lr_input)), click[0])
+        f.write(line)
+    logger.info('write to %s' % output_trainset_path)
+
+with open(output_testset_path, 'w') as f:
+    for id, record in enumerate(dataset.test()):
+        dnn_input, lr_input, click = record
+        dnn_input = ids2dense(dnn_input, feature_dims['dnn_input'])
+        lr_input = ids2sparse(lr_input)
+        line = "%s\t%s\t%d\n" % (' '.join(map(str, dnn_input)),
+                                 ' '.join(map(str, lr_input)), click[0])
+        f.write(line)
+    logger.info('write to %s' % output_testset_path)
+
+with open(output_infer_path, 'w') as f:
+    for id, record in enumerate(dataset.infer()):
+        dnn_input, lr_input = record
+        dnn_input = ids2dense(dnn_input, feature_dims['dnn_input'])
+        lr_input = ids2sparse(lr_input)
+        line = "%s\t%s\n" % (' '.join(map(str, dnn_input)),
+                             ' '.join(map(str, lr_input)), )
+        f.write(line)
+        if id > args.test_set_size:
+            break
+    logger.info('write to %s' % output_infer_path)
+
+with open(output_meta_path, 'w') as f:
+    lines = [
+        "dnn_input_dim: %d" % feature_dims['dnn_input'],
+        "lr_input_dim: %d" % feature_dims['lr_input']
+    ]
+    f.write('\n'.join(lines))
+    logger.info('write data meta into %s' % output_meta_path)

ctr/dataset.md

 # 数据及处理
 ## 数据集介绍
 
+本教程演示使用Kaggle上CTR任务的数据集\[[3](#参考文献)\]的预处理方法，最终产生本模型需要的格式，详细的数据格式参考[README.md](./README.md)。
+
+Wide && Deep Model\[[2](#参考文献)\]的优势是融合稠密特征和大规模稀疏特征，
+因此特征处理方面也针对稠密和稀疏两种特征作处理，
+其中Deep部分的稠密值全部转化为ID类特征，
+通过embedding 来转化为稠密的向量输入；Wide部分主要通过ID的叉乘提升维度。
+
 数据集使用 `csv` 格式存储，其中各个字段内容如下：
 
 -   `id` : ad identifier

ctr/index.html

@@ -42,15 +42,30 @@
 <div id="markdown" style='display:none'>
 # 点击率预估
 
+以下是本例目录包含的文件以及对应说明:
+
+```
+├── README.md               # 本教程markdown 文档
+├── dataset.md              # 数据集处理教程
+├── images                  # 本教程图片目录
+│   ├── lr_vs_dnn.jpg
+│   └── wide_deep.png
+├── infer.py                # 预测脚本
+├── network_conf.py         # 模型网络配置
+├── reader.py               # data reader
+├── train.py                # 训练脚本
+└── utils.py                # helper functions
+└── avazu_data_processer.py # 示例数据预处理脚本
+```
+
 ## 背景介绍
 
-CTR(Click-Through Rate，点击率预估)\[[1](https://en.wikipedia.org/wiki/Click-through_rate)\] 是用来表示用户点击一个特定链接的概率，
-通常被用来衡量一个在线广告系统的有效性。
+CTR(Click-Through Rate，点击率预估)\[[1](https://en.wikipedia.org/wiki/Click-through_rate)\]
+是对用户点击一个特定链接的概率做出预测，是广告投放过程中的一个重要环节。精准的点击率预估对在线广告系统收益最大化具有重要意义。
 
-当有多个广告位时，CTR 预估一般会作为排序的基准。
-比如在搜索引擎的广告系统里，当用户输入一个带商业价值的搜索词（query）时，系统大体上会执行下列步骤来展示广告：
+当有多个广告位时，CTR 预估一般会作为排序的基准，比如在搜索引擎的广告系统里，当用户输入一个带商业价值的搜索词（query）时，系统大体上会执行下列步骤来展示广告：
 
-1.  召回满足 query 的广告集合
+1.  获取与用户搜索词相关的广告集合
 2.  业务规则和相关性过滤
 3.  根据拍卖机制和 CTR 排序
 4.  展出广告
@@ -78,13 +93,11 @@ Figure 1. LR 和 DNN 模型结构对比
 </p>
 
 LR 的蓝色箭头部分可以直接类比到 DNN 中对应的结构，可以看到 LR 和 DNN 有一些共通之处（比如权重累加），
-但前者的模型复杂度在相同输入维度下比后者可能低很多（从某方面讲，模型越复杂，越有潜力学习到更复杂的信息）。
-
+但前者的模型复杂度在相同输入维度下比后者可能低很多（从某方面讲，模型越复杂，越有潜力学习到更复杂的信息）；
 如果 LR 要达到匹敌 DNN 的学习能力，必须增加输入的维度，也就是增加特征的数量，
 这也就是为何 LR 和大规模的特征工程必须绑定在一起的原因。
 
-LR 对于 DNN 模型的优势是对大规模稀疏特征的容纳能力，包括内存和计算量等方面，工业界都有非常成熟的优化方法。
-
+LR 对于 DNN 模型的优势是对大规模稀疏特征的容纳能力，包括内存和计算量等方面，工业界都有非常成熟的优化方法；
 而 DNN 模型具有自己学习新特征的能力，一定程度上能够提升特征使用的效率，
 这使得 DNN 模型在同样规模特征的情况下，更有可能达到更好的学习效果。
 
@@ -101,10 +114,62 @@ LR 对于 DNN 模型的优势是对大规模稀疏特征的容纳能力，包括
 
 我们直接使用第一种方法做分类任务。
 
-我们使用 Kaggle 上 `Click-through rate prediction` 任务的数据集\[[2](https://www.kaggle.com/c/avazu-ctr-prediction/data)\] 来演示模型。
+我们使用 Kaggle 上 `Click-through rate prediction` 任务的数据集\[[2](https://www.kaggle.com/c/avazu-ctr-prediction/data)\] 来演示本例中的模型。
+
+具体的特征处理方法参看 [data process](./dataset.md)。
+
+本教程中演示模型的输入格式如下：
+
+```
+# <dnn input ids> \t <lr input sparse values> \t click
+1 23 190 \t 230:0.12 3421:0.9 23451:0.12 \t 0
+23 231 \t 1230:0.12 13421:0.9 \t 1
+```
+
+详细的格式描述如下：
+
+- `dnn input ids` 采用 one-hot 表示，只需要填写值为1的ID（注意这里不是变长输入）
+- `lr input sparse values` 使用了 `ID:VALUE` 的表示，值部分最好规约到值域 `[-1, 1]`。
+
+此外，模型训练时需要传入一个文件描述 dnn 和 lr两个子模型的输入维度，文件的格式如下：
 
-具体的特征处理方法参看 [data process](./dataset.md)
+```
+dnn_input_dim: <int>
+lr_input_dim: <int>
+```
+
+其中， `<int>` 表示一个整型数值。
+
+本目录下的 `avazu_data_processor.py` 可以对下载的演示数据集\[[2](#参考文档)\] 进行处理，具体使用方法参考如下说明：
+
+```
+usage: avazu_data_processer.py [-h] --data_path DATA_PATH --output_dir
+                               OUTPUT_DIR
+                               [--num_lines_to_detect NUM_LINES_TO_DETECT]
+                               [--test_set_size TEST_SET_SIZE]
+                               [--train_size TRAIN_SIZE]
+
+PaddlePaddle CTR example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --data_path DATA_PATH
+                        path of the Avazu dataset
+  --output_dir OUTPUT_DIR
+                        directory to output
+  --num_lines_to_detect NUM_LINES_TO_DETECT
+                        number of records to detect dataset's meta info
+  --test_set_size TEST_SET_SIZE
+                        size of the validation dataset(default: 10000)
+  --train_size TRAIN_SIZE
+                        size of the trainset (default: 100000)
+```
 
+- `data_path` 是待处理的数据路径
+- `output_dir` 生成数据的输出路径
+- `num_lines_to_detect` 预先扫描数据生成ID的个数，这里是扫描的文件行数
+- `test_set_size` 生成测试集的行数
+- `train_size` 生成训练姐的行数
 
 ## Wide & Deep Learning Model
 
@@ -243,18 +308,20 @@ trainer.train(
 ## 运行训练和测试
 训练模型需要如下步骤：
 
-1. 下载训练数据，可以使用 Kaggle 上 CTR 比赛的数据\[[2](#参考文献)\]
+1. 准备训练数据
     1. 从 [Kaggle CTR](https://www.kaggle.com/c/avazu-ctr-prediction/data) 下载 train.gz
     2. 解压 train.gz 得到 train.txt
-2. 执行 `python train.py --train_data_path train.txt` ，开始训练
+    3. `mkdir -p output; python avazu_data_processer.py --data_path train.txt --output_dir output --num_lines_to_detect 1000 --test_set_size 100` 生成演示数据
+2. 执行 `python train.py --train_data_path ./output/train.txt --test_data_path ./output/test.txt --data_meta_file ./output/data.meta.txt --model_type=0` 开始训练
 
 上面第2个步骤可以为 `train.py` 填充命令行参数来定制模型的训练过程，具体的命令行参数及用法如下
 
 ```
 usage: train.py [-h] --train_data_path TRAIN_DATA_PATH
-                [--batch_size BATCH_SIZE] [--test_set_size TEST_SET_SIZE]
+                [--test_data_path TEST_DATA_PATH] [--batch_size BATCH_SIZE]
                 [--num_passes NUM_PASSES]
-                [--num_lines_to_detact NUM_LINES_TO_DETACT]
+                [--model_output_prefix MODEL_OUTPUT_PREFIX] --data_meta_file
+                DATA_META_FILE --model_type MODEL_TYPE
 
 PaddlePaddle CTR example
 
@@ -262,16 +329,78 @@ optional arguments:
   -h, --help            show this help message and exit
   --train_data_path TRAIN_DATA_PATH
                         path of training dataset
+  --test_data_path TEST_DATA_PATH
+                        path of testing dataset
   --batch_size BATCH_SIZE
                         size of mini-batch (default:10000)
-  --test_set_size TEST_SET_SIZE
-                        size of the validation dataset(default: 10000)
   --num_passes NUM_PASSES
                         number of passes to train
-  --num_lines_to_detact NUM_LINES_TO_DETACT
-                        number of records to detect dataset's meta info
+  --model_output_prefix MODEL_OUTPUT_PREFIX
+                        prefix of path for model to store (default:
+                        ./ctr_models)
+  --data_meta_file DATA_META_FILE
+                        path of data meta info file
+  --model_type MODEL_TYPE
+                        model type, classification: 0, regression 1 (default
+                        classification)
+```
+
+- `train_data_path` ： 训练集的路径
+- `test_data_path` : 测试集的路径
+- `num_passes`: 模型训练多少轮
+- `data_meta_file`: 参考[数据和任务抽象](### 数据和任务抽象)的描述。
+- `model_type`: 模型分类或回归
+
+
+## 用训好的模型做预测
+训好的模型可以用来预测新的数据， 预测数据的格式为
+
+```
+# <dnn input ids> \t <lr input sparse values>
+1 23 190 \t 230:0.12 3421:0.9 23451:0.12
+23 231 \t 1230:0.12 13421:0.9
 ```
 
+这里与训练数据的格式唯一不同的地方，就是没有标签，也就是训练数据中第3列 `click` 对应的数值。
+
+`infer.py` 的使用方法如下
+
+```
+usage: infer.py [-h] --model_gz_path MODEL_GZ_PATH --data_path DATA_PATH
+                --prediction_output_path PREDICTION_OUTPUT_PATH
+                [--data_meta_path DATA_META_PATH] --model_type MODEL_TYPE
+
+PaddlePaddle CTR example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --model_gz_path MODEL_GZ_PATH
+                        path of model parameters gz file
+  --data_path DATA_PATH
+                        path of the dataset to infer
+  --prediction_output_path PREDICTION_OUTPUT_PATH
+                        path to output the prediction
+  --data_meta_path DATA_META_PATH
+                        path of trainset's meta info, default is ./data.meta
+  --model_type MODEL_TYPE
+                        model type, classification: 0, regression 1 (default
+                        classification)
+```
+
+- `model_gz_path_model`：用 `gz` 压缩过的模型路径
+- `data_path` ： 需要预测的数据路径
+- `prediction_output_paht`：预测输出的路径
+- `data_meta_file` ：参考[数据和任务抽象](### 数据和任务抽象)的描述。
+- `model_type` ：分类或回归
+
+示例数据可以用如下命令预测
+
+```
+python infer.py --model_gz_path <model_path> --data_path output/infer.txt --prediction_output_path predictions.txt --data_meta_path data.meta.txt
+```
+
+最终的预测结果位于 `predictions.txt`。
+
 ## 参考文献
 1. <https://en.wikipedia.org/wiki/Click-through_rate>
 2. <https://www.kaggle.com/c/avazu-ctr-prediction/data>

ctr/infer.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import gzip
+import argparse
+import itertools
+
+import paddle.v2 as paddle
+import network_conf
+from train import dnn_layer_dims
+import reader
+from utils import logger, ModelType
+
+parser = argparse.ArgumentParser(description="PaddlePaddle CTR example")
+parser.add_argument(
+    '--model_gz_path',
+    type=str,
+    required=True,
+    help="path of model parameters gz file")
+parser.add_argument(
+    '--data_path', type=str, required=True, help="path of the dataset to infer")
+parser.add_argument(
+    '--prediction_output_path',
+    type=str,
+    required=True,
+    help="path to output the prediction")
+parser.add_argument(
+    '--data_meta_path',
+    type=str,
+    default="./data.meta",
+    help="path of trainset's meta info, default is ./data.meta")
+parser.add_argument(
+    '--model_type',
+    type=int,
+    required=True,
+    default=ModelType.CLASSIFICATION,
+    help='model type, classification: %d, regression %d (default classification)'
+    % (ModelType.CLASSIFICATION, ModelType.REGRESSION))
+
+args = parser.parse_args()
+
+paddle.init(use_gpu=False, trainer_count=1)
+
+
+class CTRInferer(object):
+    def __init__(self, param_path):
+        logger.info("create CTR model")
+        dnn_input_dim, lr_input_dim = reader.load_data_meta(args.data_meta_path)
+        # create the mdoel
+        self.ctr_model = network_conf.CTRmodel(
+            dnn_layer_dims,
+            dnn_input_dim,
+            lr_input_dim,
+            model_type=ModelType(args.model_type),
+            is_infer=True)
+        # load parameter
+        logger.info("load model parameters from %s" % param_path)
+        self.parameters = paddle.parameters.Parameters.from_tar(
+            gzip.open(param_path, 'r'))
+        self.inferer = paddle.inference.Inference(
+            output_layer=self.ctr_model.model,
+            parameters=self.parameters, )
+
+    def infer(self, data_path):
+        logger.info("infer data...")
+        dataset = reader.Dataset()
+        infer_reader = paddle.batch(
+            dataset.infer(args.data_path), batch_size=1000)
+        logger.warning('write predictions to %s' % args.prediction_output_path)
+        output_f = open(args.prediction_output_path, 'w')
+        for id, batch in enumerate(infer_reader()):
+            res = self.inferer.infer(input=batch)
+            predictions = [x for x in itertools.chain.from_iterable(res)]
+            assert len(batch) == len(
+                predictions), "predict error, %d inputs, but %d predictions" % (
+                    len(batch), len(predictions))
+            output_f.write('\n'.join(map(str, predictions)) + '\n')
+
+
+if __name__ == '__main__':
+    ctr_inferer = CTRInferer(args.model_gz_path)
+    ctr_inferer.infer(args.data_path)

ctr/network_conf.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import paddle.v2 as paddle
+from paddle.v2 import layer
+from paddle.v2 import data_type as dtype
+from utils import logger, ModelType
+
+
+class CTRmodel(object):
+    '''
+    A CTR model which implements wide && deep learning model.
+    '''
+
+    def __init__(self,
+                 dnn_layer_dims,
+                 dnn_input_dim,
+                 lr_input_dim,
+                 model_type=ModelType.create_classification(),
+                 is_infer=False):
+        '''
+        @dnn_layer_dims: list of integer
+            dims of each layer in dnn
+        @dnn_input_dim: int
+            size of dnn's input layer
+        @lr_input_dim: int
+            size of lr's input layer
+        @is_infer: bool
+            whether to build a infer model
+        '''
+        self.dnn_layer_dims = dnn_layer_dims
+        self.dnn_input_dim = dnn_input_dim
+        self.lr_input_dim = lr_input_dim
+        self.model_type = model_type
+        self.is_infer = is_infer
+
+        self._declare_input_layers()
+
+        self.dnn = self._build_dnn_submodel_(self.dnn_layer_dims)
+        self.lr = self._build_lr_submodel_()
+
+        # model's prediction
+        # TODO(superjom) rename it to prediction
+        if self.model_type.is_classification():
+            self.model = self._build_classification_model(self.dnn, self.lr)
+        if self.model_type.is_regression():
+            self.model = self._build_regression_model(self.dnn, self.lr)
+
+    def _declare_input_layers(self):
+        self.dnn_merged_input = layer.data(
+            name='dnn_input',
+            type=paddle.data_type.sparse_binary_vector(self.dnn_input_dim))
+
+        self.lr_merged_input = layer.data(
+            name='lr_input',
+            type=paddle.data_type.sparse_vector(self.lr_input_dim))
+
+        if not self.is_infer:
+            self.click = paddle.layer.data(
+                name='click', type=dtype.dense_vector(1))
+
+    def _build_dnn_submodel_(self, dnn_layer_dims):
+        '''
+        build DNN submodel.
+        '''
+        dnn_embedding = layer.fc(
+            input=self.dnn_merged_input, size=dnn_layer_dims[0])
+        _input_layer = dnn_embedding
+        for i, dim in enumerate(dnn_layer_dims[1:]):
+            fc = layer.fc(
+                input=_input_layer,
+                size=dim,
+                act=paddle.activation.Relu(),
+                name='dnn-fc-%d' % i)
+            _input_layer = fc
+        return _input_layer
+
+    def _build_lr_submodel_(self):
+        '''
+        config LR submodel
+        '''
+        fc = layer.fc(
+            input=self.lr_merged_input, size=1, act=paddle.activation.Relu())
+        return fc
+
+    def _build_classification_model(self, dnn, lr):
+        merge_layer = layer.concat(input=[dnn, lr])
+        self.output = layer.fc(
+            input=merge_layer,
+            size=1,
+            # use sigmoid function to approximate ctr rate, a float value between 0 and 1.
+            act=paddle.activation.Sigmoid())
+
+        if not self.is_infer:
+            self.train_cost = paddle.layer.multi_binary_label_cross_entropy_cost(
+                input=self.output, label=self.click)
+        return self.output
+
+    def _build_regression_model(self, dnn, lr):
+        merge_layer = layer.concat(input=[dnn, lr])
+        self.output = layer.fc(
+            input=merge_layer, size=1, act=paddle.activation.Sigmoid())
+        if not self.is_infer:
+            self.train_cost = paddle.layer.mse_cost(
+                input=self.output, label=self.click)
+        return self.output

ctr/reader.py

+from utils import logger, TaskMode, load_dnn_input_record, load_lr_input_record
+
+feeding_index = {'dnn_input': 0, 'lr_input': 1, 'click': 2}
+
+
+class Dataset(object):
+    def __init__(self):
+        self.mode = TaskMode.create_train()
+
+    def train(self, path):
+        '''
+        Load trainset.
+        '''
+        logger.info("load trainset from %s" % path)
+        self.mode = TaskMode.create_train()
+        self.path = path
+        return self._parse
+
+    def test(self, path):
+        '''
+        Load testset.
+        '''
+        logger.info("load testset from %s" % path)
+        self.path = path
+        self.mode = TaskMode.create_test()
+        return self._parse
+
+    def infer(self, path):
+        '''
+        Load infer set.
+        '''
+        logger.info("load inferset from %s" % path)
+        self.path = path
+        self.mode = TaskMode.create_infer()
+        return self._parse
+
+    def _parse(self):
+        '''
+        Parse dataset.
+        '''
+        with open(self.path) as f:
+            for line_id, line in enumerate(f):
+                fs = line.strip().split('\t')
+                dnn_input = load_dnn_input_record(fs[0])
+                lr_input = load_lr_input_record(fs[1])
+                if not self.mode.is_infer():
+                    click = [int(fs[2])]
+                    yield dnn_input, lr_input, click
+                else:
+                    yield dnn_input, lr_input
+
+
+def load_data_meta(path):
+    '''
+    load data meta info from path, return (dnn_input_dim, lr_input_dim)
+    '''
+    with open(path) as f:
+        lines = f.read().split('\n')
+        err_info = "wrong meta format"
+        assert len(lines) == 2, err_info
+        assert 'dnn_input_dim:' in lines[0] and 'lr_input_dim:' in lines[
+            1], err_info
+        res = map(int, [_.split(':')[1] for _ in lines])
+        logger.info('dnn input dim: %d' % res[0])
+        logger.info('lr input dim: %d' % res[1])
+        return res

ctr/train.py

 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
+# -*- coding: utf-8 -*-import os
 import argparse
-import logging
-import paddle.v2 as paddle
-from paddle.v2 import layer
-from paddle.v2 import data_type as dtype
-from data_provider import field_index, detect_dataset, AvazuDataset
-
-parser = argparse.ArgumentParser(description="PaddlePaddle CTR example")
-parser.add_argument(
-    '--train_data_path',
-    type=str,
-    required=True,
-    help="path of training dataset")
-parser.add_argument(
-    '--batch_size',
-    type=int,
-    default=10000,
-    help="size of mini-batch (default:10000)")
-parser.add_argument(
-    '--test_set_size',
-    type=int,
-    default=10000,
-    help="size of the validation dataset(default: 10000)")
-parser.add_argument(
-    '--num_passes', type=int, default=10, help="number of passes to train")
-parser.add_argument(
-    '--num_lines_to_detact',
-    type=int,
-    default=500000,
-    help="number of records to detect dataset's meta info")
-
-args = parser.parse_args()
-
-dnn_layer_dims = [128, 64, 32, 1]
-data_meta_info = detect_dataset(args.train_data_path, args.num_lines_to_detact)
-
-logging.warning('detect categorical fields in dataset %s' %
-                args.train_data_path)
-for key, item in data_meta_info.items():
-    logging.warning('    - {}\t{}'.format(key, item))
-
-paddle.init(use_gpu=False, trainer_count=1)
+import gzip
 
-# ==============================================================================
-#                    input layers
-# ==============================================================================
-dnn_merged_input = layer.data(
-    name='dnn_input',
-    type=paddle.data_type.sparse_binary_vector(data_meta_info['dnn_input']))
-
-lr_merged_input = layer.data(
-    name='lr_input',
-    type=paddle.data_type.sparse_binary_vector(data_meta_info['lr_input']))
-
-click = paddle.layer.data(name='click', type=dtype.dense_vector(1))
+import reader
+import paddle.v2 as paddle
+from utils import logger, ModelType
+from network_conf import CTRmodel
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="PaddlePaddle CTR example")
+    parser.add_argument(
+        '--train_data_path',
+        type=str,
+        required=True,
+        help="path of training dataset")
+    parser.add_argument(
+        '--test_data_path', type=str, help='path of testing dataset')
+    parser.add_argument(
+        '--batch_size',
+        type=int,
+        default=10000,
+        help="size of mini-batch (default:10000)")
+    parser.add_argument(
+        '--num_passes', type=int, default=10, help="number of passes to train")
+    parser.add_argument(
+        '--model_output_prefix',
+        type=str,
+        default='./ctr_models',
+        help='prefix of path for model to store (default: ./ctr_models)')
+    parser.add_argument(
+        '--data_meta_file',
+        type=str,
+        required=True,
+        help='path of data meta info file', )
+    parser.add_argument(
+        '--model_type',
+        type=int,
+        required=True,
+        default=ModelType.CLASSIFICATION,
+        help='model type, classification: %d, regression %d (default classification)'
+        % (ModelType.CLASSIFICATION, ModelType.REGRESSION))
+
+    return parser.parse_args()
 
 
-# ==============================================================================
-#                    network structure
-# ==============================================================================
-def build_dnn_submodel(dnn_layer_dims):
-    dnn_embedding = layer.fc(input=dnn_merged_input, size=dnn_layer_dims[0])
-    _input_layer = dnn_embedding
-    for i, dim in enumerate(dnn_layer_dims[1:]):
-        fc = layer.fc(
-            input=_input_layer,
-            size=dim,
-            act=paddle.activation.Relu(),
-            name='dnn-fc-%d' % i)
-        _input_layer = fc
-    return _input_layer
-
-
-# config LR submodel
-def build_lr_submodel():
-    fc = layer.fc(
-        input=lr_merged_input, size=1, name='lr', act=paddle.activation.Relu())
-    return fc
-
-
-# conbine DNN and LR submodels
-def combine_submodels(dnn, lr):
-    merge_layer = layer.concat(input=[dnn, lr])
-    fc = layer.fc(
-        input=merge_layer,
-        size=1,
-        name='output',
-        # use sigmoid function to approximate ctr rate, a float value between 0 and 1.
-        act=paddle.activation.Sigmoid())
-    return fc
-
-
-dnn = build_dnn_submodel(dnn_layer_dims)
-lr = build_lr_submodel()
-output = combine_submodels(dnn, lr)
+dnn_layer_dims = [128, 64, 32, 1]
 
 # ==============================================================================
 #                   cost and train period
 # ==============================================================================
-classification_cost = paddle.layer.multi_binary_label_cross_entropy_cost(
-    input=output, label=click)
-
-params = paddle.parameters.create(classification_cost)
-
-optimizer = paddle.optimizer.Momentum(momentum=0.01)
-
-trainer = paddle.trainer.SGD(
-    cost=classification_cost, parameters=params, update_equation=optimizer)
-
-dataset = AvazuDataset(
-    args.train_data_path, n_records_as_test=args.test_set_size)
-
-
-def event_handler(event):
-    if isinstance(event, paddle.event.EndIteration):
-        num_samples = event.batch_id * args.batch_size
-        if event.batch_id % 100 == 0:
-            logging.warning("Pass %d, Samples %d, Cost %f" %
-                            (event.pass_id, num_samples, event.cost))
-
-        if event.batch_id % 1000 == 0:
-            result = trainer.test(
-                reader=paddle.batch(dataset.test, batch_size=args.batch_size),
-                feeding=field_index)
-            logging.warning("Test %d-%d, Cost %f" %
-                            (event.pass_id, event.batch_id, result.cost))
 
 
-trainer.train(
-    reader=paddle.batch(
-        paddle.reader.shuffle(dataset.train, buf_size=500),
-        batch_size=args.batch_size),
-    feeding=field_index,
-    event_handler=event_handler,
-    num_passes=args.num_passes)
+def train():
+    args = parse_args()
+    args.model_type = ModelType(args.model_type)
+    paddle.init(use_gpu=False, trainer_count=1)
+    dnn_input_dim, lr_input_dim = reader.load_data_meta(args.data_meta_file)
+
+    # create ctr model.
+    model = CTRmodel(
+        dnn_layer_dims,
+        dnn_input_dim,
+        lr_input_dim,
+        model_type=args.model_type,
+        is_infer=False)
+
+    params = paddle.parameters.create(model.train_cost)
+    optimizer = paddle.optimizer.AdaGrad()
+
+    trainer = paddle.trainer.SGD(
+        cost=model.train_cost, parameters=params, update_equation=optimizer)
+
+    dataset = reader.Dataset()
+
+    def __event_handler__(event):
+        if isinstance(event, paddle.event.EndIteration):
+            num_samples = event.batch_id * args.batch_size
+            if event.batch_id % 100 == 0:
+                logger.warning("Pass %d, Samples %d, Cost %f, %s" % (
+                    event.pass_id, num_samples, event.cost, event.metrics))
+
+            if event.batch_id % 1000 == 0:
+                if args.test_data_path:
+                    result = trainer.test(
+                        reader=paddle.batch(
+                            dataset.test(args.test_data_path),
+                            batch_size=args.batch_size),
+                        feeding=reader.feeding_index)
+                    logger.warning("Test %d-%d, Cost %f, %s" %
+                                   (event.pass_id, event.batch_id, result.cost,
+                                    result.metrics))
+
+                path = "{}-pass-{}-batch-{}-test-{}.tar.gz".format(
+                    args.model_output_prefix, event.pass_id, event.batch_id,
+                    result.cost)
+                with gzip.open(path, 'w') as f:
+                    params.to_tar(f)
+
+    trainer.train(
+        reader=paddle.batch(
+            paddle.reader.shuffle(
+                dataset.train(args.train_data_path), buf_size=500),
+            batch_size=args.batch_size),
+        feeding=reader.feeding_index,
+        event_handler=__event_handler__,
+        num_passes=args.num_passes)
+
+
+if __name__ == '__main__':
+    train()

ctr/utils.py

+import logging
+
+logging.basicConfig()
+logger = logging.getLogger("paddle")
+logger.setLevel(logging.INFO)
+
+
+class TaskMode:
+    TRAIN_MODE = 0
+    TEST_MODE = 1
+    INFER_MODE = 2
+
+    def __init__(self, mode):
+        self.mode = mode
+
+    def is_train(self):
+        return self.mode == self.TRAIN_MODE
+
+    def is_test(self):
+        return self.mode == self.TEST_MODE
+
+    def is_infer(self):
+        return self.mode == self.INFER_MODE
+
+    @staticmethod
+    def create_train():
+        return TaskMode(TaskMode.TRAIN_MODE)
+
+    @staticmethod
+    def create_test():
+        return TaskMode(TaskMode.TEST_MODE)
+
+    @staticmethod
+    def create_infer():
+        return TaskMode(TaskMode.INFER_MODE)
+
+
+class ModelType:
+    CLASSIFICATION = 0
+    REGRESSION = 1
+
+    def __init__(self, mode):
+        self.mode = mode
+
+    def is_classification(self):
+        return self.mode == self.CLASSIFICATION
+
+    def is_regression(self):
+        return self.mode == self.REGRESSION
+
+    @staticmethod
+    def create_classification():
+        return ModelType(ModelType.CLASSIFICATION)
+
+    @staticmethod
+    def create_regression():
+        return ModelType(ModelType.REGRESSION)
+
+
+def load_dnn_input_record(sent):
+    return map(int, sent.split())
+
+
+def load_lr_input_record(sent):
+    res = []
+    for _ in [x.split(':') for x in sent.split()]:
+        res.append((int(_[0]), float(_[1]), ))
+    return res

deep_speech_2/README.md

@@ -38,7 +38,13 @@ python datasets/librispeech/librispeech.py --help
 python compute_mean_std.py
 ```
 
-`python compute_mean_std.py` computes mean and stdandard deviation for audio features, and save them to a file with a default name `./mean_std.npz`. This file will be used in both training and inferencing.
+It will compute mean and stdandard deviation for audio features, and save them to a file with a default name `./mean_std.npz`. This file will be used in both training and inferencing. The default feature of audio data is power spectrum, and the mfcc feature is also supported. To train and infer based on mfcc feature, please generate this file by
+
+```
+python compute_mean_std.py --specgram_type mfcc
+```
+
+and specify ```--specgram_type mfcc``` when running train.py, infer.py, evaluator.py or tune.py.
 
 More help for arguments:
 
@@ -66,14 +72,69 @@ More help for arguments:
 python train.py --help
 ```
 
-### Inferencing
+### Preparing language model
+
+The following steps, inference, parameters tuning and evaluating, will require a language model during decoding.
+A compressed language model is provided and can be accessed by
+
+```
+cd ./lm
+sh run.sh
+cd ..
+```
+
+### Inference
+
+For GPU inference
 
 ```
 CUDA_VISIBLE_DEVICES=0 python infer.py
 ```
 
+For CPU inference
+
+```
+python infer.py --use_gpu=False
+```
+
 More help for arguments:
 
 ```
 python infer.py --help
 ```
+
+### Evaluating
+
+```
+CUDA_VISIBLE_DEVICES=0 python evaluate.py
+```
+
+More help for arguments:
+
+```
+python evaluate.py --help
+```
+
+### Parameters tuning
+
+Usually, the parameters $\alpha$ and $\beta$ for the CTC [prefix beam search](https://arxiv.org/abs/1408.2873) decoder need to be tuned after retraining the acoustic model.
+
+For GPU tuning
+
+```
+CUDA_VISIBLE_DEVICES=0 python tune.py
+```
+
+For CPU tuning
+
+```
+python tune.py --use_gpu=False
+```
+
+More help for arguments:
+
+```
+python tune.py --help
+```
+
+Then reset parameters with the tuning result before inference or evaluating.

deep_speech_2/compute_mean_std.py

@@ -10,6 +10,12 @@ from data_utils.featurizer.audio_featurizer import AudioFeaturizer
 
 parser = argparse.ArgumentParser(
     description='Computing mean and stddev for feature normalizer.')
+parser.add_argument(
+    "--specgram_type",
+    default='linear',
+    type=str,
+    help="Feature type of audio data: 'linear' (power spectrum)"
+    " or 'mfcc'. (default: %(default)s)")
 parser.add_argument(
     "--manifest_path",
     default='datasets/manifest.train',
@@ -39,7 +45,7 @@ args = parser.parse_args()
 
 def main():
     augmentation_pipeline = AugmentationPipeline(args.augmentation_config)
-    audio_featurizer = AudioFeaturizer()
+    audio_featurizer = AudioFeaturizer(specgram_type=args.specgram_type)
 
     def augment_and_featurize(audio_segment):
         augmentation_pipeline.transform_audio(audio_segment)

deep_speech_2/data_utils/featurizer/audio_featurizer.py

@@ -6,13 +6,15 @@ from __future__ import print_function
 import numpy as np
 from data_utils import utils
 from data_utils.audio import AudioSegment
+from python_speech_features import mfcc
+from python_speech_features import delta
 
 
 class AudioFeaturizer(object):
     """Audio featurizer, for extracting features from audio contents of
     AudioSegment or SpeechSegment.
 
-    Currently, it only supports feature type of linear spectrogram.
+    Currently, it supports feature types of linear spectrogram and mfcc.
 
     :param specgram_type: Specgram feature type. Options: 'linear'.
     :type specgram_type: str
@@ -20,9 +22,10 @@ class AudioFeaturizer(object):
     :type stride_ms: float
     :param window_ms: Window size (in milliseconds) for generating frames.
     :type window_ms: float
-    :param max_freq: Used when specgram_type is 'linear', only FFT bins
+    :param max_freq: When specgram_type is 'linear', only FFT bins
                      corresponding to frequencies between [0, max_freq] are
-                     returned.
+                     returned; when specgram_type is 'mfcc', max_feq is the
+                     highest band edge of mel filters.
     :types max_freq: None|float
     :param target_sample_rate: Audio are resampled (if upsampling or
                                downsampling is allowed) to this before
@@ -91,6 +94,9 @@ class AudioFeaturizer(object):
             return self._compute_linear_specgram(
                 samples, sample_rate, self._stride_ms, self._window_ms,
                 self._max_freq)
+        elif self._specgram_type == 'mfcc':
+            return self._compute_mfcc(samples, sample_rate, self._stride_ms,
+                                      self._window_ms, self._max_freq)
         else:
             raise ValueError("Unknown specgram_type %s. "
                              "Supported values: linear." % self._specgram_type)
@@ -142,3 +148,39 @@ class AudioFeaturizer(object):
         # prepare fft frequency list
         freqs = float(sample_rate) / window_size * np.arange(fft.shape[0])
         return fft, freqs
+
+    def _compute_mfcc(self,
+                      samples,
+                      sample_rate,
+                      stride_ms=10.0,
+                      window_ms=20.0,
+                      max_freq=None):
+        """Compute mfcc from samples."""
+        if max_freq is None:
+            max_freq = sample_rate / 2
+        if max_freq > sample_rate / 2:
+            raise ValueError("max_freq must be greater than half of "
+                             "sample rate.")
+        if stride_ms > window_ms:
+            raise ValueError("Stride size must not be greater than "
+                             "window size.")
+        # compute 13 cepstral coefficients, and the first one is replaced
+        # by log(frame energy)
+        mfcc_feat = mfcc(
+            signal=samples,
+            samplerate=sample_rate,
+            winlen=0.001 * window_ms,
+            winstep=0.001 * stride_ms,
+            highfreq=max_freq)
+        # Deltas
+        d_mfcc_feat = delta(mfcc_feat, 2)
+        # Deltas-Deltas
+        dd_mfcc_feat = delta(d_mfcc_feat, 2)
+        # concat above three features
+        concat_mfcc_feat = [
+            np.concatenate((mfcc_feat[i], d_mfcc_feat[i], dd_mfcc_feat[i]))
+            for i in xrange(len(mfcc_feat))
+        ]
+        # transpose to be consistent with the linear specgram situation
+        concat_mfcc_feat = np.transpose(concat_mfcc_feat)
+        return concat_mfcc_feat

deep_speech_2/data_utils/featurizer/speech_featurizer.py

@@ -11,23 +11,24 @@ class SpeechFeaturizer(object):
     """Speech featurizer, for extracting features from both audio and transcript
     contents of SpeechSegment.
 
-    Currently, for audio parts, it only supports feature type of linear
-    spectrogram; for transcript parts, it only supports char-level tokenizing
-    and conversion into a list of token indices. Note that the token indexing
-    order follows the given vocabulary file.
+    Currently, for audio parts, it supports feature types of linear
+    spectrogram and mfcc; for transcript parts, it only supports char-level
+    tokenizing and conversion into a list of token indices. Note that the
+    token indexing order follows the given vocabulary file.
 
     :param vocab_filepath: Filepath to load vocabulary for token indices
                            conversion.
     :type specgram_type: basestring
-    :param specgram_type: Specgram feature type. Options: 'linear'.
+    :param specgram_type: Specgram feature type. Options: 'linear', 'mfcc'.
     :type specgram_type: str
     :param stride_ms: Striding size (in milliseconds) for generating frames.
     :type stride_ms: float
     :param window_ms: Window size (in milliseconds) for generating frames.
     :type window_ms: float
-    :param max_freq: Used when specgram_type is 'linear', only FFT bins
+    :param max_freq: When specgram_type is 'linear', only FFT bins
                      corresponding to frequencies between [0, max_freq] are
-                     returned.
+                     returned; when specgram_type is 'mfcc', max_freq is the
+                     highest band edge of mel filters.
     :types max_freq: None|float
     :param target_sample_rate: Speech are resampled (if upsampling or
                                downsampling is allowed) to this before

deep_speech_2/data_utils/normalizer.py

@@ -16,7 +16,7 @@ class FeatureNormalizer(object):
     if mean_std_filepath is provided (not None), the normalizer will directly
     initilize from the file. Otherwise, both manifest_path and featurize_func
     should be given for on-the-fly mean and stddev computing.
-    
+
     :param mean_std_filepath: File containing the pre-computed mean and stddev.
     :type mean_std_filepath: None|basestring
     :param manifest_path: Manifest of instances for computing mean and stddev.

deep_speech_2/decoder.py

-"""Contains various CTC decoder."""
+"""Contains various CTC decoders."""
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
-import numpy as np
 from itertools import groupby
+import numpy as np
+from math import log
+import multiprocessing
 
 
-def ctc_best_path_decode(probs_seq, vocabulary):
-    """Best path decoding, also called argmax decoding or greedy decoding.
+def ctc_best_path_decoder(probs_seq, vocabulary):
+    """Best path decoder, also called argmax decoder or greedy decoder.
     Path consisting of the most probable tokens are further post-processed to
     remove consecutive repetitions and all blanks.
 
@@ -36,24 +38,200 @@ def ctc_best_path_decode(probs_seq, vocabulary):
     return ''.join([vocabulary[index] for index in index_list])
 
 
-def ctc_decode(probs_seq, vocabulary, method):
-    """CTC-like sequence decoding from a sequence of likelihood probablilites.
+def ctc_beam_search_decoder(probs_seq,
+                            beam_size,
+                            vocabulary,
+                            blank_id,
+                            cutoff_prob=1.0,
+                            ext_scoring_func=None,
+                            nproc=False):
+    """Beam search decoder for CTC-trained network. It utilizes beam search
+    to approximately select top best decoding labels and returning results
+    in the descending order. The implementation is based on Prefix
+    Beam Search (https://arxiv.org/abs/1408.2873), and the unclear part is
+    redesigned. Two important modifications: 1) in the iterative computation
+    of probabilities, the assignment operation is changed to accumulation for
+    one prefix may comes from different paths; 2) the if condition "if l^+ not
+    in A_prev then" after probabilities' computation is deprecated for it is
+    hard to understand and seems unnecessary.
 
-    :param probs_seq: 2-D list of probabilities over the vocabulary for each
-                      character. Each element is a list of float probabilities
-                      for one character.
-    :type probs_seq: list
+    :param probs_seq: 2-D list of probability distributions over each time
+                      step, with each element being a list of normalized
+                      probabilities over vocabulary and blank.
+    :type probs_seq: 2-D list
+    :param beam_size: Width for beam search.
+    :type beam_size: int
     :param vocabulary: Vocabulary list.
     :type vocabulary: list
-    :param method: Decoding method name, with options: "best_path".
-    :type method: basestring
-    :return: Decoding result string.
-    :rtype: baseline
+    :param blank_id: ID of blank.
+    :type blank_id: int
+    :param cutoff_prob: Cutoff probability in pruning,
+                        default 1.0, no pruning.
+    :type cutoff_prob: float
+    :param ext_scoring_func: External scoring function for
+                            partially decoded sentence, e.g. word count
+                            or language model.
+    :type external_scoring_func: callable
+    :param nproc: Whether the decoder used in multiprocesses.
+    :type nproc: bool
+    :return: List of tuples of log probability and sentence as decoding
+             results, in descending order of the probability.
+    :rtype: list
     """
+    # dimension check
     for prob_list in probs_seq:
         if not len(prob_list) == len(vocabulary) + 1:
-            raise ValueError("probs dimension mismatchedd with vocabulary")
-    if method == "best_path":
-        return ctc_best_path_decode(probs_seq, vocabulary)
-    else:
-        raise ValueError("Decoding method [%s] is not supported.")
+            raise ValueError("The shape of prob_seq does not match with the "
+                             "shape of the vocabulary.")
+
+    # blank_id check
+    if not blank_id < len(probs_seq[0]):
+        raise ValueError("blank_id shouldn't be greater than probs dimension")
+
+    # If the decoder called in the multiprocesses, then use the global scorer
+    # instantiated in ctc_beam_search_decoder_batch().
+    if nproc is True:
+        global ext_nproc_scorer
+        ext_scoring_func = ext_nproc_scorer
+
+    ## initialize
+    # prefix_set_prev: the set containing selected prefixes
+    # probs_b_prev: prefixes' probability ending with blank in previous step
+    # probs_nb_prev: prefixes' probability ending with non-blank in previous step
+    prefix_set_prev = {'\t': 1.0}
+    probs_b_prev, probs_nb_prev = {'\t': 1.0}, {'\t': 0.0}
+
+    ## extend prefix in loop
+    for time_step in xrange(len(probs_seq)):
+        # prefix_set_next: the set containing candidate prefixes
+        # probs_b_cur: prefixes' probability ending with blank in current step
+        # probs_nb_cur: prefixes' probability ending with non-blank in current step
+        prefix_set_next, probs_b_cur, probs_nb_cur = {}, {}, {}
+
+        prob_idx = list(enumerate(probs_seq[time_step]))
+        cutoff_len = len(prob_idx)
+        #If pruning is enabled
+        if cutoff_prob < 1.0:
+            prob_idx = sorted(prob_idx, key=lambda asd: asd[1], reverse=True)
+            cutoff_len, cum_prob = 0, 0.0
+            for i in xrange(len(prob_idx)):
+                cum_prob += prob_idx[i][1]
+                cutoff_len += 1
+                if cum_prob >= cutoff_prob:
+                    break
+            prob_idx = prob_idx[0:cutoff_len]
+
+        for l in prefix_set_prev:
+            if not prefix_set_next.has_key(l):
+                probs_b_cur[l], probs_nb_cur[l] = 0.0, 0.0
+
+            # extend prefix by travering prob_idx
+            for index in xrange(cutoff_len):
+                c, prob_c = prob_idx[index][0], prob_idx[index][1]
+
+                if c == blank_id:
+                    probs_b_cur[l] += prob_c * (
+                        probs_b_prev[l] + probs_nb_prev[l])
+                else:
+                    last_char = l[-1]
+                    new_char = vocabulary[c]
+                    l_plus = l + new_char
+                    if not prefix_set_next.has_key(l_plus):
+                        probs_b_cur[l_plus], probs_nb_cur[l_plus] = 0.0, 0.0
+
+                    if new_char == last_char:
+                        probs_nb_cur[l_plus] += prob_c * probs_b_prev[l]
+                        probs_nb_cur[l] += prob_c * probs_nb_prev[l]
+                    elif new_char == ' ':
+                        if (ext_scoring_func is None) or (len(l) == 1):
+                            score = 1.0
+                        else:
+                            prefix = l[1:]
+                            score = ext_scoring_func(prefix)
+                        probs_nb_cur[l_plus] += score * prob_c * (
+                            probs_b_prev[l] + probs_nb_prev[l])
+                    else:
+                        probs_nb_cur[l_plus] += prob_c * (
+                            probs_b_prev[l] + probs_nb_prev[l])
+                    # add l_plus into prefix_set_next
+                    prefix_set_next[l_plus] = probs_nb_cur[
+                        l_plus] + probs_b_cur[l_plus]
+            # add l into prefix_set_next
+            prefix_set_next[l] = probs_b_cur[l] + probs_nb_cur[l]
+        # update probs
+        probs_b_prev, probs_nb_prev = probs_b_cur, probs_nb_cur
+
+        ## store top beam_size prefixes
+        prefix_set_prev = sorted(
+            prefix_set_next.iteritems(), key=lambda asd: asd[1], reverse=True)
+        if beam_size < len(prefix_set_prev):
+            prefix_set_prev = prefix_set_prev[:beam_size]
+        prefix_set_prev = dict(prefix_set_prev)
+
+    beam_result = []
+    for seq, prob in prefix_set_prev.items():
+        if prob > 0.0 and len(seq) > 1:
+            result = seq[1:]
+            # score last word by external scorer
+            if (ext_scoring_func is not None) and (result[-1] != ' '):
+                prob = prob * ext_scoring_func(result)
+            log_prob = log(prob)
+            beam_result.append((log_prob, result))
+
+    ## output top beam_size decoding results
+    beam_result = sorted(beam_result, key=lambda asd: asd[0], reverse=True)
+    return beam_result
+
+
+def ctc_beam_search_decoder_batch(probs_split,
+                                  beam_size,
+                                  vocabulary,
+                                  blank_id,
+                                  num_processes,
+                                  cutoff_prob=1.0,
+                                  ext_scoring_func=None):
+    """CTC beam search decoder using multiple processes.
+
+    :param probs_seq: 3-D list with each element as an instance of 2-D list
+                      of probabilities used by ctc_beam_search_decoder().
+    :type probs_seq: 3-D list
+    :param beam_size: Width for beam search.
+    :type beam_size: int
+    :param vocabulary: Vocabulary list.
+    :type vocabulary: list
+    :param blank_id: ID of blank.
+    :type blank_id: int
+    :param num_processes: Number of parallel processes.
+    :type num_processes: int
+    :param cutoff_prob: Cutoff probability in pruning,
+                        default 1.0, no pruning.
+    :param num_processes: Number of parallel processes.
+    :type num_processes: int
+    :type cutoff_prob: float
+    :param ext_scoring_func: External scoring function for
+                            partially decoded sentence, e.g. word count
+                            or language model.
+    :type external_scoring_function: callable
+    :return: List of tuples of log probability and sentence as decoding
+             results, in descending order of the probability.
+    :rtype: list
+    """
+    if not num_processes > 0:
+        raise ValueError("Number of processes must be positive!")
+
+    # use global variable to pass the externnal scorer to beam search decoder
+    global ext_nproc_scorer
+    ext_nproc_scorer = ext_scoring_func
+    nproc = True
+
+    pool = multiprocessing.Pool(processes=num_processes)
+    results = []
+    for i, probs_list in enumerate(probs_split):
+        args = (probs_list, beam_size, vocabulary, blank_id, cutoff_prob, None,
+                nproc)
+        results.append(pool.apply_async(ctc_beam_search_decoder, args))
+
+    pool.close()
+    pool.join()
+    beam_search_results = [result.get() for result in results]
+    return beam_search_results

deep_speech_2/evaluate.py

+"""Evaluation for DeepSpeech2 model."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import distutils.util
+import argparse
+import gzip
+import paddle.v2 as paddle
+from data_utils.data import DataGenerator
+from model import deep_speech2
+from decoder import *
+from lm.lm_scorer import LmScorer
+from error_rate import wer
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument(
+    "--batch_size",
+    default=100,
+    type=int,
+    help="Minibatch size for evaluation. (default: %(default)s)")
+parser.add_argument(
+    "--num_conv_layers",
+    default=2,
+    type=int,
+    help="Convolution layer number. (default: %(default)s)")
+parser.add_argument(
+    "--num_rnn_layers",
+    default=3,
+    type=int,
+    help="RNN layer number. (default: %(default)s)")
+parser.add_argument(
+    "--rnn_layer_size",
+    default=512,
+    type=int,
+    help="RNN layer cell number. (default: %(default)s)")
+parser.add_argument(
+    "--use_gpu",
+    default=True,
+    type=distutils.util.strtobool,
+    help="Use gpu or not. (default: %(default)s)")
+parser.add_argument(
+    "--num_threads_data",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu threads for preprocessing data. (default: %(default)s)")
+parser.add_argument(
+    "--num_processes_beam_search",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu processes for beam search. (default: %(default)s)")
+parser.add_argument(
+    "--mean_std_filepath",
+    default='mean_std.npz',
+    type=str,
+    help="Manifest path for normalizer. (default: %(default)s)")
+parser.add_argument(
+    "--decode_method",
+    default='beam_search',
+    type=str,
+    help="Method for ctc decoding, best_path or beam_search. (default: %(default)s)"
+)
+parser.add_argument(
+    "--language_model_path",
+    default="lm/data/common_crawl_00.prune01111.trie.klm",
+    type=str,
+    help="Path for language model. (default: %(default)s)")
+parser.add_argument(
+    "--alpha",
+    default=0.26,
+    type=float,
+    help="Parameter associated with language model. (default: %(default)f)")
+parser.add_argument(
+    "--beta",
+    default=0.1,
+    type=float,
+    help="Parameter associated with word count. (default: %(default)f)")
+parser.add_argument(
+    "--cutoff_prob",
+    default=0.99,
+    type=float,
+    help="The cutoff probability of pruning"
+    "in beam search. (default: %(default)f)")
+parser.add_argument(
+    "--beam_size",
+    default=500,
+    type=int,
+    help="Width for beam search decoding. (default: %(default)d)")
+parser.add_argument(
+    "--specgram_type",
+    default='linear',
+    type=str,
+    help="Feature type of audio data: 'linear' (power spectrum)"
+    " or 'mfcc'. (default: %(default)s)")
+parser.add_argument(
+    "--decode_manifest_path",
+    default='datasets/manifest.test',
+    type=str,
+    help="Manifest path for decoding. (default: %(default)s)")
+parser.add_argument(
+    "--model_filepath",
+    default='checkpoints/params.latest.tar.gz',
+    type=str,
+    help="Model filepath. (default: %(default)s)")
+parser.add_argument(
+    "--vocab_filepath",
+    default='datasets/vocab/eng_vocab.txt',
+    type=str,
+    help="Vocabulary filepath. (default: %(default)s)")
+args = parser.parse_args()
+
+
+def evaluate():
+    """Evaluate on whole test data for DeepSpeech2."""
+    # initialize data generator
+    data_generator = DataGenerator(
+        vocab_filepath=args.vocab_filepath,
+        mean_std_filepath=args.mean_std_filepath,
+        augmentation_config='{}',
+        specgram_type=args.specgram_type,
+        num_threads=args.num_threads_data)
+
+    # create network config
+    # paddle.data_type.dense_array is used for variable batch input.
+    # The size 161 * 161 is only an placeholder value and the real shape
+    # of input batch data will be induced during training.
+    audio_data = paddle.layer.data(
+        name="audio_spectrogram", type=paddle.data_type.dense_array(161 * 161))
+    text_data = paddle.layer.data(
+        name="transcript_text",
+        type=paddle.data_type.integer_value_sequence(data_generator.vocab_size))
+    output_probs = deep_speech2(
+        audio_data=audio_data,
+        text_data=text_data,
+        dict_size=data_generator.vocab_size,
+        num_conv_layers=args.num_conv_layers,
+        num_rnn_layers=args.num_rnn_layers,
+        rnn_size=args.rnn_layer_size,
+        is_inference=True)
+
+    # load parameters
+    parameters = paddle.parameters.Parameters.from_tar(
+        gzip.open(args.model_filepath))
+
+    # prepare infer data
+    batch_reader = data_generator.batch_reader_creator(
+        manifest_path=args.decode_manifest_path,
+        batch_size=args.batch_size,
+        min_batch_size=1,
+        sortagrad=False,
+        shuffle_method=None)
+
+    # define inferer
+    inferer = paddle.inference.Inference(
+        output_layer=output_probs, parameters=parameters)
+
+    # initialize external scorer for beam search decoding
+    if args.decode_method == 'beam_search':
+        ext_scorer = LmScorer(args.alpha, args.beta, args.language_model_path)
+
+    wer_counter, wer_sum = 0, 0.0
+    for infer_data in batch_reader():
+        # run inference
+        infer_results = inferer.infer(input=infer_data)
+        num_steps = len(infer_results) // len(infer_data)
+        probs_split = [
+            infer_results[i * num_steps:(i + 1) * num_steps]
+            for i in xrange(0, len(infer_data))
+        ]
+        # target transcription
+        target_transcription = [
+            ''.join([
+                data_generator.vocab_list[index] for index in infer_data[i][1]
+            ]) for i, probs in enumerate(probs_split)
+        ]
+        # decode and print
+        # best path decode
+        if args.decode_method == "best_path":
+            for i, probs in enumerate(probs_split):
+                output_transcription = ctc_best_path_decoder(
+                    probs_seq=probs, vocabulary=data_generator.vocab_list)
+                wer_sum += wer(target_transcription[i], output_transcription)
+                wer_counter += 1
+        # beam search decode
+        elif args.decode_method == "beam_search":
+            # beam search using multiple processes
+            beam_search_results = ctc_beam_search_decoder_batch(
+                probs_split=probs_split,
+                vocabulary=data_generator.vocab_list,
+                beam_size=args.beam_size,
+                blank_id=len(data_generator.vocab_list),
+                num_processes=args.num_processes_beam_search,
+                ext_scoring_func=ext_scorer,
+                cutoff_prob=args.cutoff_prob, )
+            for i, beam_search_result in enumerate(beam_search_results):
+                wer_sum += wer(target_transcription[i],
+                               beam_search_result[0][1])
+                wer_counter += 1
+        else:
+            raise ValueError("Decoding method [%s] is not supported." %
+                             decode_method)
+
+    print("Final WER = %f" % (wer_sum / wer_counter))
+
+
+def main():
+    paddle.init(use_gpu=args.use_gpu, trainer_count=1)
+    evaluate()
+
+
+if __name__ == '__main__':
+    main()

deep_speech_2/infer.py

@@ -10,7 +10,9 @@ import multiprocessing
 import paddle.v2 as paddle
 from data_utils.data import DataGenerator
 from model import deep_speech2
-from decoder import ctc_decode
+from decoder import *
+from lm.lm_scorer import LmScorer
+from error_rate import wer
 import utils
 
 parser = argparse.ArgumentParser(description=__doc__)
@@ -44,6 +46,17 @@ parser.add_argument(
     default=multiprocessing.cpu_count(),
     type=int,
     help="Number of cpu threads for preprocessing data. (default: %(default)s)")
+parser.add_argument(
+    "--num_processes_beam_search",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu processes for beam search. (default: %(default)s)")
+parser.add_argument(
+    "--specgram_type",
+    default='linear',
+    type=str,
+    help="Feature type of audio data: 'linear' (power spectrum)"
+    " or 'mfcc'. (default: %(default)s)")
 parser.add_argument(
     "--mean_std_filepath",
     default='mean_std.npz',
@@ -64,16 +77,54 @@ parser.add_argument(
     default='datasets/vocab/eng_vocab.txt',
     type=str,
     help="Vocabulary filepath. (default: %(default)s)")
+parser.add_argument(
+    "--decode_method",
+    default='beam_search',
+    type=str,
+    help="Method for ctc decoding: best_path or beam_search. (default: %(default)s)"
+)
+parser.add_argument(
+    "--beam_size",
+    default=500,
+    type=int,
+    help="Width for beam search decoding. (default: %(default)d)")
+parser.add_argument(
+    "--num_results_per_sample",
+    default=1,
+    type=int,
+    help="Number of output per sample in beam search. (default: %(default)d)")
+parser.add_argument(
+    "--language_model_path",
+    default="lm/data/common_crawl_00.prune01111.trie.klm",
+    type=str,
+    help="Path for language model. (default: %(default)s)")
+parser.add_argument(
+    "--alpha",
+    default=0.26,
+    type=float,
+    help="Parameter associated with language model. (default: %(default)f)")
+parser.add_argument(
+    "--beta",
+    default=0.1,
+    type=float,
+    help="Parameter associated with word count. (default: %(default)f)")
+parser.add_argument(
+    "--cutoff_prob",
+    default=0.99,
+    type=float,
+    help="The cutoff probability of pruning"
+    "in beam search. (default: %(default)f)")
 args = parser.parse_args()
 
 
 def infer():
-    """Max-ctc-decoding for DeepSpeech2."""
+    """Inference for DeepSpeech2."""
     # initialize data generator
     data_generator = DataGenerator(
         vocab_filepath=args.vocab_filepath,
         mean_std_filepath=args.mean_std_filepath,
         augmentation_config='{}',
+        specgram_type=args.specgram_type,
         num_threads=args.num_threads_data)
 
     # create network config
@@ -102,6 +153,7 @@ def infer():
     batch_reader = data_generator.batch_reader_creator(
         manifest_path=args.decode_manifest_path,
         batch_size=args.num_samples,
+        min_batch_size=1,
         sortagrad=False,
         shuffle_method=None)
     infer_data = batch_reader().next()
@@ -115,16 +167,52 @@ def infer():
         for i in xrange(len(infer_data))
     ]
 
-    # decode and print
-    for i, probs in enumerate(probs_split):
-        output_transcription = ctc_decode(
-            probs_seq=probs,
-            vocabulary=data_generator.vocab_list,
-            method="best_path")
-        target_transcription = ''.join(
+    # targe transcription
+    target_transcription = [
+        ''.join(
             [data_generator.vocab_list[index] for index in infer_data[i][1]])
-        print("Target Transcription: %s \nOutput Transcription: %s \n" %
-              (target_transcription, output_transcription))
+        for i, probs in enumerate(probs_split)
+    ]
+
+    ## decode and print
+    # best path decode
+    wer_sum, wer_counter = 0, 0
+    if args.decode_method == "best_path":
+        for i, probs in enumerate(probs_split):
+            best_path_transcription = ctc_best_path_decoder(
+                probs_seq=probs, vocabulary=data_generator.vocab_list)
+            print("\nTarget Transcription: %s\nOutput Transcription: %s" %
+                  (target_transcription[i], best_path_transcription))
+            wer_cur = wer(target_transcription[i], best_path_transcription)
+            wer_sum += wer_cur
+            wer_counter += 1
+            print("cur wer = %f, average wer = %f" %
+                  (wer_cur, wer_sum / wer_counter))
+    # beam search decode
+    elif args.decode_method == "beam_search":
+        ext_scorer = LmScorer(args.alpha, args.beta, args.language_model_path)
+        beam_search_batch_results = ctc_beam_search_decoder_batch(
+            probs_split=probs_split,
+            vocabulary=data_generator.vocab_list,
+            beam_size=args.beam_size,
+            blank_id=len(data_generator.vocab_list),
+            num_processes=args.num_processes_beam_search,
+            cutoff_prob=args.cutoff_prob,
+            ext_scoring_func=ext_scorer, )
+        for i, beam_search_result in enumerate(beam_search_batch_results):
+            print("\nTarget Transcription:\t%s" % target_transcription[i])
+            for index in xrange(args.num_results_per_sample):
+                result = beam_search_result[index]
+                #output: index, log prob, beam result
+                print("Beam %d: %f \t%s" % (index, result[0], result[1]))
+            wer_cur = wer(target_transcription[i], beam_search_result[0][1])
+            wer_sum += wer_cur
+            wer_counter += 1
+            print("cur wer = %f , average wer = %f" %
+                  (wer_cur, wer_sum / wer_counter))
+    else:
+        raise ValueError("Decoding method [%s] is not supported." %
+                         decode_method)
 
 
 def main():

deep_speech_2/lm/lm_scorer.py

+"""External Scorer for Beam Search Decoder."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import kenlm
+import numpy as np
+
+
+class LmScorer(object):
+    """External scorer to evaluate a prefix or whole sentence in
+       beam search decoding, including the score from n-gram language
+       model and word count.
+
+    :param alpha: Parameter associated with language model. Don't use
+                  language model when alpha = 0.
+    :type alpha: float
+    :param beta: Parameter associated with word count. Don't use word
+                count when beta = 0.
+    :type beta: float
+    :model_path: Path to load language model.
+    :type model_path: basestring
+    """
+
+    def __init__(self, alpha, beta, model_path):
+        self._alpha = alpha
+        self._beta = beta
+        if not os.path.isfile(model_path):
+            raise IOError("Invaid language model path: %s" % model_path)
+        self._language_model = kenlm.LanguageModel(model_path)
+
+    # n-gram language model scoring
+    def _language_model_score(self, sentence):
+        #log10 prob of last word
+        log_cond_prob = list(
+            self._language_model.full_scores(sentence, eos=False))[-1][0]
+        return np.power(10, log_cond_prob)
+
+    # word insertion term
+    def _word_count(self, sentence):
+        words = sentence.strip().split(' ')
+        return len(words)
+
+    # reset alpha and beta
+    def reset_params(self, alpha, beta):
+        self._alpha = alpha
+        self._beta = beta
+
+    # execute evaluation
+    def __call__(self, sentence, log=False):
+        """Evaluation function, gathering all the different scores
+        and return the final one.
+
+        :param sentence: The input sentence for evalutation
+        :type sentence: basestring
+        :param log: Whether return the score in log representation.
+        :type log: bool
+        :return: Evaluation score, in the decimal or log.
+        :rtype: float
+        """
+        lm = self._language_model_score(sentence)
+        word_cnt = self._word_count(sentence)
+        if log == False:
+            score = np.power(lm, self._alpha) * np.power(word_cnt, self._beta)
+        else:
+            score = self._alpha * np.log(lm) + self._beta * np.log(word_cnt)
+        return score

deep_speech_2/lm/run.sh

+echo "Downloading language model ..."
+
+mkdir data
+
+LM=common_crawl_00.prune01111.trie.klm
+MD5="099a601759d467cd0a8523ff939819c5"
+
+wget -c http://paddlepaddle.bj.bcebos.com/model_zoo/speech/$LM -P ./data
+
+echo "Checking md5sum ..."
+md5_tmp=`md5sum ./data/$LM | awk -F[' '] '{print $1}'`
+
+if [ $MD5 != $md5_tmp ]; then
+    echo "Fail to download the language model!"
+    exit 1
+fi
+
+
+

deep_speech_2/requirements.txt

 wget==3.2
 scipy==0.13.1
-resampy==0.1.5
+resampy==0.1.5
\ No newline at end of file
+https://github.com/kpu/kenlm/archive/master.zip
+python_speech_features

deep_speech_2/tests/test_decoders.py

+"""Test decoders."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import unittest
+from decoder import *
+
+
+class TestDecoders(unittest.TestCase):
+    def setUp(self):
+        self.vocab_list = ["\'", ' ', 'a', 'b', 'c', 'd']
+        self.beam_size = 20
+        self.probs_seq1 = [[
+            0.06390443, 0.21124858, 0.27323887, 0.06870235, 0.0361254,
+            0.18184413, 0.16493624
+        ], [
+            0.03309247, 0.22866108, 0.24390638, 0.09699597, 0.31895462,
+            0.0094893, 0.06890021
+        ], [
+            0.218104, 0.19992557, 0.18245131, 0.08503348, 0.14903535,
+            0.08424043, 0.08120984
+        ], [
+            0.12094152, 0.19162472, 0.01473646, 0.28045061, 0.24246305,
+            0.05206269, 0.09772094
+        ], [
+            0.1333387, 0.00550838, 0.00301669, 0.21745861, 0.20803985,
+            0.41317442, 0.01946335
+        ], [
+            0.16468227, 0.1980699, 0.1906545, 0.18963251, 0.19860937,
+            0.04377724, 0.01457421
+        ]]
+        self.probs_seq2 = [[
+            0.08034842, 0.22671944, 0.05799633, 0.36814645, 0.11307441,
+            0.04468023, 0.10903471
+        ], [
+            0.09742457, 0.12959763, 0.09435383, 0.21889204, 0.15113123,
+            0.10219457, 0.20640612
+        ], [
+            0.45033529, 0.09091417, 0.15333208, 0.07939558, 0.08649316,
+            0.12298585, 0.01654384
+        ], [
+            0.02512238, 0.22079203, 0.19664364, 0.11906379, 0.07816055,
+            0.22538587, 0.13483174
+        ], [
+            0.17928453, 0.06065261, 0.41153005, 0.1172041, 0.11880313,
+            0.07113197, 0.04139363
+        ], [
+            0.15882358, 0.1235788, 0.23376776, 0.20510435, 0.00279306,
+            0.05294827, 0.22298418
+        ]]
+        self.best_path_result = ["ac'bdc", "b'da"]
+        self.beam_search_result = ['acdc', "b'a"]
+
+    def test_best_path_decoder_1(self):
+        bst_result = ctc_best_path_decoder(self.probs_seq1, self.vocab_list)
+        self.assertEqual(bst_result, self.best_path_result[0])
+
+    def test_best_path_decoder_2(self):
+        bst_result = ctc_best_path_decoder(self.probs_seq2, self.vocab_list)
+        self.assertEqual(bst_result, self.best_path_result[1])
+
+    def test_beam_search_decoder_1(self):
+        beam_result = ctc_beam_search_decoder(
+            probs_seq=self.probs_seq1,
+            beam_size=self.beam_size,
+            vocabulary=self.vocab_list,
+            blank_id=len(self.vocab_list))
+        self.assertEqual(beam_result[0][1], self.beam_search_result[0])
+
+    def test_beam_search_decoder_2(self):
+        beam_result = ctc_beam_search_decoder(
+            probs_seq=self.probs_seq2,
+            beam_size=self.beam_size,
+            vocabulary=self.vocab_list,
+            blank_id=len(self.vocab_list))
+        self.assertEqual(beam_result[0][1], self.beam_search_result[1])
+
+    def test_beam_search_decoder_batch(self):
+        beam_results = ctc_beam_search_decoder_batch(
+            probs_split=[self.probs_seq1, self.probs_seq2],
+            beam_size=self.beam_size,
+            vocabulary=self.vocab_list,
+            blank_id=len(self.vocab_list),
+            num_processes=24)
+        self.assertEqual(beam_results[0][0][1], self.beam_search_result[0])
+        self.assertEqual(beam_results[1][0][1], self.beam_search_result[1])
+
+
+if __name__ == '__main__':
+    unittest.main()

deep_speech_2/train.py

@@ -53,6 +53,12 @@ parser.add_argument(
     default=True,
     type=distutils.util.strtobool,
     help="Use sortagrad or not. (default: %(default)s)")
+parser.add_argument(
+    "--specgram_type",
+    default='linear',
+    type=str,
+    help="Feature type of audio data: 'linear' (power spectrum)"
+    " or 'mfcc'. (default: %(default)s)")
 parser.add_argument(
     "--max_duration",
     default=27.0,
@@ -130,6 +136,7 @@ def train():
             augmentation_config=args.augmentation_config,
             max_duration=args.max_duration,
             min_duration=args.min_duration,
+            specgram_type=args.specgram_type,
             num_threads=args.num_threads_data)
 
     train_generator = data_generator()

deep_speech_2/tune.py

+"""Parameters tuning for DeepSpeech2 model."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import distutils.util
+import argparse
+import gzip
+import paddle.v2 as paddle
+from data_utils.data import DataGenerator
+from model import deep_speech2
+from decoder import *
+from lm.lm_scorer import LmScorer
+from error_rate import wer
+import utils
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument(
+    "--num_samples",
+    default=100,
+    type=int,
+    help="Number of samples for parameters tuning. (default: %(default)s)")
+parser.add_argument(
+    "--num_conv_layers",
+    default=2,
+    type=int,
+    help="Convolution layer number. (default: %(default)s)")
+parser.add_argument(
+    "--num_rnn_layers",
+    default=3,
+    type=int,
+    help="RNN layer number. (default: %(default)s)")
+parser.add_argument(
+    "--rnn_layer_size",
+    default=512,
+    type=int,
+    help="RNN layer cell number. (default: %(default)s)")
+parser.add_argument(
+    "--use_gpu",
+    default=True,
+    type=distutils.util.strtobool,
+    help="Use gpu or not. (default: %(default)s)")
+parser.add_argument(
+    "--num_threads_data",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu threads for preprocessing data. (default: %(default)s)")
+parser.add_argument(
+    "--num_processes_beam_search",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu processes for beam search. (default: %(default)s)")
+parser.add_argument(
+    "--specgram_type",
+    default='linear',
+    type=str,
+    help="Feature type of audio data: 'linear' (power spectrum)"
+    " or 'mfcc'. (default: %(default)s)")
+parser.add_argument(
+    "--mean_std_filepath",
+    default='mean_std.npz',
+    type=str,
+    help="Manifest path for normalizer. (default: %(default)s)")
+parser.add_argument(
+    "--decode_manifest_path",
+    default='datasets/manifest.test',
+    type=str,
+    help="Manifest path for decoding. (default: %(default)s)")
+parser.add_argument(
+    "--model_filepath",
+    default='checkpoints/params.latest.tar.gz',
+    type=str,
+    help="Model filepath. (default: %(default)s)")
+parser.add_argument(
+    "--vocab_filepath",
+    default='datasets/vocab/eng_vocab.txt',
+    type=str,
+    help="Vocabulary filepath. (default: %(default)s)")
+parser.add_argument(
+    "--beam_size",
+    default=500,
+    type=int,
+    help="Width for beam search decoding. (default: %(default)d)")
+parser.add_argument(
+    "--language_model_path",
+    default="lm/data/common_crawl_00.prune01111.trie.klm",
+    type=str,
+    help="Path for language model. (default: %(default)s)")
+parser.add_argument(
+    "--alpha_from",
+    default=0.1,
+    type=float,
+    help="Where alpha starts from. (default: %(default)f)")
+parser.add_argument(
+    "--num_alphas",
+    default=14,
+    type=int,
+    help="Number of candidate alphas. (default: %(default)d)")
+parser.add_argument(
+    "--alpha_to",
+    default=0.36,
+    type=float,
+    help="Where alpha ends with. (default: %(default)f)")
+parser.add_argument(
+    "--beta_from",
+    default=0.05,
+    type=float,
+    help="Where beta starts from. (default: %(default)f)")
+parser.add_argument(
+    "--num_betas",
+    default=20,
+    type=float,
+    help="Number of candidate betas. (default: %(default)d)")
+parser.add_argument(
+    "--beta_to",
+    default=1.0,
+    type=float,
+    help="Where beta ends with. (default: %(default)f)")
+parser.add_argument(
+    "--cutoff_prob",
+    default=0.99,
+    type=float,
+    help="The cutoff probability of pruning"
+    "in beam search. (default: %(default)f)")
+args = parser.parse_args()
+
+
+def tune():
+    """Tune parameters alpha and beta on one minibatch."""
+
+    if not args.num_alphas >= 0:
+        raise ValueError("num_alphas must be non-negative!")
+
+    if not args.num_betas >= 0:
+        raise ValueError("num_betas must be non-negative!")
+
+    # initialize data generator
+    data_generator = DataGenerator(
+        vocab_filepath=args.vocab_filepath,
+        mean_std_filepath=args.mean_std_filepath,
+        augmentation_config='{}',
+        specgram_type=args.specgram_type,
+        num_threads=args.num_threads_data)
+
+    # create network config
+    # paddle.data_type.dense_array is used for variable batch input.
+    # The size 161 * 161 is only an placeholder value and the real shape
+    # of input batch data will be induced during training.
+    audio_data = paddle.layer.data(
+        name="audio_spectrogram", type=paddle.data_type.dense_array(161 * 161))
+    text_data = paddle.layer.data(
+        name="transcript_text",
+        type=paddle.data_type.integer_value_sequence(data_generator.vocab_size))
+    output_probs = deep_speech2(
+        audio_data=audio_data,
+        text_data=text_data,
+        dict_size=data_generator.vocab_size,
+        num_conv_layers=args.num_conv_layers,
+        num_rnn_layers=args.num_rnn_layers,
+        rnn_size=args.rnn_layer_size,
+        is_inference=True)
+
+    # load parameters
+    parameters = paddle.parameters.Parameters.from_tar(
+        gzip.open(args.model_filepath))
+
+    # prepare infer data
+    batch_reader = data_generator.batch_reader_creator(
+        manifest_path=args.decode_manifest_path,
+        batch_size=args.num_samples,
+        sortagrad=False,
+        shuffle_method=None)
+    # get one batch data for tuning
+    infer_data = batch_reader().next()
+
+    # run inference
+    infer_results = paddle.infer(
+        output_layer=output_probs, parameters=parameters, input=infer_data)
+    num_steps = len(infer_results) // len(infer_data)
+    probs_split = [
+        infer_results[i * num_steps:(i + 1) * num_steps]
+        for i in xrange(0, len(infer_data))
+    ]
+
+    # create grid for search
+    cand_alphas = np.linspace(args.alpha_from, args.alpha_to, args.num_alphas)
+    cand_betas = np.linspace(args.beta_from, args.beta_to, args.num_betas)
+    params_grid = [(alpha, beta) for alpha in cand_alphas
+                   for beta in cand_betas]
+
+    ext_scorer = LmScorer(args.alpha_from, args.beta_from,
+                          args.language_model_path)
+    ## tune parameters in loop
+    for alpha, beta in params_grid:
+        wer_sum, wer_counter = 0, 0
+        # reset scorer
+        ext_scorer.reset_params(alpha, beta)
+        # beam search using multiple processes
+        beam_search_results = ctc_beam_search_decoder_batch(
+            probs_split=probs_split,
+            vocabulary=data_generator.vocab_list,
+            beam_size=args.beam_size,
+            cutoff_prob=args.cutoff_prob,
+            blank_id=len(data_generator.vocab_list),
+            num_processes=args.num_processes_beam_search,
+            ext_scoring_func=ext_scorer, )
+        for i, beam_search_result in enumerate(beam_search_results):
+            target_transcription = ''.join([
+                data_generator.vocab_list[index] for index in infer_data[i][1]
+            ])
+            wer_sum += wer(target_transcription, beam_search_result[0][1])
+            wer_counter += 1
+
+        print("alpha = %f\tbeta = %f\tWER = %f" %
+              (alpha, beta, wer_sum / wer_counter))
+
+
+def main():
+    paddle.init(use_gpu=args.use_gpu, trainer_count=1)
+    tune()
+
+
+if __name__ == '__main__':
+    main()

dssm/README.md

+# 深度结构化语义模型 (Deep Structured Semantic Models, DSSM)
+DSSM使用DNN模型在一个连续的语义空间中学习文本低纬的表示向量，并且建模两个句子间的语义相似度。
+本例演示如何使用 PaddlePaddle实现一个通用的DSSM 模型，用于建模两个字符串间的语义相似度，
+模型实现支持通用的数据格式，用户替换数据便可以在真实场景中使用该模型。
+
+## 背景介绍
+DSSM \[[1](##参考文献)\]是微软研究院13年提出来的经典的语义模型，用于学习两个文本之间的语义距离，
+广义上模型也可以推广和适用如下场景：
+
+1. CTR预估模型，衡量用户搜索词（Query）与候选网页集合（Documents）之间的相关联程度。
+2. 文本相关性，衡量两个字符串间的语义相关程度。
+3. 自动推荐，衡量User与被推荐的Item之间的关联程度。
+
+DSSM 已经发展成了一个框架，可以很自然地建模两个记录之间的距离关系，
+例如对于文本相关性问题，可以用余弦相似度 (cosin similarity) 来刻画语义距离；
+而对于搜索引擎的结果排序，可以在DSSM上接上Rank损失训练处一个排序模型。
+
+## 模型简介
+在原论文\[[1](#参考文献)\]中，DSSM模型用来衡量用户搜索词 Query 和文档集合 Documents 之间隐含的语义关系，模型结构如下
+
+<p align="center">
+<img src="./images/dssm.png"/><br/><br/>
+图 1. DSSM 原始结构
+</p>
+
+其贯彻的思想是， **用DNN将高维特征向量转化为低纬空间的连续向量（图中红色框部分）** ，
+**在上层用cosin similarity来衡量用户搜索词与候选文档间的语义相关性** 。
+
+在最顶层损失函数的设计上，原始模型使用类似Word2Vec中负例采样的方法，
+一个Query会抽取正例 $D+$ 和4个负例 $D-$ 整体上算条件概率用对数似然函数作为损失，
+这也就是图 1中类似 $P(D_1|Q)$ 的结构，具体细节请参考原论文。
+
+随着后续优化DSSM模型的结构得以简化\[[3](#参考文献)\]，演变为：
+
+<p align="center">
+<img src="./images/dssm2.png" width="600"/><br/><br/>
+图 2. DSSM通用结构
+</p>
+
+图中的空白方框可以用任何模型替代，比如全连接FC，卷积CNN，RNN等都可以，
+该模型结构专门用于衡量两个元素（比如字符串）间的语义距离。
+
+在现实使用中，DSSM模型会作为基础的积木，搭配上不同的损失函数来实现具体的功能，比如
+
+- 在排序学习中，将 图 2 中结构添加 pairwise rank损失，变成一个排序模型
+- 在CTR预估中，对点击与否做0，1二元分类，添加交叉熵损失变成一个分类模型
+- 在需要对一个子串打分时，可以使用余弦相似度来计算相似度，变成一个回归模型
+
+本例将尝试面向应用提供一个比较通用的解决方案，在模型任务类型上支持
+
+- 分类
+- [-1, 1] 值域内的回归
+- Pairwise-Rank
+
+在生成低纬语义向量的模型结构上，本模型支持以下三种：
+
+- FC, 多层全连接层
+- CNN，卷积神经网络
+- RNN，递归神经网络
+
+## 模型实现
+DSSM模型可以拆成三小块实现，分别是左边和右边的DNN，以及顶层的损失函数。
+在复杂任务中，左右两边DNN的结构可以是不同的，比如在原始论文中左右分别学习Query和Document的semantic vector，
+两者数据的数据不同，建议对应定制DNN的结构。
+
+本例中为了简便和通用，将左右两个DNN的结构都设为相同的，因此只有三个选项FC,CNN,RNN等。
+
+在损失函数的设计方面，也支持三种，分类, 回归, 排序；
+其中，在回归和排序两种损失中，左右两边的匹配程度通过余弦相似度（cossim）来计算；
+在分类任务中，类别预测的分布通过softmax计算。
+
+在其它教程中，对上述很多内容都有过详细的介绍，例如：
+
+- 如何CNN, FC 做文本信息提取可以参考 [text classification](https://github.com/PaddlePaddle/models/blob/develop/text_classification/README.md#模型详解)
+- RNN/GRU 的内容可以参考 [Machine Translation](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md#gated-recurrent-unit-gru)
+- Pairwise Rank即排序学习可参考 [learn to rank](https://github.com/PaddlePaddle/models/blob/develop/ltr/README.md)
+
+相关原理在此不再赘述，本文接下来的篇幅主要集中介绍使用PaddlePaddle实现这些结构上。
+
+如图3，回归和分类模型的结构很相似
+
+<p align="center">
+<img src="./images/dssm3.jpg"/><br/><br/>
+图 3. DSSM for REGRESSION or CLASSIFICATION
+</p>
+
+最重要的组成部分包括词向量，图中`(1)`,`(2)`两个低纬向量的学习器（可以用RNN/CNN/FC中的任意一种实现），
+最上层对应的损失函数。
+
+而Pairwise Rank的结构会复杂一些，类似两个 图 4. 中的结构，增加了对应的损失函数：
+
+- 模型总体思想是，用同一个source(源)为左右两个target(目标)分别打分——`(a),(b)`，学习目标是(a),(b)间的大小关系
+- `(a)`和`(b)`类似图3中结构，用于给source和target的pair打分
+- `(1)`和`(2)`的结构其实是共用的，都表示同一个source，图中为了表达效果展开成两个
+
+<p align="center">
+<img src="./images/dssm2.jpg"/><br/><br/>
+图 4. DSSM for Pairwise Rank
+</p>
+
+下面是各个部分具体的实现方法，所有的代码均包含在 `./network_conf.py` 中。
+
+
+### 创建文本的词向量表
+
+```python
+def create_embedding(self, input, prefix=''):
+    '''
+    Create an embedding table whose name has a `prefix`.
+    '''
+    logger.info("create embedding table [%s] which dimention is %d" %
+                (prefix, self.dnn_dims[0]))
+    emb = paddle.layer.embedding(
+        input=input,
+        size=self.dnn_dims[0],
+        param_attr=ParamAttr(name='%s_emb.w' % prefix))
+    return emb
+```
+
+由于输入给词向量表(embedding table)的是一个句子对应的词的ID的列表 ，因此词向量表输出的是词向量的序列。
+
+### CNN 结构实现
+
+```python
+def create_cnn(self, emb, prefix=''):
+    '''
+    A multi-layer CNN.
+
+    @emb: paddle.layer
+        output of the embedding layer
+    @prefix: str
+        prefix of layers' names, used to share parameters between more than one `cnn` parts.
+    '''
+    def create_conv(context_len, hidden_size, prefix):
+        key = "%s_%d_%d" % (prefix, context_len, hidden_size)
+        conv = paddle.networks.sequence_conv_pool(
+            input=emb,
+            context_len=context_len,
+            hidden_size=hidden_size,
+            # set parameter attr for parameter sharing
+            context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
+            fc_param_attr=ParamAttr(name=key + '_fc.w'),
+            fc_bias_attr=ParamAttr(name=key + '_fc.b'),
+            pool_bias_attr=ParamAttr(name=key + '_pool.b'))
+        return conv
+
+    logger.info('create a sequence_conv_pool which context width is 3')
+    conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
+    logger.info('create a sequence_conv_pool which context width is 4')
+    conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
+    return conv_3, conv_4
+```
+
+CNN 接受 embedding table输出的词向量序列，通过卷积和池化操作捕捉到原始句子的关键信息，
+最终输出一个语义向量（可以认为是句子向量）。
+
+本例的实现中，分别使用了窗口长度为3和4的CNN学到的句子向量按元素求和得到最终的句子向量。
+
+### RNN 结构实现
+
+RNN很适合学习变长序列的信息，使用RNN来学习句子的信息几乎是自然语言处理任务的标配。
+
+```python
+def create_rnn(self, emb, prefix=''):
+    '''
+    A GRU sentence vector learner.
+    '''
+    gru = paddle.layer.gru_memory(input=emb,)
+    sent_vec = paddle.layer.last_seq(gru)
+    return sent_vec
+```
+
+### FC 结构实现
+
+```python
+def create_fc(self, emb, prefix=''):
+    '''
+    A multi-layer fully connected neural networks.
+
+    @emb: paddle.layer
+        output of the embedding layer
+    @prefix: str
+        prefix of layers' names, used to share parameters between more than one `fc` parts.
+    '''
+    _input_layer = paddle.layer.pooling(
+        input=emb, pooling_type=paddle.pooling.Max())
+    fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
+    return fc
+```
+
+在构建FC时需要首先使用`paddle.layer.pooling` 对词向量序列进行最大池化操作，将边长序列转化为一个固定维度向量，
+作为整个句子的语义表达，使用最大池化能够降低句子长度对句向量表达的影响。
+
+### 多层DNN实现
+在 CNN/DNN/FC提取出 semantic vector后，在上层可继续接多层FC来实现深层DNN结构。
+
+```python
+def create_dnn(self, sent_vec, prefix):
+    # if more than three layers exists, a fc layer will be added.
+    if len(self.dnn_dims) > 1:
+        _input_layer = sent_vec
+        for id, dim in enumerate(self.dnn_dims[1:]):
+            name = "%s_fc_%d_%d" % (prefix, id, dim)
+            logger.info("create fc layer [%s] which dimention is %d" %
+                        (name, dim))
+            fc = paddle.layer.fc(
+                input=_input_layer,
+                size=dim,
+                name=name,
+                act=paddle.activation.Tanh(),
+                param_attr=ParamAttr(name='%s.w' % name),
+                bias_attr=ParamAttr(name='%s.b' % name),
+                )
+            _input_layer = fc
+    return _input_layer
+```
+
+### 分类或回归实现
+分类和回归的结构比较相似，因此可以用一个函数创建出来
+
+```python
+def _build_classification_or_regression_model(self, is_classification):
+    '''
+    Build a classification/regression model, and the cost is returned.
+
+    A Classification has 3 inputs:
+      - source sentence
+      - target sentence
+      - classification label
+
+    '''
+    # prepare inputs.
+    assert self.class_num
+
+    source = paddle.layer.data(
+        name='source_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+    target = paddle.layer.data(
+        name='target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    label = paddle.layer.data(
+        name='label_input',
+        type=paddle.data_type.integer_value(self.class_num)
+        if is_classification else paddle.data_type.dense_input)
+
+    prefixs = '_ _'.split(
+    ) if self.share_semantic_generator else 'left right'.split()
+    embed_prefixs = '_ _'.split(
+    ) if self.share_embed else 'left right'.split()
+
+    word_vecs = []
+    for id, input in enumerate([source, target]):
+        x = self.create_embedding(input, prefix=embed_prefixs[id])
+        word_vecs.append(x)
+
+    semantics = []
+    for id, input in enumerate(word_vecs):
+        x = self.model_arch_creater(input, prefix=prefixs[id])
+        semantics.append(x)
+
+    concated_vector = paddle.layer.concat(semantics)
+    prediction = paddle.layer.fc(
+        input=concated_vector,
+        size=self.class_num,
+        act=paddle.activation.Softmax())
+    cost = paddle.layer.classification_cost(
+        input=prediction,
+        label=label) if is_classification else paddle.layer.mse_cost(
+            prediction, label)
+    return cost, prediction, label
+```
+### Pairwise Rank实现
+Pairwise Rank复用上面的DNN结构，同一个source对两个target求相似度打分，
+如果左边的target打分高，预测为1，否则预测为 0。
+
+```python
+def _build_rank_model(self):
+    '''
+    Build a pairwise rank model, and the cost is returned.
+
+    A pairwise rank model has 3 inputs:
+      - source sentence
+      - left_target sentence
+      - right_target sentence
+      - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
+    '''
+    source = paddle.layer.data(
+        name='source_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+    left_target = paddle.layer.data(
+        name='left_target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    right_target = paddle.layer.data(
+        name='right_target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    label = paddle.layer.data(
+        name='label_input', type=paddle.data_type.integer_value(1))
+
+    prefixs = '_ _ _'.split(
+    ) if self.share_semantic_generator else 'source left right'.split()
+    embed_prefixs = '_ _'.split(
+    ) if self.share_embed else 'source target target'.split()
+
+    word_vecs = []
+    for id, input in enumerate([source, left_target, right_target]):
+        x = self.create_embedding(input, prefix=embed_prefixs[id])
+        word_vecs.append(x)
+
+    semantics = []
+    for id, input in enumerate(word_vecs):
+        x = self.model_arch_creater(input, prefix=prefixs[id])
+        semantics.append(x)
+
+    # cossim score of source and left_target
+    left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
+    # cossim score of source and right target
+    right_score = paddle.layer.cos_sim(semantics[0], semantics[2])
+
+    # rank cost
+    cost = paddle.layer.rank_cost(left_score, right_score, label=label)
+    # prediction = left_score - right_score
+    # but this operator is not supported currently.
+    # so AUC will not used.
+    return cost, None, None
+```
+## 数据格式
+在 `./data` 中有简单的示例数据
+
+### 回归的数据格式
+```
+# 3 fields each line:
+#   - source's word ids
+#   - target's word ids
+#   - target
+<ids> \t <ids> \t <float>
+```
+
+比如：
+
+```
+3 6 10 \t 6 8 33 \t 0.7
+6 0 \t 6 9 330 \t 0.03
+```
+### 分类的数据格式
+```
+# 3 fields each line:
+#   - source's word ids
+#   - target's word ids
+#   - target
+<ids> \t <ids> \t <label>
+```
+
+比如：
+
+```
+3 6 10 \t 6 8 33 \t 0
+6 10 \t 8 3 1 \t 1
+```
+
+
+### 排序的数据格式
+```
+# 4 fields each line:
+#   - source's word ids
+#   - target1's word ids
+#   - target2's word ids
+#   - label
+<ids> \t <ids> \t <ids> \t <label>
+```
+
+比如：
+
+```
+7 2 4 \t 2 10 12 \t 9 2 7 10 23 \t 0
+7 2 4 \t 10 12 \t 9 2 21 23 \t 1
+```
+
+## 执行训练
+
+可以直接执行 `python train.py -y 0 --model_arch 0` 使用 `./data/classification` 目录里简单的数据来训练一个分类的FC模型。
+
+其他模型结构也可以通过命令行实现定制，详细命令行参数如下
+
+```
+usage: train.py [-h] [-i TRAIN_DATA_PATH] [-t TEST_DATA_PATH]
+                [-s SOURCE_DIC_PATH] [--target_dic_path TARGET_DIC_PATH]
+                [-b BATCH_SIZE] [-p NUM_PASSES] -y MODEL_TYPE -a MODEL_ARCH
+                [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET]
+                [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS]
+                [--num_workers NUM_WORKERS] [--use_gpu USE_GPU] [-c CLASS_NUM]
+                [--model_output_prefix MODEL_OUTPUT_PREFIX]
+                [-g NUM_BATCHES_TO_LOG] [-e NUM_BATCHES_TO_TEST]
+                [-z NUM_BATCHES_TO_SAVE_MODEL]
+
+PaddlePaddle DSSM example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -i TRAIN_DATA_PATH, --train_data_path TRAIN_DATA_PATH
+                        path of training dataset
+  -t TEST_DATA_PATH, --test_data_path TEST_DATA_PATH
+                        path of testing dataset
+  -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH
+                        path of the source's word dic
+  --target_dic_path TARGET_DIC_PATH
+                        path of the target's word dic, if not set, the
+                        `source_dic_path` will be used
+  -b BATCH_SIZE, --batch_size BATCH_SIZE
+                        size of mini-batch (default:10)
+  -p NUM_PASSES, --num_passes NUM_PASSES
+                        number of passes to run(default:10)
+  -y MODEL_TYPE, --model_type MODEL_TYPE
+                        model type, 0 for classification, 1 for pairwise rank,
+                        2 for regression (default: classification)
+  -a MODEL_ARCH, --model_arch MODEL_ARCH
+                        model architecture, 1 for CNN, 0 for FC, 2 for RNN
+  --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET
+                        whether to share network parameters between source and
+                        target
+  --share_embed SHARE_EMBED
+                        whether to share word embedding between source and
+                        target
+  --dnn_dims DNN_DIMS   dimentions of dnn layers, default is '256,128,64,32',
+                        which means create a 4-layer dnn, demention of each
+                        layer is 256, 128, 64 and 32
+  --num_workers NUM_WORKERS
+                        num worker threads, default 1
+  --use_gpu USE_GPU     whether to use GPU devices (default: False)
+  -c CLASS_NUM, --class_num CLASS_NUM
+                        number of categories for classification task.
+  --model_output_prefix MODEL_OUTPUT_PREFIX
+                        prefix of the path for model to store, (default: ./)
+  -g NUM_BATCHES_TO_LOG, --num_batches_to_log NUM_BATCHES_TO_LOG
+                        number of batches to output train log, (default: 100)
+  -e NUM_BATCHES_TO_TEST, --num_batches_to_test NUM_BATCHES_TO_TEST
+                        number of batches to test, (default: 200)
+  -z NUM_BATCHES_TO_SAVE_MODEL, --num_batches_to_save_model NUM_BATCHES_TO_SAVE_MODEL
+                        number of batches to output model, (default: 400)
+```
+
+重要的参数描述如下
+
+- `train_data_path` 训练数据路径
+- `test_data_path` 测试数据路局，可以不设置
+- `source_dic_path` 源字典字典路径
+- `target_dic_path` 目标字典路径
+- `model_type` 模型的损失函数的类型，分类0，排序1，回归2
+- `model_arch` 模型结构，FC 0， CNN 1, RNN 2
+- `dnn_dims` 模型各层的维度设置，默认为 `256,128,64,32`，即模型有4层，各层维度如上设置
+
+## 用训练好的模型预测
+```
+usage: infer.py [-h] --model_path MODEL_PATH -i DATA_PATH -o
+                PREDICTION_OUTPUT_PATH -y MODEL_TYPE [-s SOURCE_DIC_PATH]
+                [--target_dic_path TARGET_DIC_PATH] -a MODEL_ARCH
+                [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET]
+                [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS]
+                [-c CLASS_NUM]
+
+PaddlePaddle DSSM infer
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --model_path MODEL_PATH
+                        path of model parameters file
+  -i DATA_PATH, --data_path DATA_PATH
+                        path of the dataset to infer
+  -o PREDICTION_OUTPUT_PATH, --prediction_output_path PREDICTION_OUTPUT_PATH
+                        path to output the prediction
+  -y MODEL_TYPE, --model_type MODEL_TYPE
+                        model type, 0 for classification, 1 for pairwise rank,
+                        2 for regression (default: classification)
+  -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH
+                        path of the source's word dic
+  --target_dic_path TARGET_DIC_PATH
+                        path of the target's word dic, if not set, the
+                        `source_dic_path` will be used
+  -a MODEL_ARCH, --model_arch MODEL_ARCH
+                        model architecture, 1 for CNN, 0 for FC, 2 for RNN
+  --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET
+                        whether to share network parameters between source and
+                        target
+  --share_embed SHARE_EMBED
+                        whether to share word embedding between source and
+                        target
+  --dnn_dims DNN_DIMS   dimentions of dnn layers, default is '256,128,64,32',
+                        which means create a 4-layer dnn, demention of each
+                        layer is 256, 128, 64 and 32
+  -c CLASS_NUM, --class_num CLASS_NUM
+                        number of categories for classification task.
+```
+
+部分参数可以参考 `train.py`，重要参数解释如下
+
+- `data_path` 需要预测的数据路径
+- `prediction_output_path` 预测的输出路径
+
+## 参考文献
+
+1. Huang P S, He X, Gao J, et al. Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 2013: 2333-2338.
+2. [Microsoft Learning to Rank Datasets](https://www.microsoft.com/en-us/research/project/mslr/)
+3. [Gao J, He X, Deng L. Deep Learning for Web Search and Natural Language Processing[J]. Microsoft Research Technical Report, 2015.](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/wsdm2015.v3.pdf)

dssm/data/classification/test.txt

+苹果	苹果 6s	0
+汽车 驾驶	驾校 培训	1

dssm/data/classification/train.txt

+苹果 六 袋	苹果 6s	0
+新手 汽车 驾驶	驾校 培训	1

dssm/data/rank/test.txt

+苹果 六 袋	苹果 6s	新手 汽车 驾驶	1
+新手 汽车 驾驶	驾校 培训	苹果 6s	0

dssm/data/rank/train.txt

+苹果 六 袋	苹果 6s	新手 汽车 驾驶	1
+新手 汽车 驾驶	驾校 培训	苹果 6s	1

dssm/data/vocab.txt

+UNK
+苹果
+六
+袋
+6s
+新手
+汽车
+驾驶
+驾校
+培训
\ No newline at end of file

dssm/infer.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import argparse
+import itertools
+
+import reader
+import paddle.v2 as paddle
+from network_conf import DSSM
+from utils import logger, ModelType, ModelArch, load_dic
+
+parser = argparse.ArgumentParser(description="PaddlePaddle DSSM infer")
+parser.add_argument(
+    '--model_path',
+    type=str,
+    required=True,
+    help="path of model parameters file")
+parser.add_argument(
+    '-i',
+    '--data_path',
+    type=str,
+    required=True,
+    help="path of the dataset to infer")
+parser.add_argument(
+    '-o',
+    '--prediction_output_path',
+    type=str,
+    required=True,
+    help="path to output the prediction")
+parser.add_argument(
+    '-y',
+    '--model_type',
+    type=int,
+    required=True,
+    default=ModelType.CLASSIFICATION_MODE,
+    help="model type, %d for classification, %d for pairwise rank, %d for regression (default: classification)"
+    % (ModelType.CLASSIFICATION_MODE, ModelType.RANK_MODE,
+       ModelType.REGRESSION_MODE))
+parser.add_argument(
+    '-s',
+    '--source_dic_path',
+    type=str,
+    required=False,
+    help="path of the source's word dic")
+parser.add_argument(
+    '--target_dic_path',
+    type=str,
+    required=False,
+    help="path of the target's word dic, if not set, the `source_dic_path` will be used"
+)
+parser.add_argument(
+    '-a',
+    '--model_arch',
+    type=int,
+    required=True,
+    default=ModelArch.CNN_MODE,
+    help="model architecture, %d for CNN, %d for FC, %d for RNN" %
+    (ModelArch.CNN_MODE, ModelArch.FC_MODE, ModelArch.RNN_MODE))
+parser.add_argument(
+    '--share_network_between_source_target',
+    type=bool,
+    default=False,
+    help="whether to share network parameters between source and target")
+parser.add_argument(
+    '--share_embed',
+    type=bool,
+    default=False,
+    help="whether to share word embedding between source and target")
+parser.add_argument(
+    '--dnn_dims',
+    type=str,
+    default='256,128,64,32',
+    help="dimentions of dnn layers, default is '256,128,64,32', which means create a 4-layer dnn, demention of each layer is 256, 128, 64 and 32"
+)
+parser.add_argument(
+    '-c',
+    '--class_num',
+    type=int,
+    default=0,
+    help="number of categories for classification task.")
+
+args = parser.parse_args()
+args.model_type = ModelType(args.model_type)
+args.model_arch = ModelArch(args.model_arch)
+if args.model_type.is_classification():
+    assert args.class_num > 1, "--class_num should be set in classification task."
+
+layer_dims = map(int, args.dnn_dims.split(','))
+args.target_dic_path = args.source_dic_path if not args.target_dic_path else args.target_dic_path
+
+paddle.init(use_gpu=False, trainer_count=1)
+
+
+class Inferer(object):
+    def __init__(self, param_path):
+        logger.info("create DSSM model")
+
+        cost, prediction, label = DSSM(
+            dnn_dims=layer_dims,
+            vocab_sizes=[
+                len(load_dic(path))
+                for path in [args.source_dic_path, args.target_dic_path]
+            ],
+            model_type=args.model_type,
+            model_arch=args.model_arch,
+            share_semantic_generator=args.share_network_between_source_target,
+            class_num=args.class_num,
+            share_embed=args.share_embed)()
+
+        # load parameter
+        logger.info("load model parameters from %s" % param_path)
+        self.parameters = paddle.parameters.Parameters.from_tar(
+            open(param_path, 'r'))
+        self.inferer = paddle.inference.Inference(
+            output_layer=prediction, parameters=self.parameters)
+
+    def infer(self, data_path):
+        logger.info("infer data...")
+        dataset = reader.Dataset(
+            train_path=data_path,
+            test_path=None,
+            source_dic_path=args.source_dic_path,
+            target_dic_path=args.target_dic_path,
+            model_type=args.model_type, )
+        infer_reader = paddle.batch(dataset.infer, batch_size=1000)
+        logger.warning('write predictions to %s' % args.prediction_output_path)
+
+        output_f = open(args.prediction_output_path, 'w')
+
+        for id, batch in enumerate(infer_reader()):
+            res = self.inferer.infer(input=batch)
+            predictions = [' '.join(map(str, x)) for x in res]
+            assert len(batch) == len(
+                predictions), "predict error, %d inputs, but %d predictions" % (
+                    len(batch), len(predictions))
+            output_f.write('\n'.join(map(str, predictions)) + '\n')
+
+
+if __name__ == '__main__':
+    inferer = Inferer(args.model_path)
+    inferer.infer(args.data_path)

dssm/network_conf.py

+from paddle import v2 as paddle
+from paddle.v2.attr import ParamAttr
+from utils import TaskType, logger, ModelType, ModelArch
+
+
+class DSSM(object):
+    def __init__(self,
+                 dnn_dims=[],
+                 vocab_sizes=[],
+                 model_type=ModelType.create_classification(),
+                 model_arch=ModelArch.create_cnn(),
+                 share_semantic_generator=False,
+                 class_num=None,
+                 share_embed=False,
+                 is_infer=False):
+        '''
+        @dnn_dims: list of int
+            dimentions of each layer in semantic vector generator.
+        @vocab_sizes: 2-d tuple
+            size of both left and right items.
+        @model_type: int
+            type of task, should be 'rank: 0', 'regression: 1' or 'classification: 2'
+        @model_arch: int
+            model architecture
+        @share_semantic_generator: bool
+            whether to share the semantic vector generator for both left and right.
+        @share_embed: bool
+            whether to share the embeddings between left and right.
+        @class_num: int
+            number of categories.
+        '''
+        assert len(
+            vocab_sizes
+        ) == 2, "vocab_sizes specify the sizes left and right inputs, and dim should be 2."
+        assert len(dnn_dims) > 1, "more than two layers is needed."
+
+        self.dnn_dims = dnn_dims
+        self.vocab_sizes = vocab_sizes
+        self.share_semantic_generator = share_semantic_generator
+        self.share_embed = share_embed
+        self.model_type = ModelType(model_type)
+        self.model_arch = ModelArch(model_arch)
+        self.class_num = class_num
+        self.is_infer = is_infer
+        logger.warning("build DSSM model with config of %s, %s" %
+                       (self.model_type, self.model_arch))
+        logger.info("vocabulary sizes: %s" % str(self.vocab_sizes))
+
+        # bind model architecture
+        _model_arch = {
+            'cnn': self.create_cnn,
+            'fc': self.create_fc,
+            'rnn': self.create_rnn,
+        }
+
+        def _model_arch_creater(emb, prefix=''):
+            sent_vec = _model_arch.get(str(model_arch))(emb, prefix)
+            dnn = self.create_dnn(sent_vec, prefix)
+            return dnn
+
+        self.model_arch_creater = _model_arch_creater
+
+        # build model type
+        _model_type = {
+            'classification': self._build_classification_model,
+            'rank': self._build_rank_model,
+            'regression': self._build_regression_model,
+        }
+        print 'model type: ', str(self.model_type)
+        self.model_type_creater = _model_type[str(self.model_type)]
+
+    def __call__(self):
+        return self.model_type_creater()
+
+    def create_embedding(self, input, prefix=''):
+        '''
+        Create an embedding table whose name has a `prefix`.
+        '''
+        logger.info("create embedding table [%s] which dimention is %d" %
+                    (prefix, self.dnn_dims[0]))
+        emb = paddle.layer.embedding(
+            input=input,
+            size=self.dnn_dims[0],
+            param_attr=ParamAttr(name='%s_emb.w' % prefix))
+        return emb
+
+    def create_fc(self, emb, prefix=''):
+        '''
+        A multi-layer fully connected neural networks.
+
+        @emb: paddle.layer
+            output of the embedding layer
+        @prefix: str
+            prefix of layers' names, used to share parameters between more than one `fc` parts.
+        '''
+        _input_layer = paddle.layer.pooling(
+            input=emb, pooling_type=paddle.pooling.Max())
+        fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
+        return fc
+
+    def create_rnn(self, emb, prefix=''):
+        '''
+        A GRU sentence vector learner.
+        '''
+        gru = paddle.layer.gru_memory(
+            input=emb, )
+        sent_vec = paddle.layer.last_seq(gru)
+        return sent_vec
+
+    def create_cnn(self, emb, prefix=''):
+        '''
+        A multi-layer CNN.
+
+        @emb: paddle.layer
+            output of the embedding layer
+        @prefix: str
+            prefix of layers' names, used to share parameters between more than one `cnn` parts.
+        '''
+
+        def create_conv(context_len, hidden_size, prefix):
+            key = "%s_%d_%d" % (prefix, context_len, hidden_size)
+            conv = paddle.networks.sequence_conv_pool(
+                input=emb,
+                context_len=context_len,
+                hidden_size=hidden_size,
+                # set parameter attr for parameter sharing
+                context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
+                fc_param_attr=ParamAttr(name=key + '_fc.w'),
+                fc_bias_attr=ParamAttr(name=key + '_fc.b'),
+                pool_bias_attr=ParamAttr(name=key + '_pool.b'))
+            return conv
+
+        logger.info('create a sequence_conv_pool which context width is 3')
+        conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
+        logger.info('create a sequence_conv_pool which context width is 4')
+        conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
+
+        return conv_3, conv_4
+
+    def create_dnn(self, sent_vec, prefix):
+        # if more than three layers, than a fc layer will be added.
+        if len(self.dnn_dims) > 1:
+            _input_layer = sent_vec
+            for id, dim in enumerate(self.dnn_dims[1:]):
+                name = "%s_fc_%d_%d" % (prefix, id, dim)
+                logger.info("create fc layer [%s] which dimention is %d" %
+                            (name, dim))
+                fc = paddle.layer.fc(
+                    name=name,
+                    input=_input_layer,
+                    size=dim,
+                    act=paddle.activation.Tanh(),
+                    param_attr=ParamAttr(name='%s.w' % name),
+                    bias_attr=ParamAttr(name='%s.b' % name))
+                _input_layer = fc
+        return _input_layer
+
+    def _build_classification_model(self):
+        logger.info("build classification model")
+        assert self.model_type.is_classification()
+        return self._build_classification_or_regression_model(
+            is_classification=True)
+
+    def _build_regression_model(self):
+        logger.info("build regression model")
+        assert self.model_type.is_regression()
+        return self._build_classification_or_regression_model(
+            is_classification=False)
+
+    def _build_rank_model(self):
+        '''
+        Build a pairwise rank model, and the cost is returned.
+
+        A pairwise rank model has 3 inputs:
+          - source sentence
+          - left_target sentence
+          - right_target sentence
+          - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
+        '''
+        logger.info("build rank model")
+        assert self.model_type.is_rank()
+        source = paddle.layer.data(
+            name='source_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+        left_target = paddle.layer.data(
+            name='left_target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        right_target = paddle.layer.data(
+            name='right_target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        if not self.is_infer:
+            label = paddle.layer.data(
+                name='label_input', type=paddle.data_type.integer_value(1))
+
+        prefixs = '_ _ _'.split(
+        ) if self.share_semantic_generator else 'source left right'.split()
+        embed_prefixs = '_ _'.split(
+        ) if self.share_embed else 'source target target'.split()
+
+        word_vecs = []
+        for id, input in enumerate([source, left_target, right_target]):
+            x = self.create_embedding(input, prefix=embed_prefixs[id])
+            word_vecs.append(x)
+
+        semantics = []
+        for id, input in enumerate(word_vecs):
+            x = self.model_arch_creater(input, prefix=prefixs[id])
+            semantics.append(x)
+
+        # cossim score of source and left_target
+        left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
+        # cossim score of source and right target
+        right_score = paddle.layer.cos_sim(semantics[0], semantics[2])
+
+        if not self.is_infer:
+            # rank cost
+            cost = paddle.layer.rank_cost(left_score, right_score, label=label)
+            # prediction = left_score - right_score
+            # but this operator is not supported currently.
+            # so AUC will not used.
+            return cost, None, label
+        return None, [left_score, right_score], label
+
+    def _build_classification_or_regression_model(self, is_classification):
+        '''
+        Build a classification/regression model, and the cost is returned.
+
+        A Classification has 3 inputs:
+          - source sentence
+          - target sentence
+          - classification label
+
+        '''
+        if is_classification:
+            # prepare inputs.
+            assert self.class_num
+
+        source = paddle.layer.data(
+            name='source_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+        target = paddle.layer.data(
+            name='target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        label = paddle.layer.data(
+            name='label_input',
+            type=paddle.data_type.integer_value(self.class_num)
+            if is_classification else paddle.data_type.dense_vector(1))
+
+        prefixs = '_ _'.split(
+        ) if self.share_semantic_generator else 'left right'.split()
+        embed_prefixs = '_ _'.split(
+        ) if self.share_embed else 'left right'.split()
+
+        word_vecs = []
+        for id, input in enumerate([source, target]):
+            x = self.create_embedding(input, prefix=embed_prefixs[id])
+            word_vecs.append(x)
+
+        semantics = []
+        for id, input in enumerate(word_vecs):
+            x = self.model_arch_creater(input, prefix=prefixs[id])
+            semantics.append(x)
+
+        if is_classification:
+            concated_vector = paddle.layer.concat(semantics)
+            prediction = paddle.layer.fc(
+                input=concated_vector,
+                size=self.class_num,
+                act=paddle.activation.Softmax())
+            cost = paddle.layer.classification_cost(
+                input=prediction, label=label)
+        else:
+            prediction = paddle.layer.cos_sim(*semantics)
+            cost = paddle.layer.mse_cost(prediction, label)
+
+        if not self.is_infer:
+            return cost, prediction, label
+        return None, prediction, label

dssm/reader.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+from utils import UNK, ModelType, TaskType, load_dic, sent2ids, logger, ModelType
+
+
+class Dataset(object):
+    def __init__(self, train_path, test_path, source_dic_path, target_dic_path,
+                 model_type):
+        self.train_path = train_path
+        self.test_path = test_path
+        self.source_dic_path = source_dic_path
+        self.target_dic_path = target_dic_path
+        self.model_type = ModelType(model_type)
+
+        self.source_dic = load_dic(self.source_dic_path)
+        self.target_dic = load_dic(self.target_dic_path)
+
+        _record_reader = {
+            ModelType.CLASSIFICATION_MODE: self._read_classification_record,
+            ModelType.REGRESSION_MODE: self._read_regression_record,
+            ModelType.RANK_MODE: self._read_rank_record,
+        }
+
+        assert isinstance(model_type, ModelType)
+        self.record_reader = _record_reader[model_type.mode]
+        self.is_infer = False
+
+    def train(self):
+        '''
+        Load trainset.
+        '''
+        logger.info("[reader] load trainset from %s" % self.train_path)
+        with open(self.train_path) as f:
+            for line_id, line in enumerate(f):
+                yield self.record_reader(line)
+
+    def test(self):
+        '''
+        Load testset.
+        '''
+        # logger.info("[reader] load testset from %s" % self.test_path)
+        with open(self.test_path) as f:
+            for line_id, line in enumerate(f):
+                yield self.record_reader(line)
+
+    def infer(self):
+        self.is_infer = True
+        with open(self.train_path) as f:
+            for line in f:
+                yield self.record_reader(line)
+
+    def _read_classification_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <target words> [TAB] <label>
+
+        @line: str
+            a string line which represent a record.
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 3, "wrong format for classification\n" + \
+            "the format shoud be " +\
+            "<source words> [TAB] <target words> [TAB] <label>'"
+        source = sent2ids(fs[0], self.source_dic)
+        target = sent2ids(fs[1], self.target_dic)
+        if not self.is_infer:
+            label = int(fs[2])
+            return (source, target, label, )
+        return source, target
+
+    def _read_regression_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <target words> [TAB] <label>
+
+        @line: str
+            a string line which represent a record.
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 3, "wrong format for regression\n" + \
+            "the format shoud be " +\
+            "<source words> [TAB] <target words> [TAB] <label>'"
+        source = sent2ids(fs[0], self.source_dic)
+        target = sent2ids(fs[1], self.target_dic)
+        if not self.is_infer:
+            label = float(fs[2])
+            return (source, target, [label], )
+        return source, target
+
+    def _read_rank_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <left_target words> [TAB] <right_target words> [TAB] <label>
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 4, "wrong format for rank\n" + \
+            "the format should be " +\
+            "<source words> [TAB] <left_target words> [TAB] <right_target words> [TAB] <label>"
+
+        source = sent2ids(fs[0], self.source_dic)
+        left_target = sent2ids(fs[1], self.target_dic)
+        right_target = sent2ids(fs[2], self.target_dic)
+        if not self.is_infer:
+            label = int(fs[3])
+            return (source, left_target, right_target, label)
+        return source, left_target, right_target
+
+
+if __name__ == '__main__':
+    path = './data/classification/train.txt'
+    test_path = './data/classification/test.txt'
+    source_dic = './data/vocab.txt'
+    dataset = Dataset(path, test_path, source_dic, source_dic,
+                      ModelType.CLASSIFICATION)
+
+    for rcd in dataset.train():
+        print rcd

dssm/train.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import argparse
+
+import paddle.v2 as paddle
+from network_conf import DSSM
+import reader
+from utils import TaskType, load_dic, logger, ModelType, ModelArch, display_args
+
+parser = argparse.ArgumentParser(description="PaddlePaddle DSSM example")
+
+parser.add_argument(
+    '-i',
+    '--train_data_path',
+    type=str,
+    required=False,
+    help="path of training dataset")
+parser.add_argument(
+    '-t',
+    '--test_data_path',
+    type=str,
+    required=False,
+    help="path of testing dataset")
+parser.add_argument(
+    '-s',
+    '--source_dic_path',
+    type=str,
+    required=False,
+    help="path of the source's word dic")
+parser.add_argument(
+    '--target_dic_path',
+    type=str,
+    required=False,
+    help="path of the target's word dic, if not set, the `source_dic_path` will be used"
+)
+parser.add_argument(
+    '-b',
+    '--batch_size',
+    type=int,
+    default=10,
+    help="size of mini-batch (default:10)")
+parser.add_argument(
+    '-p',
+    '--num_passes',
+    type=int,
+    default=10,
+    help="number of passes to run(default:10)")
+parser.add_argument(
+    '-y',
+    '--model_type',
+    type=int,
+    required=True,
+    default=ModelType.CLASSIFICATION_MODE,
+    help="model type, %d for classification, %d for pairwise rank, %d for regression (default: classification)"
+    % (ModelType.CLASSIFICATION_MODE, ModelType.RANK_MODE,
+       ModelType.REGRESSION_MODE))
+parser.add_argument(
+    '-a',
+    '--model_arch',
+    type=int,
+    required=True,
+    default=ModelArch.CNN_MODE,
+    help="model architecture, %d for CNN, %d for FC, %d for RNN" %
+    (ModelArch.CNN_MODE, ModelArch.FC_MODE, ModelArch.RNN_MODE))
+parser.add_argument(
+    '--share_network_between_source_target',
+    type=bool,
+    default=False,
+    help="whether to share network parameters between source and target")
+parser.add_argument(
+    '--share_embed',
+    type=bool,
+    default=False,
+    help="whether to share word embedding between source and target")
+parser.add_argument(
+    '--dnn_dims',
+    type=str,
+    default='256,128,64,32',
+    help="dimentions of dnn layers, default is '256,128,64,32', which means create a 4-layer dnn, demention of each layer is 256, 128, 64 and 32"
+)
+parser.add_argument(
+    '--num_workers', type=int, default=1, help="num worker threads, default 1")
+parser.add_argument(
+    '--use_gpu',
+    type=bool,
+    default=False,
+    help="whether to use GPU devices (default: False)")
+parser.add_argument(
+    '-c',
+    '--class_num',
+    type=int,
+    default=0,
+    help="number of categories for classification task.")
+parser.add_argument(
+    '--model_output_prefix',
+    type=str,
+    default="./",
+    help="prefix of the path for model to store, (default: ./)")
+parser.add_argument(
+    '-g',
+    '--num_batches_to_log',
+    type=int,
+    default=100,
+    help="number of batches to output train log, (default: 100)")
+parser.add_argument(
+    '-e',
+    '--num_batches_to_test',
+    type=int,
+    default=200,
+    help="number of batches to test, (default: 200)")
+parser.add_argument(
+    '-z',
+    '--num_batches_to_save_model',
+    type=int,
+    default=400,
+    help="number of batches to output model, (default: 400)")
+
+# arguments check.
+args = parser.parse_args()
+args.model_type = ModelType(args.model_type)
+args.model_arch = ModelArch(args.model_arch)
+if args.model_type.is_classification():
+    assert args.class_num > 1, "--class_num should be set in classification task."
+
+layer_dims = [int(i) for i in args.dnn_dims.split(',')]
+args.target_dic_path = args.source_dic_path if not args.target_dic_path else args.target_dic_path
+
+
+def train(train_data_path=None,
+          test_data_path=None,
+          source_dic_path=None,
+          target_dic_path=None,
+          model_type=ModelType.create_classification(),
+          model_arch=ModelArch.create_cnn(),
+          batch_size=10,
+          num_passes=10,
+          share_semantic_generator=False,
+          share_embed=False,
+          class_num=None,
+          num_workers=1,
+          use_gpu=False):
+    '''
+    Train the DSSM.
+    '''
+    default_train_path = './data/rank/train.txt'
+    default_test_path = './data/rank/test.txt'
+    default_dic_path = './data/vocab.txt'
+    if not model_type.is_rank():
+        default_train_path = './data/classification/train.txt'
+        default_test_path = './data/classification/test.txt'
+
+    use_default_data = not train_data_path
+
+    if use_default_data:
+        train_data_path = default_train_path
+        test_data_path = default_test_path
+        source_dic_path = default_dic_path
+        target_dic_path = default_dic_path
+
+    dataset = reader.Dataset(
+        train_path=train_data_path,
+        test_path=test_data_path,
+        source_dic_path=source_dic_path,
+        target_dic_path=target_dic_path,
+        model_type=model_type, )
+
+    train_reader = paddle.batch(
+        paddle.reader.shuffle(dataset.train, buf_size=1000),
+        batch_size=batch_size)
+
+    test_reader = paddle.batch(
+        paddle.reader.shuffle(dataset.test, buf_size=1000),
+        batch_size=batch_size)
+
+    paddle.init(use_gpu=use_gpu, trainer_count=num_workers)
+
+    cost, prediction, label = DSSM(
+        dnn_dims=layer_dims,
+        vocab_sizes=[
+            len(load_dic(path)) for path in [source_dic_path, target_dic_path]
+        ],
+        model_type=model_type,
+        model_arch=model_arch,
+        share_semantic_generator=share_semantic_generator,
+        class_num=class_num,
+        share_embed=share_embed)()
+
+    parameters = paddle.parameters.create(cost)
+
+    adam_optimizer = paddle.optimizer.Adam(
+        learning_rate=1e-3,
+        regularization=paddle.optimizer.L2Regularization(rate=1e-3),
+        model_average=paddle.optimizer.ModelAverage(average_window=0.5))
+
+    trainer = paddle.trainer.SGD(
+        cost=cost,
+        extra_layers=paddle.evaluator.auc(input=prediction, label=label)
+        if not model_type.is_rank() else None,
+        parameters=parameters,
+        update_equation=adam_optimizer)
+
+    feeding = {}
+    if model_type.is_classification() or model_type.is_regression():
+        feeding = {'source_input': 0, 'target_input': 1, 'label_input': 2}
+    else:
+        feeding = {
+            'source_input': 0,
+            'left_target_input': 1,
+            'right_target_input': 2,
+            'label_input': 3
+        }
+
+    def _event_handler(event):
+        '''
+        Define batch handler
+        '''
+        if isinstance(event, paddle.event.EndIteration):
+            # output train log
+            if event.batch_id % args.num_batches_to_log == 0:
+                logger.info("Pass %d, Batch %d, Cost %f, %s" % (
+                    event.pass_id, event.batch_id, event.cost, event.metrics))
+
+            # test model
+            if event.batch_id > 0 and event.batch_id % args.num_batches_to_test == 0:
+                if test_reader is not None:
+                    if model_type.is_classification():
+                        result = trainer.test(
+                            reader=test_reader, feeding=feeding)
+                        logger.info("Test at Pass %d, %s" % (event.pass_id,
+                                                             result.metrics))
+                    else:
+                        result = None
+            # save model
+            if event.batch_id > 0 and event.batch_id % args.num_batches_to_save_model == 0:
+                model_desc = "{type}_{arch}".format(
+                    type=str(args.model_type), arch=str(args.model_arch))
+                with open("%sdssm_%s_pass_%05d.tar" %
+                          (args.model_output_prefix, model_desc,
+                           event.pass_id), "w") as f:
+                    parameters.to_tar(f)
+
+    trainer.train(
+        reader=train_reader,
+        event_handler=_event_handler,
+        feeding=feeding,
+        num_passes=num_passes)
+
+    logger.info("Training has finished.")
+
+
+if __name__ == '__main__':
+    display_args(args)
+    train(
+        train_data_path=args.train_data_path,
+        test_data_path=args.test_data_path,
+        source_dic_path=args.source_dic_path,
+        target_dic_path=args.target_dic_path,
+        model_type=ModelType(args.model_type),
+        model_arch=ModelArch(args.model_arch),
+        batch_size=args.batch_size,
+        num_passes=args.num_passes,
+        share_semantic_generator=args.share_network_between_source_target,
+        share_embed=args.share_embed,
+        class_num=args.class_num,
+        num_workers=args.num_workers,
+        use_gpu=args.use_gpu)

dssm/utils.py

+import logging
+import paddle
+
+UNK = 0
+
+logger = logging.getLogger("paddle")
+logger.setLevel(logging.INFO)
+
+
+def mode_attr_name(mode):
+    return mode.upper() + '_MODE'
+
+
+def create_attrs(cls):
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, mode_attr_name(mode), id)
+
+
+def make_check_method(cls):
+    '''
+    create methods for classes.
+    '''
+
+    def method(mode):
+        def _method(self):
+            return self.mode == getattr(cls, mode_attr_name(mode))
+
+        return _method
+
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, 'is_' + mode, method(mode))
+
+
+def make_create_method(cls):
+    def method(mode):
+        @staticmethod
+        def _method():
+            key = getattr(cls, mode_attr_name(mode))
+            return cls(key)
+
+        return _method
+
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, 'create_' + mode, method(mode))
+
+
+def make_str_method(cls, type_name='unk'):
+    def _str_(self):
+        for mode in cls.modes:
+            if self.mode == getattr(cls, mode_attr_name(mode)):
+                return mode
+
+    def _hash_(self):
+        return self.mode
+
+    setattr(cls, '__str__', _str_)
+    setattr(cls, '__repr__', _str_)
+    setattr(cls, '__hash__', _hash_)
+    cls.__name__ = type_name
+
+
+def _init_(self, mode, cls):
+    if isinstance(mode, int):
+        self.mode = mode
+    elif isinstance(mode, cls):
+        self.mode = mode.mode
+    else:
+        raise Exception("wrong mode type, get type: %s, value: %s" %
+                        (type(mode), mode))
+
+
+def build_mode_class(cls):
+    create_attrs(cls)
+    make_str_method(cls)
+    make_check_method(cls)
+    make_create_method(cls)
+
+
+class TaskType(object):
+    modes = 'train test infer'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, TaskType)
+
+
+class ModelType:
+    modes = 'classification rank regression'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, ModelType)
+
+
+class ModelArch:
+    modes = 'fc cnn rnn'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, ModelArch)
+
+
+build_mode_class(TaskType)
+build_mode_class(ModelType)
+build_mode_class(ModelArch)
+
+
+def sent2ids(sent, vocab):
+    '''
+    transform a sentence to a list of ids.
+
+    @sent: str
+        a sentence.
+    @vocab: dict
+        a word dic
+    '''
+    return [vocab.get(w, UNK) for w in sent.split()]
+
+
+def load_dic(path):
+    '''
+    word dic format:
+      each line is a word
+    '''
+    dic = {}
+    with open(path) as f:
+        for id, line in enumerate(f):
+            w = line.strip()
+            dic[w] = id
+    return dic
+
+
+def display_args(args):
+    logger.info("arguments passed by command line:")
+    for k, v in sorted(v for v in vars(args).items()):
+        logger.info("{}:\t{}".format(k, v))
+
+
+if __name__ == '__main__':
+    t = TaskType(1)
+    t = TaskType.create_train()
+    print t
+    print 'is', t.is_train()

generate_sequence_by_rnn_lm/config.py

@@ -26,7 +26,7 @@ num_passes = 20  # how many passes to train the model
 log_period = 50
 save_period_by_batches = 50
 
-use_gpu = True  # to use gpu or not
+use_gpu = False  # to use gpu or not
 trainer_count = 1  # number of trainer
 
 ##################  for model configuration  ##################

generate_sequence_by_rnn_lm/train.py

@@ -35,8 +35,7 @@ def train(topology,
         os.mkdir(model_save_dir)
 
     # initialize PaddlePaddle
-    paddle.init(
-        use_gpu=conf.use_gpu, gpu_id=3, trainer_count=conf.trainer_count)
+    paddle.init(use_gpu=conf.use_gpu, trainer_count=conf.trainer_count)
 
     # create optimizer
     adam_optimizer = paddle.optimizer.Adam(

image_classification/caffe2paddle/caffe2paddle.py

@@ -72,6 +72,10 @@ class ModelConverter(object):
                 file_name = "_%s.w%s" % (name, str(i))
             param_conf = ParameterConfig()
             param_conf.name = file_name
+            dims = list(data.shape)
+            if len(dims) == 1:
+                dims.insert(1, 1)
+                param_conf.dims.extend(dims)
             param_conf.size = reduce(lambda a, b: a * b, data.shape)
             self.params[file_name] = (param_conf, data.flatten())
 

ltr/README.md

@@ -4,14 +4,14 @@
 
 RankNet模型在命令行输入：
 
-```python
-python ranknet.py
+```bash
+bash ./run_ranknet.sh
 ```
 
 LambdaRank模型在命令行输入：
 
-```python
-python lambda_rank.py
+```bash
+bash ./run_lambdarank.sh
 ```
 
 用户只需要使用以上命令就完成排序模型的训练和预测，程序会自动下载内置数据集，无需手动下载。
@@ -54,7 +54,7 @@ python lambda_rank.py
 
 例如调用接口
 
-```python
+```bash
 pairwise_train_dataset = functools.partial(paddle.dataset.mq2007.train, format="pairwise")
 for label, left_doc, right_doc in pairwise_train_dataset():
     ...
@@ -104,7 +104,7 @@ $$\lambda _{i,j}=\frac{\partial C}{\partial s_{i}} = \frac{1}{2}(1-S_{i,j})-\fra
 
 由于Pairwise中的网络结构是左右对称，可定义一半网络结构，另一半共享网络参数。在PaddlePaddle中允许网络结构中共享连接，具有相同名字的参数将会共享参数。使用PaddlePaddle实现RankNet排序模型，定义网络结构的示例代码如下：
 
-```python
+```bash
 import paddle.v2 as paddle
 
 def half_ranknet(name_prefix, input_dim):
@@ -150,7 +150,7 @@ def ranknet(input_dim):
 RankNet的训练只需要运行命令：
 
 ```python
-python ranknet.py
+run ./run_ranknet.sh
 ```
 将会自动下载数据，训练RankNet模型，并将每个轮次的模型参数存储下来。
 
@@ -277,7 +277,7 @@ def lambda_rank(input_dim):
 训练LambdaRank模型只需要运行命令：
 
 ```python
-python lambda_rank.py
+bash ./run_lambdarank.sh
 ```
 
 脚本会自动下载数据，训练LambdaRank模型，并将每个轮次的模型存储下来。
@@ -303,7 +303,7 @@ query_id : 2, relevance_score:2, feature_vector 0:0.1, 1:0.4, 2:0.1  #doc1
 
 需要转换为Listwise格式，例如
 
-<query_id><relevance_score> <feature_vector>
+`<query_id><relevance_score> <feature_vector>`
 
 ```tex
 1    1    0.1,0.2,0.4

ltr/index.html

@@ -46,14 +46,14 @@
 
 RankNet模型在命令行输入：
 
-```python
-python ranknet.py
+```bash
+bash ./run_ranknet.sh
 ```
 
 LambdaRank模型在命令行输入：
 
-```python
-python lambda_rank.py
+```bash
+bash ./run_lambdarank.sh
 ```
 
 用户只需要使用以上命令就完成排序模型的训练和预测，程序会自动下载内置数据集，无需手动下载。
@@ -96,7 +96,7 @@ python lambda_rank.py
 
 例如调用接口
 
-```python
+```bash
 pairwise_train_dataset = functools.partial(paddle.dataset.mq2007.train, format="pairwise")
 for label, left_doc, right_doc in pairwise_train_dataset():
     ...
@@ -146,7 +146,7 @@ $$\lambda _{i,j}=\frac{\partial C}{\partial s_{i}} = \frac{1}{2}(1-S_{i,j})-\fra
 
 由于Pairwise中的网络结构是左右对称，可定义一半网络结构，另一半共享网络参数。在PaddlePaddle中允许网络结构中共享连接，具有相同名字的参数将会共享参数。使用PaddlePaddle实现RankNet排序模型，定义网络结构的示例代码如下：
 
-```python
+```bash
 import paddle.v2 as paddle
 
 def half_ranknet(name_prefix, input_dim):
@@ -192,7 +192,7 @@ def ranknet(input_dim):
 RankNet的训练只需要运行命令：
 
 ```python
-python ranknet.py
+run ./run_ranknet.sh
 ```
 将会自动下载数据，训练RankNet模型，并将每个轮次的模型参数存储下来。
 
@@ -319,7 +319,7 @@ def lambda_rank(input_dim):
 训练LambdaRank模型只需要运行命令：
 
 ```python
-python lambda_rank.py
+bash ./run_lambdarank.sh
 ```
 
 脚本会自动下载数据，训练LambdaRank模型，并将每个轮次的模型存储下来。
@@ -345,7 +345,7 @@ query_id : 2, relevance_score:2, feature_vector 0:0.1, 1:0.4, 2:0.1  #doc1
 
 需要转换为Listwise格式，例如
 
-<query_id><relevance_score> <feature_vector>
+`<query_id><relevance_score> <feature_vector>`
 
 ```tex
 1    1    0.1,0.2,0.4

ltr/lambda_rank.py

@@ -3,6 +3,7 @@ import gzip
 import paddle.v2 as paddle
 import numpy as np
 import functools
+import argparse
 
 
 def lambda_rank(input_dim):
@@ -117,6 +118,15 @@ def lambda_rank_infer(pass_id):
 
 
 if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='LambdaRank demo')
+    parser.add_argument("--run_type", type=str, help="run type is train|infer")
+    parser.add_argument(
+        "--num_passes",
+        type=int,
+        help="num of passes in train| infer pass number of model")
+    args = parser.parse_args()
     paddle.init(use_gpu=False, trainer_count=1)
-    train_lambda_rank(2)
-    lambda_rank_infer(pass_id=1)
+    if args.run_type == "train":
+        train_lambda_rank(args.num_passes)
+    elif args.run_type == "infer":
+        lambda_rank_infer(pass_id=args.num_passes - 1)

ltr/ranknet.py

@@ -5,6 +5,7 @@ import functools
 import paddle.v2 as paddle
 import numpy as np
 from metrics import ndcg
+import argparse
 
 # ranknet is the classic pairwise learning to rank algorithm
 # http://icml.cc/2015/wp-content/uploads/2015/06/icml_ranking.pdf
@@ -102,9 +103,9 @@ def ranknet_infer(pass_id):
     feature_dim = 46
 
     # we just need half_ranknet to predict a rank score, which can be used in sort documents
-    output = half_ranknet("left", feature_dim)
+    output = half_ranknet("infer", feature_dim)
     parameters = paddle.parameters.Parameters.from_tar(
-        gzip.open("ranknet_params_%d.tar.gz" % (pass_id - 1)))
+        gzip.open("ranknet_params_%d.tar.gz" % (pass_id)))
 
     # load data of same query and relevance documents, need ranknet to rank these candidates
     infer_query_id = []
@@ -118,18 +119,27 @@ def ranknet_infer(pass_id):
 
     for query_id, relevance_score, feature_vector in plain_txt_test():
         infer_query_id.append(query_id)
-        infer_data.append(feature_vector)
+        infer_data.append([feature_vector])
 
     # predict score of infer_data document. Re-sort the document base on predict score
     # in descending order. then we build the ranking documents
     scores = paddle.infer(
         output_layer=output, parameters=parameters, input=infer_data)
+    print scores
     for query_id, score in zip(infer_query_id, scores):
         print "query_id : ", query_id, " ranknet rank document order : ", score
 
 
 if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Ranknet demo')
+    parser.add_argument("--run_type", type=str, help="run type is train|infer")
+    parser.add_argument(
+        "--num_passes",
+        type=int,
+        help="num of passes in train| infer pass number of model")
+    args = parser.parse_args()
     paddle.init(use_gpu=False, trainer_count=4)
-    pass_num = 2
-    train_ranknet(pass_num)
-    ranknet_infer(pass_id=pass_num - 1)
+    if args.run_type == "train":
+        train_ranknet(args.num_passes)
+    elif args.run_type == "infer":
+        ranknet_infer(pass_id=args.pass_num - 1)

ltr/run_lambdarank.sh

+#!/bin/sh
+
+python lambda_rank.py \
+       --run_type="train" \
+       --num_passes=10 \
+       2>&1 | tee lambdarank_train.log
+
+python lambda_rank.py \
+       --run_type="infer" \
+       --num_passes=10 \
+       2>&1 | tee lambdarank_infer.log

ltr/run_ranknet.sh

+#!/bin/sh
+
+python ranknet.py \
+       --run_type="train" \
+       --num_passes=10 \
+       2>&1 | tee rankenet_train.log
+
+python ranknet.py \
+       --run_type="infer" \
+       --num_passes=10 \
+       2>&1 | tee ranknet_infer.log

nce_cost/train.py

@@ -43,7 +43,10 @@ def train(model_save_dir):
                 parameters.to_tar(f)
 
     trainer.train(
-        paddle.batch(paddle.dataset.imikolov.train(word_dict, 5), 64),
+        paddle.batch(
+            paddle.reader.shuffle(
+                lambda: paddle.dataset.imikolov.train(word_dict, 5)(),
+                buf_size=1000), 64),
         num_passes=1000,
         event_handler=event_handler)
 

nmt_without_attention/train.py

@@ -63,4 +63,4 @@ def train(save_dir_path, source_dict_dim, target_dict_dim):
 
 
 if __name__ == '__main__':
-    train(save_dir_path="models", source_dict_dim=3000, target_dict_dim=3000)
+    train(save_dir_path="models", source_dict_dim=30000, target_dict_dim=30000)

sequence_tagging_for_ner/README.md

@@ -23,10 +23,10 @@
 
 序列标注可以分为Sequence Classification、Segment Classification和Temporal Classification三类[[1](#参考文献)]，本例只考虑Segment Classification，即对输入序列中的每个元素在输出序列中给出对应的标签。对于NER任务，由于需要标识边界，一般采用[BIO标注方法](http://book.paddlepaddle.org/07.label_semantic_roles/)定义的标签集，如下是一个NER的标注结果示例：
 
-<div  align="center">
-<img src="images/ner_label_ins.png" width = "80%"  align=center /><br>
+<p align="center">
+<img src="images/ner_label_ins.png" width="80%" align="center"/><br/>
 图1. BIO标注方法示例
-</div>
+</p>
 
 根据序列标注结果可以直接得到实体边界和实体类别。类似的，分词、词性标注、语块识别、[语义角色标注](http://book.paddlepaddle.org/07.label_semantic_roles/index.cn.html)等任务都可通过序列标注来解决。使用神经网络模型解决问题的思路通常是：前层网络学习输入的特征表示，网络的最后一层在特征基础上完成最终的任务；对于序列标注问题，通常：使用基于RNN的网络结构学习特征，将学习到的特征接入CRF完成序列标注。实际上是将传统CRF中的线性模型换成了非线性神经网络。沿用CRF的出发点是：CRF使用句子级别的似然概率，能够更好的解决标记偏置问题[[2](#参考文献)]。本例也将基于此思路建立模型。虽然，这里以NER任务作为示例，但所给出的模型可以应用到其他各种序列标注任务中。
 
@@ -43,10 +43,10 @@ NER任务的输入是"一句话"，目标是识别句子中的实体边界及类
 3. 将步骤2中的2个词向量序列作为双向RNN的输入，学习输入序列的特征表示，得到新的特性表示序列；
 4. CRF以步骤3中模型学习到的特征为输入，以标记序列为监督信号，实现序列标注。
 
-<div  align="center">  
-<img src="images/ner_network.png" width = "40%"  align=center /><br>
+<p align="center">
+<img src="images/ner_network.png" width="40%" align="center"/><br/>
 图2. NER 模型网络结构图
-</div>
+</p>
 
 
 ## 数据说明

sequence_tagging_for_ner/data/download.sh

-wget http://cs224d.stanford.edu/assignment2/assignment2.zip
+if [ -f assignment2.zip ]; then
+    echo "data exist"
+else
+    wget http://cs224d.stanford.edu/assignment2/assignment2.zip
+fi
 
 if [ $? -eq 0  ];then
     unzip assignment2.zip

sequence_tagging_for_ner/index.html

@@ -65,10 +65,10 @@
 
 序列标注可以分为Sequence Classification、Segment Classification和Temporal Classification三类[[1](#参考文献)]，本例只考虑Segment Classification，即对输入序列中的每个元素在输出序列中给出对应的标签。对于NER任务，由于需要标识边界，一般采用[BIO标注方法](http://book.paddlepaddle.org/07.label_semantic_roles/)定义的标签集，如下是一个NER的标注结果示例：
 
-<div  align="center">
-<img src="images/ner_label_ins.png" width = "80%"  align=center /><br>
+<p align="center">
+<img src="images/ner_label_ins.png" width="80%" align="center"/><br/>
 图1. BIO标注方法示例
-</div>
+</p>
 
 根据序列标注结果可以直接得到实体边界和实体类别。类似的，分词、词性标注、语块识别、[语义角色标注](http://book.paddlepaddle.org/07.label_semantic_roles/index.cn.html)等任务都可通过序列标注来解决。使用神经网络模型解决问题的思路通常是：前层网络学习输入的特征表示，网络的最后一层在特征基础上完成最终的任务；对于序列标注问题，通常：使用基于RNN的网络结构学习特征，将学习到的特征接入CRF完成序列标注。实际上是将传统CRF中的线性模型换成了非线性神经网络。沿用CRF的出发点是：CRF使用句子级别的似然概率，能够更好的解决标记偏置问题[[2](#参考文献)]。本例也将基于此思路建立模型。虽然，这里以NER任务作为示例，但所给出的模型可以应用到其他各种序列标注任务中。
 
@@ -85,10 +85,10 @@ NER任务的输入是"一句话"，目标是识别句子中的实体边界及类
 3. 将步骤2中的2个词向量序列作为双向RNN的输入，学习输入序列的特征表示，得到新的特性表示序列；
 4. CRF以步骤3中模型学习到的特征为输入，以标记序列为监督信号，实现序列标注。
 
-<div  align="center">  
-<img src="images/ner_network.png" width = "40%"  align=center /><br>
+<p align="center">
+<img src="images/ner_network.png" width="40%" align="center"/><br/>
 图2. NER 模型网络结构图
-</div>
+</p>
 
 
 ## 数据说明

sequence_tagging_for_ner/reader.py

@@ -21,7 +21,7 @@ def canonicalize_word(word, wordset=None, digits=True):
         if (wordset != None) and (word in wordset): return word
         word = canonicalize_digits(word)  # try to canonicalize numbers
     if (wordset == None) or (word in wordset): return word
-    else: return "<UNK>"  # unknown token
+    else: return "UUUNKKK"  # unknown token
 
 
 def data_reader(data_file, word_dict, label_dict):
@@ -35,7 +35,7 @@ def data_reader(data_file, word_dict, label_dict):
     """
 
     def reader():
-        UNK_IDX = word_dict["<UNK>"]
+        UNK_IDX = word_dict["UUUNKKK"]
 
         sentence = []
         labels = []

sequence_tagging_for_ner/train.py

@@ -106,4 +106,5 @@ if __name__ == "__main__":
         test_data_file="data/test",
         vocab_file="data/vocab.txt",
         target_file="data/target.txt",
-        emb_file="data/wordVectors.txt")
+        emb_file="data/wordVectors.txt",
+        model_save_dir="model/")