Merge pull request #141 from Superjom/dssm

add a DSSM example.

dssm/README.md

+# 深度结构化语义模型 (Deep Structured Semantic Models, DSSM)
+DSSM使用DNN模型在一个连续的语义空间中学习文本低纬的表示向量，并且建模两个句子间的语义相似度。
+本例演示如何使用 PaddlePaddle实现一个通用的DSSM 模型，用于建模两个字符串间的语义相似度，
+模型实现支持通用的数据格式，用户替换数据便可以在真实场景中使用该模型。
+
+## 背景介绍
+DSSM \[[1](##参考文献)\]是微软研究院13年提出来的经典的语义模型，用于学习两个文本之间的语义距离，
+广义上模型也可以推广和适用如下场景：
+
+1. CTR预估模型，衡量用户搜索词（Query）与候选网页集合（Documents）之间的相关联程度。
+2. 文本相关性，衡量两个字符串间的语义相关程度。
+3. 自动推荐，衡量User与被推荐的Item之间的关联程度。
+
+DSSM 已经发展成了一个框架，可以很自然地建模两个记录之间的距离关系，
+例如对于文本相关性问题，可以用余弦相似度 (cosin similarity) 来刻画语义距离；
+而对于搜索引擎的结果排序，可以在DSSM上接上Rank损失训练处一个排序模型。
+
+## 模型简介
+在原论文\[[1](#参考文献)\]中，DSSM模型用来衡量用户搜索词 Query 和文档集合 Documents 之间隐含的语义关系，模型结构如下
+
+<p align="center">
+<a id="#figure1"></a>
+<img src="./images/dssm.png"/><br/><br/>
+图 1. DSSM 原始结构
+<p>
+
+其贯彻的思想是， **用DNN将高维特征向量转化为低纬空间的连续向量（图中红色框部分）** ，
+**在上层用cosin similarity来衡量用户搜索词与候选文档间的语义相关性**。
+
+在最顶层损失函数的设计上，原始模型使用类似Word2Vec中负例采样的方法，
+一个Query会抽取正例 $D+$ 和4个负例 $D-$ 整体上算条件概率用对数似然函数作为损失，
+这也就是图 1中类似 $P(D_1|Q)$ 的结构，具体细节请参考原论文。
+
+随着后续优化DSSM模型的结构得以简化\[[3](#参考文献)\]，演变为：
+
+<p align="center">
+<img src="./images/dssm2.png" width="600"/><br/><br/>
+图 2. DSSM通用结构
+<p>
+
+图中的空白方框可以用任何模型替代，比如全连接FC，卷积CNN，RNN等都可以，
+该模型结构专门用于衡量两个元素（比如字符串）间的语义距离。
+
+在现实使用中，DSSM模型会作为基础的积木，搭配上不同的损失函数来实现具体的功能，比如
+
+- 在排序学习中，将 图 2 中结构添加 pairwise rank损失，变成一个排序模型
+- 在CTR预估中，对点击与否做0，1二元分类，添加交叉熵损失变成一个分类模型
+- 在需要对一个子串打分时，可以使用余弦相似度来计算相似度，变成一个回归模型
+
+本例将尝试面向应用提供一个比较通用的解决方案，在模型任务类型上支持
+
+- 分类
+- [-1, 1] 值域内的回归
+- Pairwise-Rank
+
+在生成低纬语义向量的模型结构上，本模型支持以下三种：
+
+- FC, 多层全连接层
+- CNN，卷积神经网络
+- RNN，递归神经网络
+
+## 模型实现
+DSSM模型可以拆成三小块实现，分别是左边和右边的DNN，以及顶层的损失函数。
+在复杂任务中，左右两边DNN的结构可以是不同的，比如在原始论文中左右分别学习Query和Document的semantic vector，
+两者数据的数据不同，建议对应定制DNN的结构。
+
+本例中为了简便和通用，将左右两个DNN的结构都设为相同的，因此只有三个选项FC,CNN,RNN等。
+
+在损失函数的设计方面，也支持三种，分类, 回归, 排序；
+其中，在回归和排序两种损失中，左右两边的匹配程度通过余弦相似度（cossim）来计算；
+在分类任务中，类别预测的分布通过softmax计算。
+
+在其它教程中，对上述很多内容都有过详细的介绍，例如：
+
+- 如何CNN, FC 做文本信息提取可以参考 [text classification](https://github.com/PaddlePaddle/models/blob/develop/text_classification/README.md#模型详解)
+- RNN/GRU 的内容可以参考 [Machine Translation](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md#gated-recurrent-unit-gru)
+- Pairwise Rank即排序学习可参考 [learn to rank](https://github.com/PaddlePaddle/models/blob/develop/ltr/README.md)
+
+相关原理在此不再赘述，本文接下来的篇幅主要集中介绍使用PaddlePaddle实现这些结构上。
+
+如图3，回归和分类模型的结构很相似
+
+<p align="center">
+<img src="./images/dssm3.jpg"/><br/><br/>
+图 3. DSSM for REGRESSION or CLASSIFICATION
+</p>
+
+最重要的组成部分包括词向量，图中`(1)`,`(2)`两个低纬向量的学习器（可以用RNN/CNN/FC中的任意一种实现），
+最上层对应的损失函数。
+
+而Pairwise Rank的结构会复杂一些，类似两个 图 4. 中的结构，增加了对应的损失函数：
+
+- 模型总体思想是，用同一个source(源)为左右两个target(目标)分别打分——`(a),(b)`，学习目标是(a),(b)间的大小关系
+- `(a)`和`(b)`类似图3中结构，用于给source和target的pair打分
+- `(1)`和`(2)`的结构其实是共用的，都表示同一个source，图中为了表达效果展开成两个
+
+<p align="center">
+<img src="./images/dssm2.jpg"/><br/><br/>
+图 4. DSSM for Pairwise Rank
+</p>
+
+下面是各个部分具体的实现方法，所有的代码均包含在 `./network_conf.py` 中。
+
+### 创建文本的词向量表
+```python
+def create_embedding(self, input, prefix=''):
+    '''
+    Create an embedding table whose name has a `prefix`.
+    '''
+    logger.info("create embedding table [%s] which dimention is %d" %
+                (prefix, self.dnn_dims[0]))
+    emb = paddle.layer.embedding(
+        input=input,
+        size=self.dnn_dims[0],
+        param_attr=ParamAttr(name='%s_emb.w' % prefix))
+    return emb
+```
+
+由于输入给词向量表(embedding table)的是一个句子对应的词的ID的列表 ，因此词向量表输出的是词向量的序列。
+
+### CNN 结构实现
+```python
+def create_cnn(self, emb, prefix=''):
+    '''
+    A multi-layer CNN.
+
+    @emb: paddle.layer
+        output of the embedding layer
+    @prefix: str
+        prefix of layers' names, used to share parameters between more than one `cnn` parts.
+    '''
+    def create_conv(context_len, hidden_size, prefix):
+        key = "%s_%d_%d" % (prefix, context_len, hidden_size)
+        conv = paddle.networks.sequence_conv_pool(
+            input=emb,
+            context_len=context_len,
+            hidden_size=hidden_size,
+            # set parameter attr for parameter sharing
+            context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
+            fc_param_attr=ParamAttr(name=key + '_fc.w'),
+            fc_bias_attr=ParamAttr(name=key + '_fc.b'),
+            pool_bias_attr=ParamAttr(name=key + '_pool.b'))
+        return conv
+
+    logger.info('create a sequence_conv_pool which context width is 3')
+    conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
+    logger.info('create a sequence_conv_pool which context width is 4')
+    conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
+    return conv_3, conv_4
+```
+
+CNN 接受 embedding table输出的词向量序列，通过卷积和池化操作捕捉到原始句子的关键信息，
+最终输出一个语义向量（可以认为是句子向量）。
+
+本例的实现中，分别使用了窗口长度为3和4的CNN学到的句子向量按元素求和得到最终的句子向量。
+### RNN 结构实现
+
+RNN很适合学习变长序列的信息，使用RNN来学习句子的信息几乎是自然语言处理任务的标配。
+
+```python
+def create_rnn(self, emb, prefix=''):
+    '''
+    A GRU sentence vector learner.
+    '''
+    gru = paddle.layer.gru_memory(input=emb,)
+    sent_vec = paddle.layer.last_seq(gru)
+    return sent_vec
+```
+### FC 结构实现
+```python
+def create_fc(self, emb, prefix=''):
+    '''
+    A multi-layer fully connected neural networks.
+
+    @emb: paddle.layer
+        output of the embedding layer
+    @prefix: str
+        prefix of layers' names, used to share parameters between more than one `fc` parts.
+    '''
+    _input_layer = paddle.layer.pooling(
+        input=emb, pooling_type=paddle.pooling.Max())
+    fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
+    return fc
+```
+
+在构建FC时需要首先使用`paddle.layer.pooling` 对词向量序列进行最大池化操作，将边长序列转化为一个固定维度向量，
+作为整个句子的语义表达，使用最大池化能够降低句子长度对句向量表达的影响。
+
+### 多层DNN实现
+在 CNN/DNN/FC提取出 semantic vector后，在上层可继续接多层FC来实现深层DNN结构。
+
+```python
+def create_dnn(self, sent_vec, prefix):
+    # if more than three layers exists, a fc layer will be added.
+    if len(self.dnn_dims) > 1:
+        _input_layer = sent_vec
+        for id, dim in enumerate(self.dnn_dims[1:]):
+            name = "%s_fc_%d_%d" % (prefix, id, dim)
+            logger.info("create fc layer [%s] which dimention is %d" %
+                        (name, dim))
+            fc = paddle.layer.fc(
+                input=_input_layer,
+                size=dim,
+                name=name,
+                act=paddle.activation.Tanh(),
+                param_attr=ParamAttr(name='%s.w' % name),
+                bias_attr=ParamAttr(name='%s.b' % name),
+                )
+            _input_layer = fc
+    return _input_layer
+```
+
+### 分类或回归实现
+分类和回归的结构比较相似，因此可以用一个函数创建出来
+
+```python
+def _build_classification_or_regression_model(self, is_classification):
+    '''
+    Build a classification/regression model, and the cost is returned.
+
+    A Classification has 3 inputs:
+      - source sentence
+      - target sentence
+      - classification label
+
+    '''
+    # prepare inputs.
+    assert self.class_num
+
+    source = paddle.layer.data(
+        name='source_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+    target = paddle.layer.data(
+        name='target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    label = paddle.layer.data(
+        name='label_input',
+        type=paddle.data_type.integer_value(self.class_num)
+        if is_classification else paddle.data_type.dense_input)
+
+    prefixs = '_ _'.split(
+    ) if self.share_semantic_generator else 'left right'.split()
+    embed_prefixs = '_ _'.split(
+    ) if self.share_embed else 'left right'.split()
+
+    word_vecs = []
+    for id, input in enumerate([source, target]):
+        x = self.create_embedding(input, prefix=embed_prefixs[id])
+        word_vecs.append(x)
+
+    semantics = []
+    for id, input in enumerate(word_vecs):
+        x = self.model_arch_creater(input, prefix=prefixs[id])
+        semantics.append(x)
+
+    concated_vector = paddle.layer.concat(semantics)
+    prediction = paddle.layer.fc(
+        input=concated_vector,
+        size=self.class_num,
+        act=paddle.activation.Softmax())
+    cost = paddle.layer.classification_cost(
+        input=prediction,
+        label=label) if is_classification else paddle.layer.mse_cost(
+            prediction, label)
+    return cost, prediction, label
+```
+### Pairwise Rank实现
+Pairwise Rank复用上面的DNN结构，同一个source对两个target求相似度打分，
+如果左边的target打分高，预测为1，否则预测为 0。
+
+```python
+def _build_rank_model(self):
+    '''
+    Build a pairwise rank model, and the cost is returned.
+
+    A pairwise rank model has 3 inputs:
+      - source sentence
+      - left_target sentence
+      - right_target sentence
+      - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
+    '''
+    source = paddle.layer.data(
+        name='source_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+    left_target = paddle.layer.data(
+        name='left_target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    right_target = paddle.layer.data(
+        name='right_target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    label = paddle.layer.data(
+        name='label_input', type=paddle.data_type.integer_value(1))
+
+    prefixs = '_ _ _'.split(
+    ) if self.share_semantic_generator else 'source left right'.split()
+    embed_prefixs = '_ _'.split(
+    ) if self.share_embed else 'source target target'.split()
+
+    word_vecs = []
+    for id, input in enumerate([source, left_target, right_target]):
+        x = self.create_embedding(input, prefix=embed_prefixs[id])
+        word_vecs.append(x)
+
+    semantics = []
+    for id, input in enumerate(word_vecs):
+        x = self.model_arch_creater(input, prefix=prefixs[id])
+        semantics.append(x)
+
+    # cossim score of source and left_target
+    left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
+    # cossim score of source and right target
+    right_score = paddle.layer.cos_sim(semantics[0], semantics[2])
+
+    # rank cost
+    cost = paddle.layer.rank_cost(left_score, right_score, label=label)
+    # prediction = left_score - right_score
+    # but this operator is not supported currently.
+    # so AUC will not used.
+    return cost, None, None
+```
+## 数据格式
+在 `./data` 中有简单的示例数据
+
+### 回归的数据格式
+```
+# 3 fields each line:
+#   - source's word ids
+#   - target's word ids
+#   - target
+<ids> \t <ids> \t <float>
+```
+
+比如：
+
+```
+3 6 10 \t 6 8 33 \t 0.7
+6 0 \t 6 9 330 \t 0.03
+```
+### 分类的数据格式
+```
+# 3 fields each line:
+#   - source's word ids
+#   - target's word ids
+#   - target
+<ids> \t <ids> \t <label>
+```
+
+比如：
+
+```
+3 6 10 \t 6 8 33 \t 0
+6 10 \t 8 3 1 \t 1
+```
+
+
+### 排序的数据格式
+```
+# 4 fields each line:
+#   - source's word ids
+#   - target1's word ids
+#   - target2's word ids
+#   - label
+<ids> \t <ids> \t <ids> \t <label>
+```
+
+比如：
+
+```
+7 2 4 \t 2 10 12 \t 9 2 7 10 23 \t 0
+7 2 4 \t 10 12 \t 9 2 21 23 \t 1
+```
+
+## 执行训练
+
+可以直接执行 `python train.py -y 0 --model_arch 0` 使用 `./data/classification` 目录里简单的数据来训练一个分类的FC模型。
+
+其他模型结构也可以通过命令行实现定制，详细命令行参数如下
+
+```
+usage: train.py [-h] [-i TRAIN_DATA_PATH] [-t TEST_DATA_PATH]
+                [-s SOURCE_DIC_PATH] [--target_dic_path TARGET_DIC_PATH]
+                [-b BATCH_SIZE] [-p NUM_PASSES] -y MODEL_TYPE --model_arch
+                MODEL_ARCH
+                [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET]
+                [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS]
+                [--num_workers NUM_WORKERS] [--use_gpu USE_GPU] [-c CLASS_NUM]
+
+PaddlePaddle DSSM example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -i TRAIN_DATA_PATH, --train_data_path TRAIN_DATA_PATH
+                        path of training dataset
+  -t TEST_DATA_PATH, --test_data_path TEST_DATA_PATH
+                        path of testing dataset
+  -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH
+                        path of the source's word dic
+  --target_dic_path TARGET_DIC_PATH
+                        path of the target's word dic, if not set, the
+                        `source_dic_path` will be used
+  -b BATCH_SIZE, --batch_size BATCH_SIZE
+                        size of mini-batch (default:10)
+  -p NUM_PASSES, --num_passes NUM_PASSES
+                        number of passes to run(default:10)
+  -y MODEL_TYPE, --model_type MODEL_TYPE
+                        model type, 0 for classification, 1 for pairwise rank
+                        (default: classification)
+  --model_arch MODEL_ARCH
+                        model architecture, 1 for CNN, 0 for FC, 2 for RNN
+  --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET
+                        whether to share network parameters between source and
+                        target
+  --share_embed SHARE_EMBED
+                        whether to share word embedding between source and
+                        target
+  --dnn_dims DNN_DIMS   dimentions of dnn layers, default is '256,128,64,32',
+                        which means create a 4-layer dnn, demention of each
+                        layer is 256, 128, 64 and 32
+  --num_workers NUM_WORKERS
+                        num worker threads, default 1
+  --use_gpu USE_GPU     whether to use GPU devices (default: False)
+  -c CLASS_NUM, --class_num CLASS_NUM
+                        number of categories for classification task.
+```
+
+## 参考文献
+
+1. Huang P S, He X, Gao J, et al. Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 2013: 2333-2338.
+2. [Microsoft Learning to Rank Datasets](https://www.microsoft.com/en-us/research/project/mslr/)
+3. [Gao J, He X, Deng L. Deep Learning for Web Search and Natural Language Processing[J]. Microsoft Research Technical Report, 2015.](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/wsdm2015.v3.pdf)

dssm/data/classification/test.txt

+苹果	苹果 6s	0
+汽车 驾驶	驾校 培训	1

dssm/data/classification/train.txt

+苹果 六 袋	苹果 6s	0
+新手 汽车 驾驶	驾校 培训	1

dssm/data/rank/test.txt

+苹果 六 袋	苹果 6s	新手 汽车 驾驶	1
+新手 汽车 驾驶	驾校 培训	苹果 6s	0

dssm/data/rank/train.txt

+苹果 六 袋	苹果 6s	新手 汽车 驾驶	1
+新手 汽车 驾驶	驾校 培训	苹果 6s	1

dssm/data/vocab.txt

+UNK
+苹果
+六
+袋
+6s
+新手
+汽车
+驾驶
+驾校
+培训
\ No newline at end of file

dssm/network_conf.py

+from paddle import v2 as paddle
+from paddle.v2.attr import ParamAttr
+from utils import TaskType, logger, ModelType, ModelArch
+
+
+class DSSM(object):
+    def __init__(self,
+                 dnn_dims=[],
+                 vocab_sizes=[],
+                 model_type=ModelType.create_classification(),
+                 model_arch=ModelArch.create_cnn(),
+                 share_semantic_generator=False,
+                 class_num=None,
+                 share_embed=False):
+        '''
+        @dnn_dims: list of int
+            dimentions of each layer in semantic vector generator.
+        @vocab_sizes: 2-d tuple
+            size of both left and right items.
+        @model_type: int
+            type of task, should be 'rank: 0', 'regression: 1' or 'classification: 2'
+        @model_arch: int
+            model architecture
+        @share_semantic_generator: bool
+            whether to share the semantic vector generator for both left and right.
+        @share_embed: bool
+            whether to share the embeddings between left and right.
+        @class_num: int
+            number of categories.
+        '''
+        assert len(
+            vocab_sizes
+        ) == 2, "vocab_sizes specify the sizes left and right inputs, and dim should be 2."
+        assert len(dnn_dims) > 1, "more than two layers is needed."
+
+        self.dnn_dims = dnn_dims
+        self.vocab_sizes = vocab_sizes
+        self.share_semantic_generator = share_semantic_generator
+        self.share_embed = share_embed
+        self.model_type = ModelType(model_type)
+        self.model_arch = ModelArch(model_arch)
+        self.class_num = class_num
+        logger.warning("build DSSM model with config of %s, %s" %
+                       (self.model_type, self.model_arch))
+        logger.info("vocabulary sizes: %s" % str(self.vocab_sizes))
+
+        # bind model architecture
+        _model_arch = {
+            'cnn': self.create_cnn,
+            'fc': self.create_fc,
+            'rnn': self.create_rnn,
+        }
+
+        def _model_arch_creater(emb, prefix=''):
+            sent_vec = _model_arch.get(str(model_arch))(emb, prefix)
+            dnn = self.create_dnn(sent_vec, prefix)
+            return dnn
+
+        self.model_arch_creater = _model_arch_creater
+
+        # build model type
+        _model_type = {
+            'classification': self._build_classification_model,
+            'rank': self._build_rank_model,
+            'regression': self._build_regression_model,
+        }
+        print 'model type: ', str(self.model_type)
+        self.model_type_creater = _model_type[str(self.model_type)]
+
+    def __call__(self):
+        # if self.model_type.is_classification():
+        #     return self._build_classification_model()
+        # return self._build_rank_model()
+        return self.model_type_creater()
+
+    def create_embedding(self, input, prefix=''):
+        '''
+        Create an embedding table whose name has a `prefix`.
+        '''
+        logger.info("create embedding table [%s] which dimention is %d" %
+                    (prefix, self.dnn_dims[0]))
+        emb = paddle.layer.embedding(
+            input=input,
+            size=self.dnn_dims[0],
+            param_attr=ParamAttr(name='%s_emb.w' % prefix))
+        return emb
+
+    def create_fc(self, emb, prefix=''):
+        '''
+        A multi-layer fully connected neural networks.
+
+        @emb: paddle.layer
+            output of the embedding layer
+        @prefix: str
+            prefix of layers' names, used to share parameters between more than one `fc` parts.
+        '''
+        _input_layer = paddle.layer.pooling(
+            input=emb, pooling_type=paddle.pooling.Max())
+        fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
+        return fc
+
+    def create_rnn(self, emb, prefix=''):
+        '''
+        A GRU sentence vector learner.
+        '''
+        gru = paddle.layer.gru_memory(
+            input=emb, )
+        sent_vec = paddle.layer.last_seq(gru)
+        return sent_vec
+
+    def create_cnn(self, emb, prefix=''):
+        '''
+        A multi-layer CNN.
+
+        @emb: paddle.layer
+            output of the embedding layer
+        @prefix: str
+            prefix of layers' names, used to share parameters between more than one `cnn` parts.
+        '''
+
+        def create_conv(context_len, hidden_size, prefix):
+            key = "%s_%d_%d" % (prefix, context_len, hidden_size)
+            conv = paddle.networks.sequence_conv_pool(
+                input=emb,
+                context_len=context_len,
+                hidden_size=hidden_size,
+                # set parameter attr for parameter sharing
+                context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
+                fc_param_attr=ParamAttr(name=key + '_fc.w'),
+                fc_bias_attr=ParamAttr(name=key + '_fc.b'),
+                pool_bias_attr=ParamAttr(name=key + '_pool.b'))
+            return conv
+
+        logger.info('create a sequence_conv_pool which context width is 3')
+        conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
+        logger.info('create a sequence_conv_pool which context width is 4')
+        conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
+
+        return conv_3, conv_4
+
+    def create_dnn(self, sent_vec, prefix):
+        # if more than three layers, than a fc layer will be added.
+        if len(self.dnn_dims) > 1:
+            _input_layer = sent_vec
+            for id, dim in enumerate(self.dnn_dims[1:]):
+                name = "%s_fc_%d_%d" % (prefix, id, dim)
+                logger.info("create fc layer [%s] which dimention is %d" %
+                            (name, dim))
+                fc = paddle.layer.fc(
+                    name=name,
+                    input=_input_layer,
+                    size=dim,
+                    act=paddle.activation.Tanh(),
+                    param_attr=ParamAttr(name='%s.w' % name),
+                    bias_attr=ParamAttr(name='%s.b' % name))
+                _input_layer = fc
+        return _input_layer
+
+    def _build_classification_model(self):
+        logger.info("build classification model")
+        assert self.model_type.is_classification()
+        return self._build_classification_or_regression_model(
+            is_classification=True)
+
+    def _build_regression_model(self):
+        logger.info("build regression model")
+        assert self.model_type.is_regression()
+        return self._build_classification_or_regression_model(
+            is_classification=False)
+
+    def _build_rank_model(self):
+        '''
+        Build a pairwise rank model, and the cost is returned.
+
+        A pairwise rank model has 3 inputs:
+          - source sentence
+          - left_target sentence
+          - right_target sentence
+          - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
+        '''
+        logger.info("build rank model")
+        assert self.model_type.is_rank()
+        source = paddle.layer.data(
+            name='source_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+        left_target = paddle.layer.data(
+            name='left_target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        right_target = paddle.layer.data(
+            name='right_target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        label = paddle.layer.data(
+            name='label_input', type=paddle.data_type.integer_value(1))
+
+        prefixs = '_ _ _'.split(
+        ) if self.share_semantic_generator else 'source left right'.split()
+        embed_prefixs = '_ _'.split(
+        ) if self.share_embed else 'source target target'.split()
+
+        word_vecs = []
+        for id, input in enumerate([source, left_target, right_target]):
+            x = self.create_embedding(input, prefix=embed_prefixs[id])
+            word_vecs.append(x)
+
+        semantics = []
+        for id, input in enumerate(word_vecs):
+            x = self.model_arch_creater(input, prefix=prefixs[id])
+            semantics.append(x)
+
+        # cossim score of source and left_target
+        left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
+        # cossim score of source and right target
+        right_score = paddle.layer.cos_sim(semantics[0], semantics[2])
+
+        # rank cost
+        cost = paddle.layer.rank_cost(left_score, right_score, label=label)
+        # prediction = left_score - right_score
+        # but this operator is not supported currently.
+        # so AUC will not used.
+        return cost, None, None
+
+    def _build_classification_or_regression_model(self, is_classification):
+        '''
+        Build a classification/regression model, and the cost is returned.
+
+        A Classification has 3 inputs:
+          - source sentence
+          - target sentence
+          - classification label
+
+        '''
+        if is_classification:
+            # prepare inputs.
+            assert self.class_num
+
+        source = paddle.layer.data(
+            name='source_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+        target = paddle.layer.data(
+            name='target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        label = paddle.layer.data(
+            name='label_input',
+            type=paddle.data_type.integer_value(self.class_num)
+            if is_classification else paddle.data_type.dense_vector(1))
+
+        prefixs = '_ _'.split(
+        ) if self.share_semantic_generator else 'left right'.split()
+        embed_prefixs = '_ _'.split(
+        ) if self.share_embed else 'left right'.split()
+
+        word_vecs = []
+        for id, input in enumerate([source, target]):
+            x = self.create_embedding(input, prefix=embed_prefixs[id])
+            word_vecs.append(x)
+
+        semantics = []
+        for id, input in enumerate(word_vecs):
+            x = self.model_arch_creater(input, prefix=prefixs[id])
+            semantics.append(x)
+
+        if is_classification:
+            concated_vector = paddle.layer.concat(semantics)
+            prediction = paddle.layer.fc(
+                input=concated_vector,
+                size=self.class_num,
+                act=paddle.activation.Softmax())
+            cost = paddle.layer.classification_cost(
+                input=prediction, label=label)
+        else:
+            prediction = paddle.layer.cos_sim(*semantics)
+            cost = paddle.layer.mse_cost(prediction, label)
+        return cost, prediction, label
+
+
+class RankMetrics(object):
+    '''
+    A custom metrics to calculate AUC.
+
+    Paddle's rank model do not support auc evaluator directly,
+    to make it, infer all the outputs and use python to calculate
+    the metrics.
+    '''
+
+    def __init__(self, model_parameters, left_score_layer, right_score_layer,
+                 label):
+        '''
+        @model_parameters: dict
+            model's parameters
+        @left_score_layer: paddle.layer
+            left part's score
+        @right_score_laeyr: paddle.layer
+            right part's score
+        @label: paddle.data_layer
+            label input
+        '''
+        self.inferer = paddle.inference.Inference(
+            output_layer=[left_score_layer, right_score_layer],
+            parameters=model_parameters)
+
+    def test(self, input):
+        scores = []
+        for id, rcd in enumerate(input()):
+            # output [left_score, right_score, label]
+            res = self.inferer(input=input)
+            scores.append(res)
+        print scores

dssm/reader.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+from utils import UNK, ModelType, TaskType, load_dic, sent2ids, logger, ModelType
+
+
+class Dataset(object):
+    def __init__(self, train_path, test_path, source_dic_path, target_dic_path,
+                 model_type):
+        self.train_path = train_path
+        self.test_path = test_path
+        self.source_dic_path = source_dic_path
+        self.target_dic_path = target_dic_path
+        self.model_type = ModelType(model_type)
+
+        self.source_dic = load_dic(self.source_dic_path)
+        self.target_dic = load_dic(self.target_dic_path)
+
+        _record_reader = {
+            ModelType.CLASSIFICATION_MODE: self._read_classification_record,
+            ModelType.REGRESSION_MODE: self._read_regression_record,
+            ModelType.RANK_MODE: self._read_rank_record,
+        }
+
+        assert isinstance(model_type, ModelType)
+        self.record_reader = _record_reader[model_type.mode]
+
+    def train(self):
+        '''
+        Load trainset.
+        '''
+        logger.info("[reader] load trainset from %s" % self.train_path)
+        with open(self.train_path) as f:
+            for line_id, line in enumerate(f):
+                yield self.record_reader(line)
+
+    def test(self):
+        '''
+        Load testset.
+        '''
+        logger.info("[reader] load testset from %s" % self.test_path)
+        with open(self.test_path) as f:
+            for line_id, line in enumerate(f):
+                yield self.record_reader(line)
+
+    def _read_classification_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <target words> [TAB] <label>
+
+        @line: str
+            a string line which represent a record.
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 3, "wrong format for classification\n" + \
+            "the format shoud be " +\
+            "<source words> [TAB] <target words> [TAB] <label>'"
+        source = sent2ids(fs[0], self.source_dic)
+        target = sent2ids(fs[1], self.target_dic)
+        label = int(fs[2])
+        return (source, target, label, )
+
+    def _read_regression_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <target words> [TAB] <label>
+
+        @line: str
+            a string line which represent a record.
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 3, "wrong format for regression\n" + \
+            "the format shoud be " +\
+            "<source words> [TAB] <target words> [TAB] <label>'"
+        source = sent2ids(fs[0], self.source_dic)
+        target = sent2ids(fs[1], self.target_dic)
+        label = float(fs[2])
+        return (source, target, [label], )
+
+    def _read_rank_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <left_target words> [TAB] <right_target words> [TAB] <label>
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 4, "wrong format for rank\n" + \
+            "the format should be " +\
+            "<source words> [TAB] <left_target words> [TAB] <right_target words> [TAB] <label>"
+
+        source = sent2ids(fs[0], self.source_dic)
+        left_target = sent2ids(fs[1], self.target_dic)
+        right_target = sent2ids(fs[2], self.target_dic)
+        label = int(fs[3])
+
+        return (source, left_target, right_target, label)
+
+
+if __name__ == '__main__':
+    path = './data/classification/train.txt'
+    test_path = './data/classification/test.txt'
+    source_dic = './data/vocab.txt'
+    dataset = Dataset(path, test_path, source_dic, source_dic,
+                      ModelType.CLASSIFICATION)
+
+    for rcd in dataset.train():
+        print rcd

dssm/train.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import argparse
+import gzip
+
+import paddle.v2 as paddle
+from network_conf import DSSM
+import reader
+from utils import TaskType, load_dic, logger, ModelType, ModelArch
+
+parser = argparse.ArgumentParser(description="PaddlePaddle DSSM example")
+
+parser.add_argument(
+    '-i',
+    '--train_data_path',
+    type=str,
+    required=False,
+    help="path of training dataset")
+parser.add_argument(
+    '-t',
+    '--test_data_path',
+    type=str,
+    required=False,
+    help="path of testing dataset")
+parser.add_argument(
+    '-s',
+    '--source_dic_path',
+    type=str,
+    required=False,
+    help="path of the source's word dic")
+parser.add_argument(
+    '--target_dic_path',
+    type=str,
+    required=False,
+    help="path of the target's word dic, if not set, the `source_dic_path` will be used"
+)
+parser.add_argument(
+    '-b',
+    '--batch_size',
+    type=int,
+    default=10,
+    help="size of mini-batch (default:10)")
+parser.add_argument(
+    '-p',
+    '--num_passes',
+    type=int,
+    default=10,
+    help="number of passes to run(default:10)")
+parser.add_argument(
+    '-y',
+    '--model_type',
+    type=int,
+    required=True,
+    default=ModelType.CLASSIFICATION_MODE,
+    help="model type, %d for classification, %d for pairwise rank, %d for regression (default: classification)"
+    % (ModelType.CLASSIFICATION_MODE, ModelType.RANK_MODE,
+       ModelType.REGRESSION_MODE))
+parser.add_argument(
+    '--model_arch',
+    type=int,
+    required=True,
+    default=ModelArch.CNN_MODE,
+    help="model architecture, %d for CNN, %d for FC, %d for RNN" %
+    (ModelArch.CNN_MODE, ModelArch.FC_MODE, ModelArch.RNN_MODE))
+parser.add_argument(
+    '--share_network_between_source_target',
+    type=bool,
+    default=False,
+    help="whether to share network parameters between source and target")
+parser.add_argument(
+    '--share_embed',
+    type=bool,
+    default=False,
+    help="whether to share word embedding between source and target")
+parser.add_argument(
+    '--dnn_dims',
+    type=str,
+    default='256,128,64,32',
+    help="dimentions of dnn layers, default is '256,128,64,32', which means create a 4-layer dnn, demention of each layer is 256, 128, 64 and 32"
+)
+parser.add_argument(
+    '--num_workers', type=int, default=1, help="num worker threads, default 1")
+parser.add_argument(
+    '--use_gpu',
+    type=bool,
+    default=False,
+    help="whether to use GPU devices (default: False)")
+parser.add_argument(
+    '-c',
+    '--class_num',
+    type=int,
+    default=0,
+    help="number of categories for classification task.")
+
+# arguments check.
+args = parser.parse_args()
+args.model_type = ModelType(args.model_type)
+args.model_arch = ModelArch(args.model_arch)
+if args.model_type.is_classification():
+    assert args.class_num > 1, "--class_num should be set in classification task."
+
+layer_dims = [int(i) for i in args.dnn_dims.split(',')]
+target_dic_path = args.source_dic_path if not args.target_dic_path else args.target_dic_path
+
+model_save_name_prefix = "dssm_pass_%s_%s" % (args.model_type,
+                                              args.model_arch, )
+
+
+def train(train_data_path=None,
+          test_data_path=None,
+          source_dic_path=None,
+          target_dic_path=None,
+          model_type=ModelType.create_classification(),
+          model_arch=ModelArch.create_cnn(),
+          batch_size=10,
+          num_passes=10,
+          share_semantic_generator=False,
+          share_embed=False,
+          class_num=None,
+          num_workers=1,
+          use_gpu=False):
+    '''
+    Train the DSSM.
+    '''
+    default_train_path = './data/rank/train.txt'
+    default_test_path = './data/rank/test.txt'
+    default_dic_path = './data/vocab.txt'
+    if not model_type.is_rank():
+        default_train_path = './data/classification/train.txt'
+        default_test_path = './data/classification/test.txt'
+
+    use_default_data = not train_data_path
+
+    if use_default_data:
+        train_data_path = default_train_path
+        test_data_path = default_test_path
+        source_dic_path = default_dic_path
+        target_dic_path = default_dic_path
+
+    dataset = reader.Dataset(
+        train_path=train_data_path,
+        test_path=test_data_path,
+        source_dic_path=source_dic_path,
+        target_dic_path=target_dic_path,
+        model_type=model_type, )
+
+    train_reader = paddle.batch(
+        paddle.reader.shuffle(dataset.train, buf_size=1000),
+        batch_size=batch_size)
+
+    test_reader = paddle.batch(
+        paddle.reader.shuffle(dataset.test, buf_size=1000),
+        batch_size=batch_size)
+
+    paddle.init(use_gpu=use_gpu, trainer_count=num_workers)
+
+    cost, prediction, label = DSSM(
+        dnn_dims=layer_dims,
+        vocab_sizes=[
+            len(load_dic(path)) for path in [source_dic_path, target_dic_path]
+        ],
+        model_type=model_type,
+        model_arch=model_arch,
+        share_semantic_generator=share_semantic_generator,
+        class_num=class_num,
+        share_embed=share_embed)()
+
+    parameters = paddle.parameters.create(cost)
+
+    adam_optimizer = paddle.optimizer.Adam(
+        learning_rate=1e-3,
+        regularization=paddle.optimizer.L2Regularization(rate=1e-3),
+        model_average=paddle.optimizer.ModelAverage(average_window=0.5))
+
+    trainer = paddle.trainer.SGD(
+        cost=cost,
+        extra_layers=None,
+        parameters=parameters,
+        update_equation=adam_optimizer)
+    # trainer = paddle.trainer.SGD(
+    #     cost=cost,
+    #     extra_layers=paddle.evaluator.auc(input=prediction, label=label)
+    #     if prediction and model_type.is_classification() else None,
+    #     parameters=parameters,
+    #     update_equation=adam_optimizer)
+
+    feeding = {}
+    if model_type.is_classification() or model_type.is_regression():
+        feeding = {'source_input': 0, 'target_input': 1, 'label_input': 2}
+    else:
+        feeding = {
+            'source_input': 0,
+            'left_target_input': 1,
+            'right_target_input': 2,
+            'label_input': 3
+        }
+
+    def _event_handler(event):
+        '''
+        Define batch handler
+        '''
+        if isinstance(event, paddle.event.EndIteration):
+            if event.batch_id % 100 == 0:
+                logger.info("Pass %d, Batch %d, Cost %f, %s\n" % (
+                    event.pass_id, event.batch_id, event.cost, event.metrics))
+
+        if isinstance(event, paddle.event.EndPass):
+            if test_reader is not None:
+                if model_type.is_classification():
+                    result = trainer.test(reader=test_reader, feeding=feeding)
+                    logger.info("Test at Pass %d, %s \n" % (event.pass_id,
+                                                            result.metrics))
+                else:
+                    result = None
+            with gzip.open("dssm_%s_pass_%05d.tar.gz" %
+                           (model_save_name_prefix, event.pass_id), "w") as f:
+                parameters.to_tar(f)
+
+    trainer.train(
+        reader=train_reader,
+        event_handler=_event_handler,
+        feeding=feeding,
+        num_passes=num_passes)
+
+    logger.info("Training has finished.")
+
+
+if __name__ == '__main__':
+    train(
+        train_data_path=args.train_data_path,
+        test_data_path=args.test_data_path,
+        source_dic_path=args.source_dic_path,
+        target_dic_path=args.target_dic_path,
+        model_type=ModelType(args.model_type),
+        model_arch=ModelArch(args.model_arch),
+        batch_size=args.batch_size,
+        num_passes=args.num_passes,
+        share_semantic_generator=args.share_network_between_source_target,
+        share_embed=args.share_embed,
+        class_num=args.class_num,
+        num_workers=args.num_workers,
+        use_gpu=args.use_gpu)

dssm/utils.py

+import paddle
+
+UNK = 0
+
+logger = logging.getLogger("paddle")
+logger.setLevel(logging.INFO)
+
+
+def mode_attr_name(mode):
+    return mode.upper() + '_MODE'
+
+
+def create_attrs(cls):
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, mode_attr_name(mode), id)
+
+
+def make_check_method(cls):
+    '''
+    create methods for classes.
+    '''
+
+    def method(mode):
+        def _method(self):
+            return self.mode == getattr(cls, mode_attr_name(mode))
+
+        return _method
+
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, 'is_' + mode, method(mode))
+
+
+def make_create_method(cls):
+    def method(mode):
+        @staticmethod
+        def _method():
+            key = getattr(cls, mode_attr_name(mode))
+            return cls(key)
+
+        return _method
+
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, 'create_' + mode, method(mode))
+
+
+def make_str_method(cls, type_name='unk'):
+    def _str_(self):
+        for mode in cls.modes:
+            if self.mode == getattr(cls, mode_attr_name(mode)):
+                return mode
+
+    def _hash_(self):
+        return self.mode
+
+    setattr(cls, '__str__', _str_)
+    setattr(cls, '__repr__', _str_)
+    setattr(cls, '__hash__', _hash_)
+    cls.__name__ = type_name
+
+
+def _init_(self, mode, cls):
+    if isinstance(mode, int):
+        self.mode = mode
+    elif isinstance(mode, cls):
+        self.mode = mode.mode
+    else:
+        raise Exception("wrong mode type, get type: %s, value: %s" %
+                        (type(mode), mode))
+
+
+def build_mode_class(cls):
+    create_attrs(cls)
+    make_str_method(cls)
+    make_check_method(cls)
+    make_create_method(cls)
+
+
+class TaskType(object):
+    modes = 'train test infer'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, TaskType)
+
+
+class ModelType:
+    modes = 'classification rank regression'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, ModelType)
+
+
+class ModelArch:
+    modes = 'fc cnn rnn'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, ModelArch)
+
+
+build_mode_class(TaskType)
+build_mode_class(ModelType)
+build_mode_class(ModelArch)
+
+
+def sent2ids(sent, vocab):
+    '''
+    transform a sentence to a list of ids.
+
+    @sent: str
+        a sentence.
+    @vocab: dict
+        a word dic
+    '''
+    return [vocab.get(w, UNK) for w in sent.split()]
+
+
+def load_dic(path):
+    '''
+    word dic format:
+      each line is a word
+    '''
+    dic = {}
+    with open(path) as f:
+        for id, line in enumerate(f):
+            w = line.strip()
+            dic[w] = id
+    return dic
+
+
+if __name__ == '__main__':
+    t = TaskType(1)
+    t = TaskType.create_train()
+    print t
+    print 'is', t.is_train()