# Deep Structured Semantic Models (DSSM)
Deep Structured Semantic Models (DSSM) is simple but powerful DNN based model for matching web search queries and the URL based documents. This example demonstrates how to use PaddlePaddle to implement a generic DSSM model for modeling the semantic similarity between two strings.

## Background Introduction
DSSM \[[1](##References)]is a classic semantic model proposed by the Institute of Physics. It is used to study the semantic distance between two texts. The general implementation of DSSM is as follows.

1. The CTR predictor measures the degree of association between a user search query and a candidate web page.
2. Text relevance, which measures the degree of semantic correlation between two strings.
3. Automatically recommend, measure the degree of association between User and the recommended Item.


## Model Architecture

In the original paper \[[1](#References)] the DSSM model uses the implicit semantic relation between the user search query and the document as metric. The model structure is as follows

<p align="center">
<img src="./images/dssm.png"/><br/><br/>
Figure 1. DSSM In the original paper
</p>


With the subsequent optimization of the DSSM model to simplify the structure \[[3](#References)]，the model becomes：

<p align="center">
<img src="./images/dssm2.png" width="600"/><br/><br/>
Figure 2. DSSM generic structure
</p>

The blank box in the figure can be replaced by any model, such as fully connected FC, convoluted CNN, RNN, etc. The structure is designed to measure the semantic distance between two elements (such as strings).

In practice，DSSM model serves as a basic building block, with different loss functions to achieve specific functions, such as

- In ranking system, the pairwise rank loss function.
- In the CTR estimate, instead of the binary classification on the click, use cross-entropy loss for a classification model
- In regression model,  the cosine similarity is used to calculate the similarity

## Model Implementation
At a high level, DSSM model is composed of three components: the left and right DNN, and loss function on top of them. In complex tasks, the structure of the left DNN and the light DNN can be different. In this example, we keep these two DNN structures the same. And we choose any of FC, CNN, and RNN for the DNN architecture.

In PaddlePaddle, the loss functions are supported for any of classification, regression, and ranking. Among them, the distance between the left and right DNN is calculated by the cosine similarity. In the classification task, the predicted distribution is calculated by softmax.

Here we demonstrate:

- How CNN, FC do text information extraction can refer to [text classification](https://github.com/PaddlePaddle/models/blob/develop/text_classification/README.md#模型详解)
- The contents of the RNN / GRU can be found in  [Machine Translation](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md#gated-recurrent-unit-gru)
- For Pairwise Rank learning, please refer to [learn to rank](https://github.com/PaddlePaddle/models/blob/develop/ltr/README.md)

Figure 3 shows the general architecture for both regression and classification models.

<p align="center">
<img src="./images/dssm3.jpg"/><br/><br/>
Figure 3. DSSM for REGRESSION or CLASSIFICATION
</p>

The structure of the Pairwise Rank is more complex, as shown in Figure 4.

<p align="center">
<img src="./images/dssm2.jpg"/><br/><br/>
图 4. DSSM for Pairwise Rank
</p>

In below, we describe how to train DSSM model in PaddlePaddle. All the codes are included in  `./network_conf.py`.


### Create a word vector table for the text
```python
def create_embedding(self, input, prefix=''):
    '''
    Create an embedding table whose name has a `prefix`.
    '''
    logger.info("create embedding table [%s] which dimention is %d" %
                (prefix, self.dnn_dims[0]))
    emb = paddle.layer.embedding(
        input=input,
        size=self.dnn_dims[0],
        param_attr=ParamAttr(name='%s_emb.w' % prefix))
    return emb
```

Since the input (embedding table) is a list of the IDs of the words corresponding to a sentence, the word vector table outputs the sequence of word vectors.

### CNN implementation
```python
def create_cnn(self, emb, prefix=''):
    '''
    A multi-layer CNN.

    @emb: paddle.layer
        output of the embedding layer
    @prefix: str
        prefix of layers' names, used to share parameters between more than one `cnn` parts.
    '''
    def create_conv(context_len, hidden_size, prefix):
        key = "%s_%d_%d" % (prefix, context_len, hidden_size)
        conv = paddle.networks.sequence_conv_pool(
            input=emb,
            context_len=context_len,
            hidden_size=hidden_size,
            # set parameter attr for parameter sharing
            context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
            fc_param_attr=ParamAttr(name=key + '_fc.w'),
            fc_bias_attr=ParamAttr(name=key + '_fc.b'),
            pool_bias_attr=ParamAttr(name=key + '_pool.b'))
        return conv

    logger.info('create a sequence_conv_pool which context width is 3')
    conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
    logger.info('create a sequence_conv_pool which context width is 4')
    conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
    return conv_3, conv_4
```

CNN accepts the word sequence of the embedding table, then process the data by convolution and pooling, and finally outputs a semantic vector.

### RNN implementation

RNN is suitable for learning variable length of the information

```python
def create_rnn(self, emb, prefix=''):
    '''
    A GRU sentence vector learner.
    '''
    gru = paddle.layer.gru_memory(input=emb,)
    sent_vec = paddle.layer.last_seq(gru)
    return sent_vec
```

### FC implementation

```python
def create_fc(self, emb, prefix=''):
    '''
    A multi-layer fully connected neural networks.

    @emb: paddle.layer
        output of the embedding layer
    @prefix: str
        prefix of layers' names, used to share parameters between more than one `fc` parts.
    '''
    _input_layer = paddle.layer.pooling(
        input=emb, pooling_type=paddle.pooling.Max())
    fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
    return fc
```

In the construction of FC, we use `paddle.layer.pooling` for the maximum pooling operation on the word vector sequence. Then we transform the sequence into a fixed dimensional vector.

### Multi-layer DNN implementation

```python
def create_dnn(self, sent_vec, prefix):
    # if more than three layers exists, a fc layer will be added.
    if len(self.dnn_dims) > 1:
        _input_layer = sent_vec
        for id, dim in enumerate(self.dnn_dims[1:]):
            name = "%s_fc_%d_%d" % (prefix, id, dim)
            logger.info("create fc layer [%s] which dimention is %d" %
                        (name, dim))
            fc = paddle.layer.fc(
                input=_input_layer,
                size=dim,
                name=name,
                act=paddle.activation.Tanh(),
                param_attr=ParamAttr(name='%s.w' % name),
                bias_attr=ParamAttr(name='%s.b' % name),
                )
            _input_layer = fc
    return _input_layer
```

### Classification / Regression
The structure of classification and regression is similar. Below function can be used for both tasks.

```python
def _build_classification_or_regression_model(self, is_classification):
    '''
    Build a classification/regression model, and the cost is returned.

    A Classification has 3 inputs:
      - source sentence
      - target sentence
      - classification label

    '''
    # prepare inputs.
    assert self.class_num

    source = paddle.layer.data(
        name='source_input',
        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
    target = paddle.layer.data(
        name='target_input',
        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
    label = paddle.layer.data(
        name='label_input',
        type=paddle.data_type.integer_value(self.class_num)
        if is_classification else paddle.data_type.dense_input)

    prefixs = '_ _'.split(
    ) if self.share_semantic_generator else 'left right'.split()
    embed_prefixs = '_ _'.split(
    ) if self.share_embed else 'left right'.split()

    word_vecs = []
    for id, input in enumerate([source, target]):
        x = self.create_embedding(input, prefix=embed_prefixs[id])
        word_vecs.append(x)

    semantics = []
    for id, input in enumerate(word_vecs):
        x = self.model_arch_creater(input, prefix=prefixs[id])
        semantics.append(x)

    concated_vector = paddle.layer.concat(semantics)
    prediction = paddle.layer.fc(
        input=concated_vector,
        size=self.class_num,
        act=paddle.activation.Softmax())
    cost = paddle.layer.classification_cost(
        input=prediction,
        label=label) if is_classification else paddle.layer.mse_cost(
            prediction, label)
    return cost, prediction, label
```

### Pairwise Rank


```python
def _build_rank_model(self):
    '''
    Build a pairwise rank model, and the cost is returned.

    A pairwise rank model has 3 inputs:
      - source sentence
      - left_target sentence
      - right_target sentence
      - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
    '''
    source = paddle.layer.data(
        name='source_input',
        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
    left_target = paddle.layer.data(
        name='left_target_input',
        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
    right_target = paddle.layer.data(
        name='right_target_input',
        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
    label = paddle.layer.data(
        name='label_input', type=paddle.data_type.integer_value(1))

    prefixs = '_ _ _'.split(
    ) if self.share_semantic_generator else 'source left right'.split()
    embed_prefixs = '_ _'.split(
    ) if self.share_embed else 'source target target'.split()

    word_vecs = []
    for id, input in enumerate([source, left_target, right_target]):
        x = self.create_embedding(input, prefix=embed_prefixs[id])
        word_vecs.append(x)

    semantics = []
    for id, input in enumerate(word_vecs):
        x = self.model_arch_creater(input, prefix=prefixs[id])
        semantics.append(x)

    # cossim score of source and left_target
    left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
    # cossim score of source and right target
    right_score = paddle.layer.cos_sim(semantics[0], semantics[2])

    # rank cost
    cost = paddle.layer.rank_cost(left_score, right_score, label=label)
    # prediction = left_score - right_score
    # but this operator is not supported currently.
    # so AUC will not used.
    return cost, None, None
```
## Data Format
Below is a simple example for the data in `./data`

### Regression data format
```
# 3 fields each line:
#   - source's word ids
#   - target's word ids
#   - target
<ids> \t <ids> \t <float>
```

The example of this format is as follows.

```
3 6 10 \t 6 8 33 \t 0.7
6 0 \t 6 9 330 \t 0.03
```

### Classification data format
```
# 3 fields each line:
#   - source's word ids
#   - target's word ids
#   - target
<ids> \t <ids> \t <label>
```

The example of this format is as follows.


```
3 6 10 \t 6 8 33 \t 0
6 10 \t 8 3 1 \t 1
```


### Ranking data format
```
# 4 fields each line:
#   - source's word ids
#   - target1's word ids
#   - target2's word ids
#   - label
<ids> \t <ids> \t <ids> \t <label>
```

The example of this format is as follows.

```
7 2 4 \t 2 10 12 \t 9 2 7 10 23 \t 0
7 2 4 \t 10 12 \t 9 2 21 23 \t 1
```

## Training

We use `python train.py -y 0 --model_arch 0` with the data in  `./data/classification` to train a DSSM model for classification.


```
usage: train.py [-h] [-i TRAIN_DATA_PATH] [-t TEST_DATA_PATH]
                [-s SOURCE_DIC_PATH] [--target_dic_path TARGET_DIC_PATH]
                [-b BATCH_SIZE] [-p NUM_PASSES] -y MODEL_TYPE -a MODEL_ARCH
                [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET]
                [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS]
                [--num_workers NUM_WORKERS] [--use_gpu USE_GPU] [-c CLASS_NUM]
                [--model_output_prefix MODEL_OUTPUT_PREFIX]
                [-g NUM_BATCHES_TO_LOG] [-e NUM_BATCHES_TO_TEST]
                [-z NUM_BATCHES_TO_SAVE_MODEL]

PaddlePaddle DSSM example

optional arguments:
  -h, --help            show this help message and exit
  -i TRAIN_DATA_PATH, --train_data_path TRAIN_DATA_PATH
                        path of training dataset
  -t TEST_DATA_PATH, --test_data_path TEST_DATA_PATH
                        path of testing dataset
  -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH
                        path of the source's word dic
  --target_dic_path TARGET_DIC_PATH
                        path of the target's word dic, if not set, the
                        `source_dic_path` will be used
  -b BATCH_SIZE, --batch_size BATCH_SIZE
                        size of mini-batch (default:10)
  -p NUM_PASSES, --num_passes NUM_PASSES
                        number of passes to run(default:10)
  -y MODEL_TYPE, --model_type MODEL_TYPE
                        model type, 0 for classification, 1 for pairwise rank,
                        2 for regression (default: classification)
  -a MODEL_ARCH, --model_arch MODEL_ARCH
                        model architecture, 1 for CNN, 0 for FC, 2 for RNN
  --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET
                        whether to share network parameters between source and
                        target
  --share_embed SHARE_EMBED
                        whether to share word embedding between source and
                        target
  --dnn_dims DNN_DIMS   dimentions of dnn layers, default is '256,128,64,32',
                        which means create a 4-layer dnn, demention of each
                        layer is 256, 128, 64 and 32
  --num_workers NUM_WORKERS
                        num worker threads, default 1
  --use_gpu USE_GPU     whether to use GPU devices (default: False)
  -c CLASS_NUM, --class_num CLASS_NUM
                        number of categories for classification task.
  --model_output_prefix MODEL_OUTPUT_PREFIX
                        prefix of the path for model to store, (default: ./)
  -g NUM_BATCHES_TO_LOG, --num_batches_to_log NUM_BATCHES_TO_LOG
                        number of batches to output train log, (default: 100)
  -e NUM_BATCHES_TO_TEST, --num_batches_to_test NUM_BATCHES_TO_TEST
                        number of batches to test, (default: 200)
  -z NUM_BATCHES_TO_SAVE_MODEL, --num_batches_to_save_model NUM_BATCHES_TO_SAVE_MODEL
                        number of batches to output model, (default: 400)
```

Parameter description:

- `train_data_path` Training data path
- `test_data_path`  Test data path, optional
- `source_dic_path`  Source dictionary path
- `target_dic_path` 目Target dictionary path
- `model_type`  The type of loss function of the model: classification 0, sort 1, regression 2
- `model_arch` Model structure: FC 0，CNN 1, RNN 2
- `dnn_dims` The dimension of each layer of the model is set, the default is `256,128,64,32`，with 4 layers.

## To predict using the trained model
```
usage: infer.py [-h] --model_path MODEL_PATH -i DATA_PATH -o
                PREDICTION_OUTPUT_PATH -y MODEL_TYPE [-s SOURCE_DIC_PATH]
                [--target_dic_path TARGET_DIC_PATH] -a MODEL_ARCH
                [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET]
                [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS]
                [-c CLASS_NUM]

PaddlePaddle DSSM infer

optional arguments:
  -h, --help            show this help message and exit
  --model_path MODEL_PATH
                        path of model parameters file
  -i DATA_PATH, --data_path DATA_PATH
                        path of the dataset to infer
  -o PREDICTION_OUTPUT_PATH, --prediction_output_path PREDICTION_OUTPUT_PATH
                        path to output the prediction
  -y MODEL_TYPE, --model_type MODEL_TYPE
                        model type, 0 for classification, 1 for pairwise rank,
                        2 for regression (default: classification)
  -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH
                        path of the source's word dic
  --target_dic_path TARGET_DIC_PATH
                        path of the target's word dic, if not set, the
                        `source_dic_path` will be used
  -a MODEL_ARCH, --model_arch MODEL_ARCH
                        model architecture, 1 for CNN, 0 for FC, 2 for RNN
  --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET
                        whether to share network parameters between source and
                        target
  --share_embed SHARE_EMBED
                        whether to share word embedding between source and
                        target
  --dnn_dims DNN_DIMS   dimentions of dnn layers, default is '256,128,64,32',
                        which means create a 4-layer dnn, demention of each
                        layer is 256, 128, 64 and 32
  -c CLASS_NUM, --class_num CLASS_NUM
                        number of categories for classification task.
```

Important parameters are

- `data_path` Path for the data to predict
- `prediction_output_path` Prediction output path

## References

1. Huang P S, He X, Gao J, et al. Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 2013: 2333-2338.
2. [Microsoft Learning to Rank Datasets](https://www.microsoft.com/en-us/research/project/mslr/)
3. [Gao J, He X, Deng L. Deep Learning for Web Search and Natural Language Processing[J]. Microsoft Research Technical Report, 2015.](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/wsdm2015.v3.pdf)
