# Deep Structured Semantic Models (DSSM) Deep Structured Semantic Models (DSSM) is simple but powerful DNN based model for matching web search queries and the URL based documents. This example demonstrates how to use PaddlePaddle to implement a generic DSSM model for modeling the semantic similarity between two strings. ## Background Introduction DSSM \[[1](##References)]is a classic semantic model proposed by the Institute of Physics. It is used to study the semantic distance between two texts. The general implementation of DSSM is as follows. 1. The CTR predictor measures the degree of association between a user search query and a candidate web page. 2. Text relevance, which measures the degree of semantic correlation between two strings. 3. Automatically recommend, measure the degree of association between User and the recommended Item. ## Model Architecture In the original paper \[[1](#References)] the DSSM model uses the implicit semantic relation between the user search query and the document as metric. The model structure is as follows

Figure 1. DSSM In the original paper

With the subsequent optimization of the DSSM model to simplify the structure \[[3](#References)]，the model becomes：

Figure 2. DSSM generic structure

The blank box in the figure can be replaced by any model, such as fully connected FC, convoluted CNN, RNN, etc. The structure is designed to measure the semantic distance between two elements (such as strings). In practice，DSSM model serves as a basic building block, with different loss functions to achieve specific functions, such as - In ranking system, the pairwise rank loss function. - In the CTR estimate, instead of the binary classification on the click, use cross-entropy loss for a classification model - In regression model, the cosine similarity is used to calculate the similarity ## Model Implementation At a high level, DSSM model is composed of three components: the left and right DNN, and loss function on top of them. In complex tasks, the structure of the left DNN and the light DNN can be different. In this example, we keep these two DNN structures the same. And we choose any of FC, CNN, and RNN for the DNN architecture. In PaddlePaddle, the loss functions are supported for any of classification, regression, and ranking. Among them, the distance between the left and right DNN is calculated by the cosine similarity. In the classification task, the predicted distribution is calculated by softmax. Here we demonstrate: - How CNN, FC do text information extraction can refer to [text classification](https://github.com/PaddlePaddle/models/blob/develop/text_classification/README.md#模型详解) - The contents of the RNN / GRU can be found in [Machine Translation](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md#gated-recurrent-unit-gru) - For Pairwise Rank learning, please refer to [learn to rank](https://github.com/PaddlePaddle/models/blob/develop/ltr/README.md) Figure 3 shows the general architecture for both regression and classification models.

Figure 3. DSSM for REGRESSION or CLASSIFICATION

The structure of the Pairwise Rank is more complex, as shown in Figure 4.

图 4. DSSM for Pairwise Rank

In below, we describe how to train DSSM model in PaddlePaddle. All the codes are included in `./network_conf.py`. ### Create a word vector table for the text ```python def create_embedding(self, input, prefix=''): ''' Create an embedding table whose name has a `prefix`. ''' logger.info("create embedding table [%s] which dimention is %d" % (prefix, self.dnn_dims[0])) emb = paddle.layer.embedding( input=input, size=self.dnn_dims[0], param_attr=ParamAttr(name='%s_emb.w' % prefix)) return emb ``` Since the input (embedding table) is a list of the IDs of the words corresponding to a sentence, the word vector table outputs the sequence of word vectors. ### CNN implementation ```python def create_cnn(self, emb, prefix=''): ''' A multi-layer CNN. @emb: paddle.layer output of the embedding layer @prefix: str prefix of layers' names, used to share parameters between more than one `cnn` parts. ''' def create_conv(context_len, hidden_size, prefix): key = "%s_%d_%d" % (prefix, context_len, hidden_size) conv = paddle.networks.sequence_conv_pool( input=emb, context_len=context_len, hidden_size=hidden_size, # set parameter attr for parameter sharing context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'), fc_param_attr=ParamAttr(name=key + '_fc.w'), fc_bias_attr=ParamAttr(name=key + '_fc.b'), pool_bias_attr=ParamAttr(name=key + '_pool.b')) return conv logger.info('create a sequence_conv_pool which context width is 3') conv_3 = create_conv(3, self.dnn_dims[1], "cnn") logger.info('create a sequence_conv_pool which context width is 4') conv_4 = create_conv(4, self.dnn_dims[1], "cnn") return conv_3, conv_4 ``` CNN accepts the word sequence of the embedding table, then process the data by convolution and pooling, and finally outputs a semantic vector. ### RNN implementation RNN is suitable for learning variable length of the information ```python def create_rnn(self, emb, prefix=''): ''' A GRU sentence vector learner. ''' gru = paddle.layer.gru_memory(input=emb,) sent_vec = paddle.layer.last_seq(gru) return sent_vec ``` ### FC implementation ```python def create_fc(self, emb, prefix=''): ''' A multi-layer fully connected neural networks. @emb: paddle.layer output of the embedding layer @prefix: str prefix of layers' names, used to share parameters between more than one `fc` parts. ''' _input_layer = paddle.layer.pooling( input=emb, pooling_type=paddle.pooling.Max()) fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1]) return fc ``` In the construction of FC, we use `paddle.layer.pooling` for the maximum pooling operation on the word vector sequence. Then we transform the sequence into a fixed dimensional vector. ### Multi-layer DNN implementation ```python def create_dnn(self, sent_vec, prefix): # if more than three layers exists, a fc layer will be added. if len(self.dnn_dims) > 1: _input_layer = sent_vec for id, dim in enumerate(self.dnn_dims[1:]): name = "%s_fc_%d_%d" % (prefix, id, dim) logger.info("create fc layer [%s] which dimention is %d" % (name, dim)) fc = paddle.layer.fc( input=_input_layer, size=dim, name=name, act=paddle.activation.Tanh(), param_attr=ParamAttr(name='%s.w' % name), bias_attr=ParamAttr(name='%s.b' % name), ) _input_layer = fc return _input_layer ``` ### Classification / Regression The structure of classification and regression is similar. Below function can be used for both tasks. ```python def _build_classification_or_regression_model(self, is_classification): ''' Build a classification/regression model, and the cost is returned. A Classification has 3 inputs: - source sentence - target sentence - classification label ''' # prepare inputs. assert self.class_num source = paddle.layer.data( name='source_input', type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0])) target = paddle.layer.data( name='target_input', type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1])) label = paddle.layer.data( name='label_input', type=paddle.data_type.integer_value(self.class_num) if is_classification else paddle.data_type.dense_input) prefixs = '_ _'.split( ) if self.share_semantic_generator else 'left right'.split() embed_prefixs = '_ _'.split( ) if self.share_embed else 'left right'.split() word_vecs = [] for id, input in enumerate([source, target]): x = self.create_embedding(input, prefix=embed_prefixs[id]) word_vecs.append(x) semantics = [] for id, input in enumerate(word_vecs): x = self.model_arch_creater(input, prefix=prefixs[id]) semantics.append(x) concated_vector = paddle.layer.concat(semantics) prediction = paddle.layer.fc( input=concated_vector, size=self.class_num, act=paddle.activation.Softmax()) cost = paddle.layer.classification_cost( input=prediction, label=label) if is_classification else paddle.layer.mse_cost( prediction, label) return cost, prediction, label ``` ### Pairwise Rank ```python def _build_rank_model(self): ''' Build a pairwise rank model, and the cost is returned. A pairwise rank model has 3 inputs: - source sentence - left_target sentence - right_target sentence - label, 1 if left_target should be sorted in front of right_target, otherwise 0. ''' source = paddle.layer.data( name='source_input', type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0])) left_target = paddle.layer.data( name='left_target_input', type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1])) right_target = paddle.layer.data( name='right_target_input', type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1])) label = paddle.layer.data( name='label_input', type=paddle.data_type.integer_value(1)) prefixs = '_ _ _'.split( ) if self.share_semantic_generator else 'source left right'.split() embed_prefixs = '_ _'.split( ) if self.share_embed else 'source target target'.split() word_vecs = [] for id, input in enumerate([source, left_target, right_target]): x = self.create_embedding(input, prefix=embed_prefixs[id]) word_vecs.append(x) semantics = [] for id, input in enumerate(word_vecs): x = self.model_arch_creater(input, prefix=prefixs[id]) semantics.append(x) # cossim score of source and left_target left_score = paddle.layer.cos_sim(semantics[0], semantics[1]) # cossim score of source and right target right_score = paddle.layer.cos_sim(semantics[0], semantics[2]) # rank cost cost = paddle.layer.rank_cost(left_score, right_score, label=label) # prediction = left_score - right_score # but this operator is not supported currently. # so AUC will not used. return cost, None, None ``` ## Data Format Below is a simple example for the data in `./data` ### Regression data format ``` # 3 fields each line: # - source's word ids # - target's word ids # - target <ids> \t <ids> \t <float> ``` The example of this format is as follows. ``` 3 6 10 \t 6 8 33 \t 0.7 6 0 \t 6 9 330 \t 0.03 ``` ### Classification data format ``` # 3 fields each line: # - source's word ids # - target's word ids # - target <ids> \t <ids> \t ``` The example of this format is as follows. ``` 3 6 10 \t 6 8 33 \t 0 6 10 \t 8 3 1 \t 1 ``` ### Ranking data format ``` # 4 fields each line: # - source's word ids # - target1's word ids # - target2's word ids # - label <ids> \t <ids> \t <ids> \t ``` The example of this format is as follows. ``` 7 2 4 \t 2 10 12 \t 9 2 7 10 23 \t 0 7 2 4 \t 10 12 \t 9 2 21 23 \t 1 ``` ## Training We use `python train.py -y 0 --model_arch 0` with the data in `./data/classification` to train a DSSM model for classification. ``` usage: train.py [-h] [-i TRAIN_DATA_PATH] [-t TEST_DATA_PATH] [-s SOURCE_DIC_PATH] [--target_dic_path TARGET_DIC_PATH] [-b BATCH_SIZE] [-p NUM_PASSES] -y MODEL_TYPE -a MODEL_ARCH [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET] [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS] [--num_workers NUM_WORKERS] [--use_gpu USE_GPU] [-c CLASS_NUM] [--model_output_prefix MODEL_OUTPUT_PREFIX] [-g NUM_BATCHES_TO_LOG] [-e NUM_BATCHES_TO_TEST] [-z NUM_BATCHES_TO_SAVE_MODEL] PaddlePaddle DSSM example optional arguments: -h, --help show this help message and exit -i TRAIN_DATA_PATH, --train_data_path TRAIN_DATA_PATH path of training dataset -t TEST_DATA_PATH, --test_data_path TEST_DATA_PATH path of testing dataset -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH path of the source's word dic --target_dic_path TARGET_DIC_PATH path of the target's word dic, if not set, the `source_dic_path` will be used -b BATCH_SIZE, --batch_size BATCH_SIZE size of mini-batch (default:10) -p NUM_PASSES, --num_passes NUM_PASSES number of passes to run(default:10) -y MODEL_TYPE, --model_type MODEL_TYPE model type, 0 for classification, 1 for pairwise rank, 2 for regression (default: classification) -a MODEL_ARCH, --model_arch MODEL_ARCH model architecture, 1 for CNN, 0 for FC, 2 for RNN --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET whether to share network parameters between source and target --share_embed SHARE_EMBED whether to share word embedding between source and target --dnn_dims DNN_DIMS dimentions of dnn layers, default is '256,128,64,32', which means create a 4-layer dnn, demention of each layer is 256, 128, 64 and 32 --num_workers NUM_WORKERS num worker threads, default 1 --use_gpu USE_GPU whether to use GPU devices (default: False) -c CLASS_NUM, --class_num CLASS_NUM number of categories for classification task. --model_output_prefix MODEL_OUTPUT_PREFIX prefix of the path for model to store, (default: ./) -g NUM_BATCHES_TO_LOG, --num_batches_to_log NUM_BATCHES_TO_LOG number of batches to output train log, (default: 100) -e NUM_BATCHES_TO_TEST, --num_batches_to_test NUM_BATCHES_TO_TEST number of batches to test, (default: 200) -z NUM_BATCHES_TO_SAVE_MODEL, --num_batches_to_save_model NUM_BATCHES_TO_SAVE_MODEL number of batches to output model, (default: 400) ``` Parameter description: - `train_data_path` Training data path - `test_data_path` Test data path, optional - `source_dic_path` Source dictionary path - `target_dic_path` 目Target dictionary path - `model_type` The type of loss function of the model: classification 0, sort 1, regression 2 - `model_arch` Model structure: FC 0，CNN 1, RNN 2 - `dnn_dims` The dimension of each layer of the model is set, the default is `256,128,64,32`，with 4 layers. ## To predict using the trained model ``` usage: infer.py [-h] --model_path MODEL_PATH -i DATA_PATH -o PREDICTION_OUTPUT_PATH -y MODEL_TYPE [-s SOURCE_DIC_PATH] [--target_dic_path TARGET_DIC_PATH] -a MODEL_ARCH [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET] [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS] [-c CLASS_NUM] PaddlePaddle DSSM infer optional arguments: -h, --help show this help message and exit --model_path MODEL_PATH path of model parameters file -i DATA_PATH, --data_path DATA_PATH path of the dataset to infer -o PREDICTION_OUTPUT_PATH, --prediction_output_path PREDICTION_OUTPUT_PATH path to output the prediction -y MODEL_TYPE, --model_type MODEL_TYPE model type, 0 for classification, 1 for pairwise rank, 2 for regression (default: classification) -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH path of the source's word dic --target_dic_path TARGET_DIC_PATH path of the target's word dic, if not set, the `source_dic_path` will be used -a MODEL_ARCH, --model_arch MODEL_ARCH model architecture, 1 for CNN, 0 for FC, 2 for RNN --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET whether to share network parameters between source and target --share_embed SHARE_EMBED whether to share word embedding between source and target --dnn_dims DNN_DIMS dimentions of dnn layers, default is '256,128,64,32', which means create a 4-layer dnn, demention of each layer is 256, 128, 64 and 32 -c CLASS_NUM, --class_num CLASS_NUM number of categories for classification task. ``` Important parameters are - `data_path` Path for the data to predict - `prediction_output_path` Prediction output path ## References 1. Huang P S, He X, Gao J, et al. Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 2013: 2333-2338. 2. [Microsoft Learning to Rank Datasets](https://www.microsoft.com/en-us/research/project/mslr/) 3. [Gao J, He X, Deng L. Deep Learning for Web Search and Natural Language Processing[J]. Microsoft Research Technical Report, 2015.](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/wsdm2015.v3.pdf)