Merge branch 'develop' of github.com:PaddlePaddle/models into ctr2

.travis.yml

-group: deprecated-2017Q2
 language: cpp
 cache: ccache
 sudo: required

deep_speech_2/README.md

@@ -66,14 +66,69 @@ More help for arguments:
 python train.py --help
 ```
 
-### Inferencing
+### Preparing language model
+
+The following steps, inference, parameters tuning and evaluating, will require a language model during decoding.
+A compressed language model is provided and can be accessed by
+
+```
+cd ./lm
+sh run.sh
+cd ..
+```
+
+### Inference
+
+For GPU inference
 
 ```
 CUDA_VISIBLE_DEVICES=0 python infer.py
 ```
 
+For CPU inference
+
+```
+python infer.py --use_gpu=False
+```
+
 More help for arguments:
 
 ```
 python infer.py --help
 ```
+
+### Evaluating
+
+```
+CUDA_VISIBLE_DEVICES=0 python evaluate.py
+```
+
+More help for arguments:
+
+```
+python evaluate.py --help
+```
+
+### Parameters tuning
+
+Usually, the parameters $\alpha$ and $\beta$ for the CTC [prefix beam search](https://arxiv.org/abs/1408.2873) decoder need to be tuned after retraining the acoustic model.
+
+For GPU tuning
+
+```
+CUDA_VISIBLE_DEVICES=0 python tune.py
+```
+
+For CPU tuning
+
+```
+python tune.py --use_gpu=False
+```
+
+More help for arguments:
+
+```
+python tune.py --help
+```
+
+Then reset parameters with the tuning result before inference or evaluating.

deep_speech_2/decoder.py

-"""Contains various CTC decoder."""
+"""Contains various CTC decoders."""
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
-import numpy as np
 from itertools import groupby
+import numpy as np
+from math import log
+import multiprocessing
 
 
-def ctc_best_path_decode(probs_seq, vocabulary):
-    """Best path decoding, also called argmax decoding or greedy decoding.
+def ctc_best_path_decoder(probs_seq, vocabulary):
+    """Best path decoder, also called argmax decoder or greedy decoder.
     Path consisting of the most probable tokens are further post-processed to
     remove consecutive repetitions and all blanks.
 
@@ -36,24 +38,200 @@ def ctc_best_path_decode(probs_seq, vocabulary):
     return ''.join([vocabulary[index] for index in index_list])
 
 
-def ctc_decode(probs_seq, vocabulary, method):
-    """CTC-like sequence decoding from a sequence of likelihood probablilites.
+def ctc_beam_search_decoder(probs_seq,
+                            beam_size,
+                            vocabulary,
+                            blank_id,
+                            cutoff_prob=1.0,
+                            ext_scoring_func=None,
+                            nproc=False):
+    """Beam search decoder for CTC-trained network. It utilizes beam search
+    to approximately select top best decoding labels and returning results
+    in the descending order. The implementation is based on Prefix
+    Beam Search (https://arxiv.org/abs/1408.2873), and the unclear part is
+    redesigned. Two important modifications: 1) in the iterative computation
+    of probabilities, the assignment operation is changed to accumulation for
+    one prefix may comes from different paths; 2) the if condition "if l^+ not
+    in A_prev then" after probabilities' computation is deprecated for it is
+    hard to understand and seems unnecessary.
 
-    :param probs_seq: 2-D list of probabilities over the vocabulary for each
-                      character. Each element is a list of float probabilities
-                      for one character.
-    :type probs_seq: list
+    :param probs_seq: 2-D list of probability distributions over each time
+                      step, with each element being a list of normalized
+                      probabilities over vocabulary and blank.
+    :type probs_seq: 2-D list
+    :param beam_size: Width for beam search.
+    :type beam_size: int
     :param vocabulary: Vocabulary list.
     :type vocabulary: list
-    :param method: Decoding method name, with options: "best_path".
-    :type method: basestring
-    :return: Decoding result string.
-    :rtype: baseline
+    :param blank_id: ID of blank.
+    :type blank_id: int
+    :param cutoff_prob: Cutoff probability in pruning,
+                        default 1.0, no pruning.
+    :type cutoff_prob: float
+    :param ext_scoring_func: External scoring function for
+                            partially decoded sentence, e.g. word count
+                            or language model.
+    :type external_scoring_func: callable
+    :param nproc: Whether the decoder used in multiprocesses.
+    :type nproc: bool
+    :return: List of tuples of log probability and sentence as decoding
+             results, in descending order of the probability.
+    :rtype: list
     """
+    # dimension check
     for prob_list in probs_seq:
         if not len(prob_list) == len(vocabulary) + 1:
-            raise ValueError("probs dimension mismatchedd with vocabulary")
-    if method == "best_path":
-        return ctc_best_path_decode(probs_seq, vocabulary)
-    else:
-        raise ValueError("Decoding method [%s] is not supported.")
+            raise ValueError("The shape of prob_seq does not match with the "
+                             "shape of the vocabulary.")
+
+    # blank_id check
+    if not blank_id < len(probs_seq[0]):
+        raise ValueError("blank_id shouldn't be greater than probs dimension")
+
+    # If the decoder called in the multiprocesses, then use the global scorer
+    # instantiated in ctc_beam_search_decoder_batch().
+    if nproc is True:
+        global ext_nproc_scorer
+        ext_scoring_func = ext_nproc_scorer
+
+    ## initialize
+    # prefix_set_prev: the set containing selected prefixes
+    # probs_b_prev: prefixes' probability ending with blank in previous step
+    # probs_nb_prev: prefixes' probability ending with non-blank in previous step
+    prefix_set_prev = {'\t': 1.0}
+    probs_b_prev, probs_nb_prev = {'\t': 1.0}, {'\t': 0.0}
+
+    ## extend prefix in loop
+    for time_step in xrange(len(probs_seq)):
+        # prefix_set_next: the set containing candidate prefixes
+        # probs_b_cur: prefixes' probability ending with blank in current step
+        # probs_nb_cur: prefixes' probability ending with non-blank in current step
+        prefix_set_next, probs_b_cur, probs_nb_cur = {}, {}, {}
+
+        prob_idx = list(enumerate(probs_seq[time_step]))
+        cutoff_len = len(prob_idx)
+        #If pruning is enabled
+        if cutoff_prob < 1.0:
+            prob_idx = sorted(prob_idx, key=lambda asd: asd[1], reverse=True)
+            cutoff_len, cum_prob = 0, 0.0
+            for i in xrange(len(prob_idx)):
+                cum_prob += prob_idx[i][1]
+                cutoff_len += 1
+                if cum_prob >= cutoff_prob:
+                    break
+            prob_idx = prob_idx[0:cutoff_len]
+
+        for l in prefix_set_prev:
+            if not prefix_set_next.has_key(l):
+                probs_b_cur[l], probs_nb_cur[l] = 0.0, 0.0
+
+            # extend prefix by travering prob_idx
+            for index in xrange(cutoff_len):
+                c, prob_c = prob_idx[index][0], prob_idx[index][1]
+
+                if c == blank_id:
+                    probs_b_cur[l] += prob_c * (
+                        probs_b_prev[l] + probs_nb_prev[l])
+                else:
+                    last_char = l[-1]
+                    new_char = vocabulary[c]
+                    l_plus = l + new_char
+                    if not prefix_set_next.has_key(l_plus):
+                        probs_b_cur[l_plus], probs_nb_cur[l_plus] = 0.0, 0.0
+
+                    if new_char == last_char:
+                        probs_nb_cur[l_plus] += prob_c * probs_b_prev[l]
+                        probs_nb_cur[l] += prob_c * probs_nb_prev[l]
+                    elif new_char == ' ':
+                        if (ext_scoring_func is None) or (len(l) == 1):
+                            score = 1.0
+                        else:
+                            prefix = l[1:]
+                            score = ext_scoring_func(prefix)
+                        probs_nb_cur[l_plus] += score * prob_c * (
+                            probs_b_prev[l] + probs_nb_prev[l])
+                    else:
+                        probs_nb_cur[l_plus] += prob_c * (
+                            probs_b_prev[l] + probs_nb_prev[l])
+                    # add l_plus into prefix_set_next
+                    prefix_set_next[l_plus] = probs_nb_cur[
+                        l_plus] + probs_b_cur[l_plus]
+            # add l into prefix_set_next
+            prefix_set_next[l] = probs_b_cur[l] + probs_nb_cur[l]
+        # update probs
+        probs_b_prev, probs_nb_prev = probs_b_cur, probs_nb_cur
+
+        ## store top beam_size prefixes
+        prefix_set_prev = sorted(
+            prefix_set_next.iteritems(), key=lambda asd: asd[1], reverse=True)
+        if beam_size < len(prefix_set_prev):
+            prefix_set_prev = prefix_set_prev[:beam_size]
+        prefix_set_prev = dict(prefix_set_prev)
+
+    beam_result = []
+    for seq, prob in prefix_set_prev.items():
+        if prob > 0.0 and len(seq) > 1:
+            result = seq[1:]
+            # score last word by external scorer
+            if (ext_scoring_func is not None) and (result[-1] != ' '):
+                prob = prob * ext_scoring_func(result)
+            log_prob = log(prob)
+            beam_result.append((log_prob, result))
+
+    ## output top beam_size decoding results
+    beam_result = sorted(beam_result, key=lambda asd: asd[0], reverse=True)
+    return beam_result
+
+
+def ctc_beam_search_decoder_batch(probs_split,
+                                  beam_size,
+                                  vocabulary,
+                                  blank_id,
+                                  num_processes,
+                                  cutoff_prob=1.0,
+                                  ext_scoring_func=None):
+    """CTC beam search decoder using multiple processes.
+
+    :param probs_seq: 3-D list with each element as an instance of 2-D list
+                      of probabilities used by ctc_beam_search_decoder().
+    :type probs_seq: 3-D list
+    :param beam_size: Width for beam search.
+    :type beam_size: int
+    :param vocabulary: Vocabulary list.
+    :type vocabulary: list
+    :param blank_id: ID of blank.
+    :type blank_id: int
+    :param num_processes: Number of parallel processes.
+    :type num_processes: int
+    :param cutoff_prob: Cutoff probability in pruning,
+                        default 1.0, no pruning.
+    :param num_processes: Number of parallel processes.
+    :type num_processes: int
+    :type cutoff_prob: float
+    :param ext_scoring_func: External scoring function for
+                            partially decoded sentence, e.g. word count
+                            or language model.
+    :type external_scoring_function: callable
+    :return: List of tuples of log probability and sentence as decoding
+             results, in descending order of the probability.
+    :rtype: list
+    """
+    if not num_processes > 0:
+        raise ValueError("Number of processes must be positive!")
+
+    # use global variable to pass the externnal scorer to beam search decoder
+    global ext_nproc_scorer
+    ext_nproc_scorer = ext_scoring_func
+    nproc = True
+
+    pool = multiprocessing.Pool(processes=num_processes)
+    results = []
+    for i, probs_list in enumerate(probs_split):
+        args = (probs_list, beam_size, vocabulary, blank_id, cutoff_prob, None,
+                nproc)
+        results.append(pool.apply_async(ctc_beam_search_decoder, args))
+
+    pool.close()
+    pool.join()
+    beam_search_results = [result.get() for result in results]
+    return beam_search_results

deep_speech_2/evaluate.py

+"""Evaluation for DeepSpeech2 model."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import distutils.util
+import argparse
+import gzip
+import paddle.v2 as paddle
+from data_utils.data import DataGenerator
+from model import deep_speech2
+from decoder import *
+from lm.lm_scorer import LmScorer
+from error_rate import wer
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument(
+    "--batch_size",
+    default=100,
+    type=int,
+    help="Minibatch size for evaluation. (default: %(default)s)")
+parser.add_argument(
+    "--num_conv_layers",
+    default=2,
+    type=int,
+    help="Convolution layer number. (default: %(default)s)")
+parser.add_argument(
+    "--num_rnn_layers",
+    default=3,
+    type=int,
+    help="RNN layer number. (default: %(default)s)")
+parser.add_argument(
+    "--rnn_layer_size",
+    default=512,
+    type=int,
+    help="RNN layer cell number. (default: %(default)s)")
+parser.add_argument(
+    "--use_gpu",
+    default=True,
+    type=distutils.util.strtobool,
+    help="Use gpu or not. (default: %(default)s)")
+parser.add_argument(
+    "--num_threads_data",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu threads for preprocessing data. (default: %(default)s)")
+parser.add_argument(
+    "--num_processes_beam_search",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu processes for beam search. (default: %(default)s)")
+parser.add_argument(
+    "--mean_std_filepath",
+    default='mean_std.npz',
+    type=str,
+    help="Manifest path for normalizer. (default: %(default)s)")
+parser.add_argument(
+    "--decode_method",
+    default='beam_search',
+    type=str,
+    help="Method for ctc decoding, best_path or beam_search. (default: %(default)s)"
+)
+parser.add_argument(
+    "--language_model_path",
+    default="lm/data/common_crawl_00.prune01111.trie.klm",
+    type=str,
+    help="Path for language model. (default: %(default)s)")
+parser.add_argument(
+    "--alpha",
+    default=0.26,
+    type=float,
+    help="Parameter associated with language model. (default: %(default)f)")
+parser.add_argument(
+    "--beta",
+    default=0.1,
+    type=float,
+    help="Parameter associated with word count. (default: %(default)f)")
+parser.add_argument(
+    "--cutoff_prob",
+    default=0.99,
+    type=float,
+    help="The cutoff probability of pruning"
+    "in beam search. (default: %(default)f)")
+parser.add_argument(
+    "--beam_size",
+    default=500,
+    type=int,
+    help="Width for beam search decoding. (default: %(default)d)")
+parser.add_argument(
+    "--decode_manifest_path",
+    default='datasets/manifest.test',
+    type=str,
+    help="Manifest path for decoding. (default: %(default)s)")
+parser.add_argument(
+    "--model_filepath",
+    default='checkpoints/params.latest.tar.gz',
+    type=str,
+    help="Model filepath. (default: %(default)s)")
+parser.add_argument(
+    "--vocab_filepath",
+    default='datasets/vocab/eng_vocab.txt',
+    type=str,
+    help="Vocabulary filepath. (default: %(default)s)")
+args = parser.parse_args()
+
+
+def evaluate():
+    """Evaluate on whole test data for DeepSpeech2."""
+    # initialize data generator
+    data_generator = DataGenerator(
+        vocab_filepath=args.vocab_filepath,
+        mean_std_filepath=args.mean_std_filepath,
+        augmentation_config='{}',
+        num_threads=args.num_threads_data)
+
+    # create network config
+    # paddle.data_type.dense_array is used for variable batch input.
+    # The size 161 * 161 is only an placeholder value and the real shape
+    # of input batch data will be induced during training.
+    audio_data = paddle.layer.data(
+        name="audio_spectrogram", type=paddle.data_type.dense_array(161 * 161))
+    text_data = paddle.layer.data(
+        name="transcript_text",
+        type=paddle.data_type.integer_value_sequence(data_generator.vocab_size))
+    output_probs = deep_speech2(
+        audio_data=audio_data,
+        text_data=text_data,
+        dict_size=data_generator.vocab_size,
+        num_conv_layers=args.num_conv_layers,
+        num_rnn_layers=args.num_rnn_layers,
+        rnn_size=args.rnn_layer_size,
+        is_inference=True)
+
+    # load parameters
+    parameters = paddle.parameters.Parameters.from_tar(
+        gzip.open(args.model_filepath))
+
+    # prepare infer data
+    batch_reader = data_generator.batch_reader_creator(
+        manifest_path=args.decode_manifest_path,
+        batch_size=args.batch_size,
+        min_batch_size=1,
+        sortagrad=False,
+        shuffle_method=None)
+
+    # define inferer
+    inferer = paddle.inference.Inference(
+        output_layer=output_probs, parameters=parameters)
+
+    # initialize external scorer for beam search decoding
+    if args.decode_method == 'beam_search':
+        ext_scorer = LmScorer(args.alpha, args.beta, args.language_model_path)
+
+    wer_counter, wer_sum = 0, 0.0
+    for infer_data in batch_reader():
+        # run inference
+        infer_results = inferer.infer(input=infer_data)
+        num_steps = len(infer_results) // len(infer_data)
+        probs_split = [
+            infer_results[i * num_steps:(i + 1) * num_steps]
+            for i in xrange(0, len(infer_data))
+        ]
+        # target transcription
+        target_transcription = [
+            ''.join([
+                data_generator.vocab_list[index] for index in infer_data[i][1]
+            ]) for i, probs in enumerate(probs_split)
+        ]
+        # decode and print
+        # best path decode
+        if args.decode_method == "best_path":
+            for i, probs in enumerate(probs_split):
+                output_transcription = ctc_best_path_decoder(
+                    probs_seq=probs, vocabulary=data_generator.vocab_list)
+                wer_sum += wer(target_transcription[i], output_transcription)
+                wer_counter += 1
+        # beam search decode
+        elif args.decode_method == "beam_search":
+            # beam search using multiple processes
+            beam_search_results = ctc_beam_search_decoder_batch(
+                probs_split=probs_split,
+                vocabulary=data_generator.vocab_list,
+                beam_size=args.beam_size,
+                blank_id=len(data_generator.vocab_list),
+                num_processes=args.num_processes_beam_search,
+                ext_scoring_func=ext_scorer,
+                cutoff_prob=args.cutoff_prob, )
+            for i, beam_search_result in enumerate(beam_search_results):
+                wer_sum += wer(target_transcription[i],
+                               beam_search_result[0][1])
+                wer_counter += 1
+        else:
+            raise ValueError("Decoding method [%s] is not supported." %
+                             decode_method)
+
+    print("Final WER = %f" % (wer_sum / wer_counter))
+
+
+def main():
+    paddle.init(use_gpu=args.use_gpu, trainer_count=1)
+    evaluate()
+
+
+if __name__ == '__main__':
+    main()

deep_speech_2/infer.py

@@ -10,7 +10,9 @@ import multiprocessing
 import paddle.v2 as paddle
 from data_utils.data import DataGenerator
 from model import deep_speech2
-from decoder import ctc_decode
+from decoder import *
+from lm.lm_scorer import LmScorer
+from error_rate import wer
 import utils
 
 parser = argparse.ArgumentParser(description=__doc__)
@@ -44,6 +46,11 @@ parser.add_argument(
     default=multiprocessing.cpu_count(),
     type=int,
     help="Number of cpu threads for preprocessing data. (default: %(default)s)")
+parser.add_argument(
+    "--num_processes_beam_search",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu processes for beam search. (default: %(default)s)")
 parser.add_argument(
     "--mean_std_filepath",
     default='mean_std.npz',
@@ -64,11 +71,48 @@ parser.add_argument(
     default='datasets/vocab/eng_vocab.txt',
     type=str,
     help="Vocabulary filepath. (default: %(default)s)")
+parser.add_argument(
+    "--decode_method",
+    default='beam_search',
+    type=str,
+    help="Method for ctc decoding: best_path or beam_search. (default: %(default)s)"
+)
+parser.add_argument(
+    "--beam_size",
+    default=500,
+    type=int,
+    help="Width for beam search decoding. (default: %(default)d)")
+parser.add_argument(
+    "--num_results_per_sample",
+    default=1,
+    type=int,
+    help="Number of output per sample in beam search. (default: %(default)d)")
+parser.add_argument(
+    "--language_model_path",
+    default="lm/data/common_crawl_00.prune01111.trie.klm",
+    type=str,
+    help="Path for language model. (default: %(default)s)")
+parser.add_argument(
+    "--alpha",
+    default=0.26,
+    type=float,
+    help="Parameter associated with language model. (default: %(default)f)")
+parser.add_argument(
+    "--beta",
+    default=0.1,
+    type=float,
+    help="Parameter associated with word count. (default: %(default)f)")
+parser.add_argument(
+    "--cutoff_prob",
+    default=0.99,
+    type=float,
+    help="The cutoff probability of pruning"
+    "in beam search. (default: %(default)f)")
 args = parser.parse_args()
 
 
 def infer():
-    """Max-ctc-decoding for DeepSpeech2."""
+    """Inference for DeepSpeech2."""
     # initialize data generator
     data_generator = DataGenerator(
         vocab_filepath=args.vocab_filepath,
@@ -102,6 +146,7 @@ def infer():
     batch_reader = data_generator.batch_reader_creator(
         manifest_path=args.decode_manifest_path,
         batch_size=args.num_samples,
+        min_batch_size=1,
         sortagrad=False,
         shuffle_method=None)
     infer_data = batch_reader().next()
@@ -115,16 +160,52 @@ def infer():
         for i in xrange(len(infer_data))
     ]
 
-    # decode and print
-    for i, probs in enumerate(probs_split):
-        output_transcription = ctc_decode(
-            probs_seq=probs,
-            vocabulary=data_generator.vocab_list,
-            method="best_path")
-        target_transcription = ''.join(
+    # targe transcription
+    target_transcription = [
+        ''.join(
             [data_generator.vocab_list[index] for index in infer_data[i][1]])
-        print("Target Transcription: %s \nOutput Transcription: %s \n" %
-              (target_transcription, output_transcription))
+        for i, probs in enumerate(probs_split)
+    ]
+
+    ## decode and print
+    # best path decode
+    wer_sum, wer_counter = 0, 0
+    if args.decode_method == "best_path":
+        for i, probs in enumerate(probs_split):
+            best_path_transcription = ctc_best_path_decoder(
+                probs_seq=probs, vocabulary=data_generator.vocab_list)
+            print("\nTarget Transcription: %s\nOutput Transcription: %s" %
+                  (target_transcription[i], best_path_transcription))
+            wer_cur = wer(target_transcription[i], best_path_transcription)
+            wer_sum += wer_cur
+            wer_counter += 1
+            print("cur wer = %f, average wer = %f" %
+                  (wer_cur, wer_sum / wer_counter))
+    # beam search decode
+    elif args.decode_method == "beam_search":
+        ext_scorer = LmScorer(args.alpha, args.beta, args.language_model_path)
+        beam_search_batch_results = ctc_beam_search_decoder_batch(
+            probs_split=probs_split,
+            vocabulary=data_generator.vocab_list,
+            beam_size=args.beam_size,
+            blank_id=len(data_generator.vocab_list),
+            num_processes=args.num_processes_beam_search,
+            cutoff_prob=args.cutoff_prob,
+            ext_scoring_func=ext_scorer, )
+        for i, beam_search_result in enumerate(beam_search_batch_results):
+            print("\nTarget Transcription:\t%s" % target_transcription[i])
+            for index in xrange(args.num_results_per_sample):
+                result = beam_search_result[index]
+                #output: index, log prob, beam result
+                print("Beam %d: %f \t%s" % (index, result[0], result[1]))
+            wer_cur = wer(target_transcription[i], beam_search_result[0][1])
+            wer_sum += wer_cur
+            wer_counter += 1
+            print("cur wer = %f , average wer = %f" %
+                  (wer_cur, wer_sum / wer_counter))
+    else:
+        raise ValueError("Decoding method [%s] is not supported." %
+                         decode_method)
 
 
 def main():

deep_speech_2/lm/lm_scorer.py

+"""External Scorer for Beam Search Decoder."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import kenlm
+import numpy as np
+
+
+class LmScorer(object):
+    """External scorer to evaluate a prefix or whole sentence in
+       beam search decoding, including the score from n-gram language
+       model and word count.
+
+    :param alpha: Parameter associated with language model. Don't use
+                  language model when alpha = 0.
+    :type alpha: float
+    :param beta: Parameter associated with word count. Don't use word
+                count when beta = 0.
+    :type beta: float
+    :model_path: Path to load language model.
+    :type model_path: basestring
+    """
+
+    def __init__(self, alpha, beta, model_path):
+        self._alpha = alpha
+        self._beta = beta
+        if not os.path.isfile(model_path):
+            raise IOError("Invaid language model path: %s" % model_path)
+        self._language_model = kenlm.LanguageModel(model_path)
+
+    # n-gram language model scoring
+    def _language_model_score(self, sentence):
+        #log10 prob of last word
+        log_cond_prob = list(
+            self._language_model.full_scores(sentence, eos=False))[-1][0]
+        return np.power(10, log_cond_prob)
+
+    # word insertion term
+    def _word_count(self, sentence):
+        words = sentence.strip().split(' ')
+        return len(words)
+
+    # reset alpha and beta
+    def reset_params(self, alpha, beta):
+        self._alpha = alpha
+        self._beta = beta
+
+    # execute evaluation
+    def __call__(self, sentence, log=False):
+        """Evaluation function, gathering all the different scores
+        and return the final one.
+
+        :param sentence: The input sentence for evalutation
+        :type sentence: basestring
+        :param log: Whether return the score in log representation.
+        :type log: bool
+        :return: Evaluation score, in the decimal or log.
+        :rtype: float
+        """
+        lm = self._language_model_score(sentence)
+        word_cnt = self._word_count(sentence)
+        if log == False:
+            score = np.power(lm, self._alpha) * np.power(word_cnt, self._beta)
+        else:
+            score = self._alpha * np.log(lm) + self._beta * np.log(word_cnt)
+        return score

deep_speech_2/lm/run.sh

+echo "Downloading language model ..."
+
+mkdir data
+
+LM=common_crawl_00.prune01111.trie.klm
+MD5="099a601759d467cd0a8523ff939819c5"
+
+wget -c http://paddlepaddle.bj.bcebos.com/model_zoo/speech/$LM -P ./data
+
+echo "Checking md5sum ..."
+md5_tmp=`md5sum ./data/$LM | awk -F[' '] '{print $1}'`
+
+if [ $MD5 != $md5_tmp ]; then
+    echo "Fail to download the language model!"
+    exit 1
+fi
+
+
+

deep_speech_2/requirements.txt

 wget==3.2
 scipy==0.13.1
-resampy==0.1.5
\ No newline at end of file
+resampy==0.1.5
+https://github.com/kpu/kenlm/archive/master.zip

deep_speech_2/tests/test_decoders.py

+"""Test decoders."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import unittest
+from decoder import *
+
+
+class TestDecoders(unittest.TestCase):
+    def setUp(self):
+        self.vocab_list = ["\'", ' ', 'a', 'b', 'c', 'd']
+        self.beam_size = 20
+        self.probs_seq1 = [[
+            0.06390443, 0.21124858, 0.27323887, 0.06870235, 0.0361254,
+            0.18184413, 0.16493624
+        ], [
+            0.03309247, 0.22866108, 0.24390638, 0.09699597, 0.31895462,
+            0.0094893, 0.06890021
+        ], [
+            0.218104, 0.19992557, 0.18245131, 0.08503348, 0.14903535,
+            0.08424043, 0.08120984
+        ], [
+            0.12094152, 0.19162472, 0.01473646, 0.28045061, 0.24246305,
+            0.05206269, 0.09772094
+        ], [
+            0.1333387, 0.00550838, 0.00301669, 0.21745861, 0.20803985,
+            0.41317442, 0.01946335
+        ], [
+            0.16468227, 0.1980699, 0.1906545, 0.18963251, 0.19860937,
+            0.04377724, 0.01457421
+        ]]
+        self.probs_seq2 = [[
+            0.08034842, 0.22671944, 0.05799633, 0.36814645, 0.11307441,
+            0.04468023, 0.10903471
+        ], [
+            0.09742457, 0.12959763, 0.09435383, 0.21889204, 0.15113123,
+            0.10219457, 0.20640612
+        ], [
+            0.45033529, 0.09091417, 0.15333208, 0.07939558, 0.08649316,
+            0.12298585, 0.01654384
+        ], [
+            0.02512238, 0.22079203, 0.19664364, 0.11906379, 0.07816055,
+            0.22538587, 0.13483174
+        ], [
+            0.17928453, 0.06065261, 0.41153005, 0.1172041, 0.11880313,
+            0.07113197, 0.04139363
+        ], [
+            0.15882358, 0.1235788, 0.23376776, 0.20510435, 0.00279306,
+            0.05294827, 0.22298418
+        ]]
+        self.best_path_result = ["ac'bdc", "b'da"]
+        self.beam_search_result = ['acdc', "b'a"]
+
+    def test_best_path_decoder_1(self):
+        bst_result = ctc_best_path_decoder(self.probs_seq1, self.vocab_list)
+        self.assertEqual(bst_result, self.best_path_result[0])
+
+    def test_best_path_decoder_2(self):
+        bst_result = ctc_best_path_decoder(self.probs_seq2, self.vocab_list)
+        self.assertEqual(bst_result, self.best_path_result[1])
+
+    def test_beam_search_decoder_1(self):
+        beam_result = ctc_beam_search_decoder(
+            probs_seq=self.probs_seq1,
+            beam_size=self.beam_size,
+            vocabulary=self.vocab_list,
+            blank_id=len(self.vocab_list))
+        self.assertEqual(beam_result[0][1], self.beam_search_result[0])
+
+    def test_beam_search_decoder_2(self):
+        beam_result = ctc_beam_search_decoder(
+            probs_seq=self.probs_seq2,
+            beam_size=self.beam_size,
+            vocabulary=self.vocab_list,
+            blank_id=len(self.vocab_list))
+        self.assertEqual(beam_result[0][1], self.beam_search_result[1])
+
+    def test_beam_search_decoder_batch(self):
+        beam_results = ctc_beam_search_decoder_batch(
+            probs_split=[self.probs_seq1, self.probs_seq2],
+            beam_size=self.beam_size,
+            vocabulary=self.vocab_list,
+            blank_id=len(self.vocab_list),
+            num_processes=24)
+        self.assertEqual(beam_results[0][0][1], self.beam_search_result[0])
+        self.assertEqual(beam_results[1][0][1], self.beam_search_result[1])
+
+
+if __name__ == '__main__':
+    unittest.main()

deep_speech_2/tune.py

+"""Parameters tuning for DeepSpeech2 model."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import distutils.util
+import argparse
+import gzip
+import paddle.v2 as paddle
+from data_utils.data import DataGenerator
+from model import deep_speech2
+from decoder import *
+from lm.lm_scorer import LmScorer
+from error_rate import wer
+import utils
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument(
+    "--num_samples",
+    default=100,
+    type=int,
+    help="Number of samples for parameters tuning. (default: %(default)s)")
+parser.add_argument(
+    "--num_conv_layers",
+    default=2,
+    type=int,
+    help="Convolution layer number. (default: %(default)s)")
+parser.add_argument(
+    "--num_rnn_layers",
+    default=3,
+    type=int,
+    help="RNN layer number. (default: %(default)s)")
+parser.add_argument(
+    "--rnn_layer_size",
+    default=512,
+    type=int,
+    help="RNN layer cell number. (default: %(default)s)")
+parser.add_argument(
+    "--use_gpu",
+    default=True,
+    type=distutils.util.strtobool,
+    help="Use gpu or not. (default: %(default)s)")
+parser.add_argument(
+    "--num_threads_data",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu threads for preprocessing data. (default: %(default)s)")
+parser.add_argument(
+    "--num_processes_beam_search",
+    default=multiprocessing.cpu_count(),
+    type=int,
+    help="Number of cpu processes for beam search. (default: %(default)s)")
+parser.add_argument(
+    "--mean_std_filepath",
+    default='mean_std.npz',
+    type=str,
+    help="Manifest path for normalizer. (default: %(default)s)")
+parser.add_argument(
+    "--decode_manifest_path",
+    default='datasets/manifest.test',
+    type=str,
+    help="Manifest path for decoding. (default: %(default)s)")
+parser.add_argument(
+    "--model_filepath",
+    default='checkpoints/params.latest.tar.gz',
+    type=str,
+    help="Model filepath. (default: %(default)s)")
+parser.add_argument(
+    "--vocab_filepath",
+    default='datasets/vocab/eng_vocab.txt',
+    type=str,
+    help="Vocabulary filepath. (default: %(default)s)")
+parser.add_argument(
+    "--beam_size",
+    default=500,
+    type=int,
+    help="Width for beam search decoding. (default: %(default)d)")
+parser.add_argument(
+    "--language_model_path",
+    default="lm/data/common_crawl_00.prune01111.trie.klm",
+    type=str,
+    help="Path for language model. (default: %(default)s)")
+parser.add_argument(
+    "--alpha_from",
+    default=0.1,
+    type=float,
+    help="Where alpha starts from. (default: %(default)f)")
+parser.add_argument(
+    "--num_alphas",
+    default=14,
+    type=int,
+    help="Number of candidate alphas. (default: %(default)d)")
+parser.add_argument(
+    "--alpha_to",
+    default=0.36,
+    type=float,
+    help="Where alpha ends with. (default: %(default)f)")
+parser.add_argument(
+    "--beta_from",
+    default=0.05,
+    type=float,
+    help="Where beta starts from. (default: %(default)f)")
+parser.add_argument(
+    "--num_betas",
+    default=20,
+    type=float,
+    help="Number of candidate betas. (default: %(default)d)")
+parser.add_argument(
+    "--beta_to",
+    default=1.0,
+    type=float,
+    help="Where beta ends with. (default: %(default)f)")
+parser.add_argument(
+    "--cutoff_prob",
+    default=0.99,
+    type=float,
+    help="The cutoff probability of pruning"
+    "in beam search. (default: %(default)f)")
+args = parser.parse_args()
+
+
+def tune():
+    """Tune parameters alpha and beta on one minibatch."""
+
+    if not args.num_alphas >= 0:
+        raise ValueError("num_alphas must be non-negative!")
+
+    if not args.num_betas >= 0:
+        raise ValueError("num_betas must be non-negative!")
+
+    # initialize data generator
+    data_generator = DataGenerator(
+        vocab_filepath=args.vocab_filepath,
+        mean_std_filepath=args.mean_std_filepath,
+        augmentation_config='{}',
+        num_threads=args.num_threads_data)
+
+    # create network config
+    # paddle.data_type.dense_array is used for variable batch input.
+    # The size 161 * 161 is only an placeholder value and the real shape
+    # of input batch data will be induced during training.
+    audio_data = paddle.layer.data(
+        name="audio_spectrogram", type=paddle.data_type.dense_array(161 * 161))
+    text_data = paddle.layer.data(
+        name="transcript_text",
+        type=paddle.data_type.integer_value_sequence(data_generator.vocab_size))
+    output_probs = deep_speech2(
+        audio_data=audio_data,
+        text_data=text_data,
+        dict_size=data_generator.vocab_size,
+        num_conv_layers=args.num_conv_layers,
+        num_rnn_layers=args.num_rnn_layers,
+        rnn_size=args.rnn_layer_size,
+        is_inference=True)
+
+    # load parameters
+    parameters = paddle.parameters.Parameters.from_tar(
+        gzip.open(args.model_filepath))
+
+    # prepare infer data
+    batch_reader = data_generator.batch_reader_creator(
+        manifest_path=args.decode_manifest_path,
+        batch_size=args.num_samples,
+        sortagrad=False,
+        shuffle_method=None)
+    # get one batch data for tuning
+    infer_data = batch_reader().next()
+
+    # run inference
+    infer_results = paddle.infer(
+        output_layer=output_probs, parameters=parameters, input=infer_data)
+    num_steps = len(infer_results) // len(infer_data)
+    probs_split = [
+        infer_results[i * num_steps:(i + 1) * num_steps]
+        for i in xrange(0, len(infer_data))
+    ]
+
+    # create grid for search
+    cand_alphas = np.linspace(args.alpha_from, args.alpha_to, args.num_alphas)
+    cand_betas = np.linspace(args.beta_from, args.beta_to, args.num_betas)
+    params_grid = [(alpha, beta) for alpha in cand_alphas
+                   for beta in cand_betas]
+
+    ext_scorer = LmScorer(args.alpha_from, args.beta_from,
+                          args.language_model_path)
+    ## tune parameters in loop
+    for alpha, beta in params_grid:
+        wer_sum, wer_counter = 0, 0
+        # reset scorer
+        ext_scorer.reset_params(alpha, beta)
+        # beam search using multiple processes
+        beam_search_results = ctc_beam_search_decoder_batch(
+            probs_split=probs_split,
+            vocabulary=data_generator.vocab_list,
+            beam_size=args.beam_size,
+            cutoff_prob=args.cutoff_prob,
+            blank_id=len(data_generator.vocab_list),
+            num_processes=args.num_processes_beam_search,
+            ext_scoring_func=ext_scorer, )
+        for i, beam_search_result in enumerate(beam_search_results):
+            target_transcription = ''.join([
+                data_generator.vocab_list[index] for index in infer_data[i][1]
+            ])
+            wer_sum += wer(target_transcription, beam_search_result[0][1])
+            wer_counter += 1
+
+        print("alpha = %f\tbeta = %f\tWER = %f" %
+              (alpha, beta, wer_sum / wer_counter))
+
+
+def main():
+    paddle.init(use_gpu=args.use_gpu, trainer_count=1)
+    tune()
+
+
+if __name__ == '__main__':
+    main()

dssm/README.md

+# 深度结构化语义模型 (Deep Structured Semantic Models, DSSM)
+DSSM使用DNN模型在一个连续的语义空间中学习文本低纬的表示向量，并且建模两个句子间的语义相似度。
+本例演示如何使用 PaddlePaddle实现一个通用的DSSM 模型，用于建模两个字符串间的语义相似度，
+模型实现支持通用的数据格式，用户替换数据便可以在真实场景中使用该模型。
+
+## 背景介绍
+DSSM \[[1](##参考文献)\]是微软研究院13年提出来的经典的语义模型，用于学习两个文本之间的语义距离，
+广义上模型也可以推广和适用如下场景：
+
+1. CTR预估模型，衡量用户搜索词（Query）与候选网页集合（Documents）之间的相关联程度。
+2. 文本相关性，衡量两个字符串间的语义相关程度。
+3. 自动推荐，衡量User与被推荐的Item之间的关联程度。
+
+DSSM 已经发展成了一个框架，可以很自然地建模两个记录之间的距离关系，
+例如对于文本相关性问题，可以用余弦相似度 (cosin similarity) 来刻画语义距离；
+而对于搜索引擎的结果排序，可以在DSSM上接上Rank损失训练处一个排序模型。
+
+## 模型简介
+在原论文\[[1](#参考文献)\]中，DSSM模型用来衡量用户搜索词 Query 和文档集合 Documents 之间隐含的语义关系，模型结构如下
+
+<p align="center">
+<a id="#figure1"></a>
+<img src="./images/dssm.png"/><br/><br/>
+图 1. DSSM 原始结构
+<p>
+
+其贯彻的思想是， **用DNN将高维特征向量转化为低纬空间的连续向量（图中红色框部分）** ，
+**在上层用cosin similarity来衡量用户搜索词与候选文档间的语义相关性**。
+
+在最顶层损失函数的设计上，原始模型使用类似Word2Vec中负例采样的方法，
+一个Query会抽取正例 $D+$ 和4个负例 $D-$ 整体上算条件概率用对数似然函数作为损失，
+这也就是图 1中类似 $P(D_1|Q)$ 的结构，具体细节请参考原论文。
+
+随着后续优化DSSM模型的结构得以简化\[[3](#参考文献)\]，演变为：
+
+<p align="center">
+<img src="./images/dssm2.png" width="600"/><br/><br/>
+图 2. DSSM通用结构
+<p>
+
+图中的空白方框可以用任何模型替代，比如全连接FC，卷积CNN，RNN等都可以，
+该模型结构专门用于衡量两个元素（比如字符串）间的语义距离。
+
+在现实使用中，DSSM模型会作为基础的积木，搭配上不同的损失函数来实现具体的功能，比如
+
+- 在排序学习中，将 图 2 中结构添加 pairwise rank损失，变成一个排序模型
+- 在CTR预估中，对点击与否做0，1二元分类，添加交叉熵损失变成一个分类模型
+- 在需要对一个子串打分时，可以使用余弦相似度来计算相似度，变成一个回归模型
+
+本例将尝试面向应用提供一个比较通用的解决方案，在模型任务类型上支持
+
+- 分类
+- [-1, 1] 值域内的回归
+- Pairwise-Rank
+
+在生成低纬语义向量的模型结构上，本模型支持以下三种：
+
+- FC, 多层全连接层
+- CNN，卷积神经网络
+- RNN，递归神经网络
+
+## 模型实现
+DSSM模型可以拆成三小块实现，分别是左边和右边的DNN，以及顶层的损失函数。
+在复杂任务中，左右两边DNN的结构可以是不同的，比如在原始论文中左右分别学习Query和Document的semantic vector，
+两者数据的数据不同，建议对应定制DNN的结构。
+
+本例中为了简便和通用，将左右两个DNN的结构都设为相同的，因此只有三个选项FC,CNN,RNN等。
+
+在损失函数的设计方面，也支持三种，分类, 回归, 排序；
+其中，在回归和排序两种损失中，左右两边的匹配程度通过余弦相似度（cossim）来计算；
+在分类任务中，类别预测的分布通过softmax计算。
+
+在其它教程中，对上述很多内容都有过详细的介绍，例如：
+
+- 如何CNN, FC 做文本信息提取可以参考 [text classification](https://github.com/PaddlePaddle/models/blob/develop/text_classification/README.md#模型详解)
+- RNN/GRU 的内容可以参考 [Machine Translation](https://github.com/PaddlePaddle/book/blob/develop/08.machine_translation/README.md#gated-recurrent-unit-gru)
+- Pairwise Rank即排序学习可参考 [learn to rank](https://github.com/PaddlePaddle/models/blob/develop/ltr/README.md)
+
+相关原理在此不再赘述，本文接下来的篇幅主要集中介绍使用PaddlePaddle实现这些结构上。
+
+如图3，回归和分类模型的结构很相似
+
+<p align="center">
+<img src="./images/dssm3.jpg"/><br/><br/>
+图 3. DSSM for REGRESSION or CLASSIFICATION
+</p>
+
+最重要的组成部分包括词向量，图中`(1)`,`(2)`两个低纬向量的学习器（可以用RNN/CNN/FC中的任意一种实现），
+最上层对应的损失函数。
+
+而Pairwise Rank的结构会复杂一些，类似两个 图 4. 中的结构，增加了对应的损失函数：
+
+- 模型总体思想是，用同一个source(源)为左右两个target(目标)分别打分——`(a),(b)`，学习目标是(a),(b)间的大小关系
+- `(a)`和`(b)`类似图3中结构，用于给source和target的pair打分
+- `(1)`和`(2)`的结构其实是共用的，都表示同一个source，图中为了表达效果展开成两个
+
+<p align="center">
+<img src="./images/dssm2.jpg"/><br/><br/>
+图 4. DSSM for Pairwise Rank
+</p>
+
+下面是各个部分具体的实现方法，所有的代码均包含在 `./network_conf.py` 中。
+
+### 创建文本的词向量表
+```python
+def create_embedding(self, input, prefix=''):
+    '''
+    Create an embedding table whose name has a `prefix`.
+    '''
+    logger.info("create embedding table [%s] which dimention is %d" %
+                (prefix, self.dnn_dims[0]))
+    emb = paddle.layer.embedding(
+        input=input,
+        size=self.dnn_dims[0],
+        param_attr=ParamAttr(name='%s_emb.w' % prefix))
+    return emb
+```
+
+由于输入给词向量表(embedding table)的是一个句子对应的词的ID的列表 ，因此词向量表输出的是词向量的序列。
+
+### CNN 结构实现
+```python
+def create_cnn(self, emb, prefix=''):
+    '''
+    A multi-layer CNN.
+
+    @emb: paddle.layer
+        output of the embedding layer
+    @prefix: str
+        prefix of layers' names, used to share parameters between more than one `cnn` parts.
+    '''
+    def create_conv(context_len, hidden_size, prefix):
+        key = "%s_%d_%d" % (prefix, context_len, hidden_size)
+        conv = paddle.networks.sequence_conv_pool(
+            input=emb,
+            context_len=context_len,
+            hidden_size=hidden_size,
+            # set parameter attr for parameter sharing
+            context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
+            fc_param_attr=ParamAttr(name=key + '_fc.w'),
+            fc_bias_attr=ParamAttr(name=key + '_fc.b'),
+            pool_bias_attr=ParamAttr(name=key + '_pool.b'))
+        return conv
+
+    logger.info('create a sequence_conv_pool which context width is 3')
+    conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
+    logger.info('create a sequence_conv_pool which context width is 4')
+    conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
+    return conv_3, conv_4
+```
+
+CNN 接受 embedding table输出的词向量序列，通过卷积和池化操作捕捉到原始句子的关键信息，
+最终输出一个语义向量（可以认为是句子向量）。
+
+本例的实现中，分别使用了窗口长度为3和4的CNN学到的句子向量按元素求和得到最终的句子向量。
+### RNN 结构实现
+
+RNN很适合学习变长序列的信息，使用RNN来学习句子的信息几乎是自然语言处理任务的标配。
+
+```python
+def create_rnn(self, emb, prefix=''):
+    '''
+    A GRU sentence vector learner.
+    '''
+    gru = paddle.layer.gru_memory(input=emb,)
+    sent_vec = paddle.layer.last_seq(gru)
+    return sent_vec
+```
+### FC 结构实现
+```python
+def create_fc(self, emb, prefix=''):
+    '''
+    A multi-layer fully connected neural networks.
+
+    @emb: paddle.layer
+        output of the embedding layer
+    @prefix: str
+        prefix of layers' names, used to share parameters between more than one `fc` parts.
+    '''
+    _input_layer = paddle.layer.pooling(
+        input=emb, pooling_type=paddle.pooling.Max())
+    fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
+    return fc
+```
+
+在构建FC时需要首先使用`paddle.layer.pooling` 对词向量序列进行最大池化操作，将边长序列转化为一个固定维度向量，
+作为整个句子的语义表达，使用最大池化能够降低句子长度对句向量表达的影响。
+
+### 多层DNN实现
+在 CNN/DNN/FC提取出 semantic vector后，在上层可继续接多层FC来实现深层DNN结构。
+
+```python
+def create_dnn(self, sent_vec, prefix):
+    # if more than three layers exists, a fc layer will be added.
+    if len(self.dnn_dims) > 1:
+        _input_layer = sent_vec
+        for id, dim in enumerate(self.dnn_dims[1:]):
+            name = "%s_fc_%d_%d" % (prefix, id, dim)
+            logger.info("create fc layer [%s] which dimention is %d" %
+                        (name, dim))
+            fc = paddle.layer.fc(
+                input=_input_layer,
+                size=dim,
+                name=name,
+                act=paddle.activation.Tanh(),
+                param_attr=ParamAttr(name='%s.w' % name),
+                bias_attr=ParamAttr(name='%s.b' % name),
+                )
+            _input_layer = fc
+    return _input_layer
+```
+
+### 分类或回归实现
+分类和回归的结构比较相似，因此可以用一个函数创建出来
+
+```python
+def _build_classification_or_regression_model(self, is_classification):
+    '''
+    Build a classification/regression model, and the cost is returned.
+
+    A Classification has 3 inputs:
+      - source sentence
+      - target sentence
+      - classification label
+
+    '''
+    # prepare inputs.
+    assert self.class_num
+
+    source = paddle.layer.data(
+        name='source_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+    target = paddle.layer.data(
+        name='target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    label = paddle.layer.data(
+        name='label_input',
+        type=paddle.data_type.integer_value(self.class_num)
+        if is_classification else paddle.data_type.dense_input)
+
+    prefixs = '_ _'.split(
+    ) if self.share_semantic_generator else 'left right'.split()
+    embed_prefixs = '_ _'.split(
+    ) if self.share_embed else 'left right'.split()
+
+    word_vecs = []
+    for id, input in enumerate([source, target]):
+        x = self.create_embedding(input, prefix=embed_prefixs[id])
+        word_vecs.append(x)
+
+    semantics = []
+    for id, input in enumerate(word_vecs):
+        x = self.model_arch_creater(input, prefix=prefixs[id])
+        semantics.append(x)
+
+    concated_vector = paddle.layer.concat(semantics)
+    prediction = paddle.layer.fc(
+        input=concated_vector,
+        size=self.class_num,
+        act=paddle.activation.Softmax())
+    cost = paddle.layer.classification_cost(
+        input=prediction,
+        label=label) if is_classification else paddle.layer.mse_cost(
+            prediction, label)
+    return cost, prediction, label
+```
+### Pairwise Rank实现
+Pairwise Rank复用上面的DNN结构，同一个source对两个target求相似度打分，
+如果左边的target打分高，预测为1，否则预测为 0。
+
+```python
+def _build_rank_model(self):
+    '''
+    Build a pairwise rank model, and the cost is returned.
+
+    A pairwise rank model has 3 inputs:
+      - source sentence
+      - left_target sentence
+      - right_target sentence
+      - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
+    '''
+    source = paddle.layer.data(
+        name='source_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+    left_target = paddle.layer.data(
+        name='left_target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    right_target = paddle.layer.data(
+        name='right_target_input',
+        type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+    label = paddle.layer.data(
+        name='label_input', type=paddle.data_type.integer_value(1))
+
+    prefixs = '_ _ _'.split(
+    ) if self.share_semantic_generator else 'source left right'.split()
+    embed_prefixs = '_ _'.split(
+    ) if self.share_embed else 'source target target'.split()
+
+    word_vecs = []
+    for id, input in enumerate([source, left_target, right_target]):
+        x = self.create_embedding(input, prefix=embed_prefixs[id])
+        word_vecs.append(x)
+
+    semantics = []
+    for id, input in enumerate(word_vecs):
+        x = self.model_arch_creater(input, prefix=prefixs[id])
+        semantics.append(x)
+
+    # cossim score of source and left_target
+    left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
+    # cossim score of source and right target
+    right_score = paddle.layer.cos_sim(semantics[0], semantics[2])
+
+    # rank cost
+    cost = paddle.layer.rank_cost(left_score, right_score, label=label)
+    # prediction = left_score - right_score
+    # but this operator is not supported currently.
+    # so AUC will not used.
+    return cost, None, None
+```
+## 数据格式
+在 `./data` 中有简单的示例数据
+
+### 回归的数据格式
+```
+# 3 fields each line:
+#   - source's word ids
+#   - target's word ids
+#   - target
+<ids> \t <ids> \t <float>
+```
+
+比如：
+
+```
+3 6 10 \t 6 8 33 \t 0.7
+6 0 \t 6 9 330 \t 0.03
+```
+### 分类的数据格式
+```
+# 3 fields each line:
+#   - source's word ids
+#   - target's word ids
+#   - target
+<ids> \t <ids> \t <label>
+```
+
+比如：
+
+```
+3 6 10 \t 6 8 33 \t 0
+6 10 \t 8 3 1 \t 1
+```
+
+
+### 排序的数据格式
+```
+# 4 fields each line:
+#   - source's word ids
+#   - target1's word ids
+#   - target2's word ids
+#   - label
+<ids> \t <ids> \t <ids> \t <label>
+```
+
+比如：
+
+```
+7 2 4 \t 2 10 12 \t 9 2 7 10 23 \t 0
+7 2 4 \t 10 12 \t 9 2 21 23 \t 1
+```
+
+## 执行训练
+
+可以直接执行 `python train.py -y 0 --model_arch 0` 使用 `./data/classification` 目录里简单的数据来训练一个分类的FC模型。
+
+其他模型结构也可以通过命令行实现定制，详细命令行参数如下
+
+```
+usage: train.py [-h] [-i TRAIN_DATA_PATH] [-t TEST_DATA_PATH]
+                [-s SOURCE_DIC_PATH] [--target_dic_path TARGET_DIC_PATH]
+                [-b BATCH_SIZE] [-p NUM_PASSES] -y MODEL_TYPE --model_arch
+                MODEL_ARCH
+                [--share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET]
+                [--share_embed SHARE_EMBED] [--dnn_dims DNN_DIMS]
+                [--num_workers NUM_WORKERS] [--use_gpu USE_GPU] [-c CLASS_NUM]
+
+PaddlePaddle DSSM example
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -i TRAIN_DATA_PATH, --train_data_path TRAIN_DATA_PATH
+                        path of training dataset
+  -t TEST_DATA_PATH, --test_data_path TEST_DATA_PATH
+                        path of testing dataset
+  -s SOURCE_DIC_PATH, --source_dic_path SOURCE_DIC_PATH
+                        path of the source's word dic
+  --target_dic_path TARGET_DIC_PATH
+                        path of the target's word dic, if not set, the
+                        `source_dic_path` will be used
+  -b BATCH_SIZE, --batch_size BATCH_SIZE
+                        size of mini-batch (default:10)
+  -p NUM_PASSES, --num_passes NUM_PASSES
+                        number of passes to run(default:10)
+  -y MODEL_TYPE, --model_type MODEL_TYPE
+                        model type, 0 for classification, 1 for pairwise rank
+                        (default: classification)
+  --model_arch MODEL_ARCH
+                        model architecture, 1 for CNN, 0 for FC, 2 for RNN
+  --share_network_between_source_target SHARE_NETWORK_BETWEEN_SOURCE_TARGET
+                        whether to share network parameters between source and
+                        target
+  --share_embed SHARE_EMBED
+                        whether to share word embedding between source and
+                        target
+  --dnn_dims DNN_DIMS   dimentions of dnn layers, default is '256,128,64,32',
+                        which means create a 4-layer dnn, demention of each
+                        layer is 256, 128, 64 and 32
+  --num_workers NUM_WORKERS
+                        num worker threads, default 1
+  --use_gpu USE_GPU     whether to use GPU devices (default: False)
+  -c CLASS_NUM, --class_num CLASS_NUM
+                        number of categories for classification task.
+```
+
+## 参考文献
+
+1. Huang P S, He X, Gao J, et al. Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM international conference on Conference on information & knowledge management. ACM, 2013: 2333-2338.
+2. [Microsoft Learning to Rank Datasets](https://www.microsoft.com/en-us/research/project/mslr/)
+3. [Gao J, He X, Deng L. Deep Learning for Web Search and Natural Language Processing[J]. Microsoft Research Technical Report, 2015.](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/wsdm2015.v3.pdf)

dssm/data/classification/test.txt

+苹果	苹果 6s	0
+汽车 驾驶	驾校 培训	1

dssm/data/classification/train.txt

+苹果 六 袋	苹果 6s	0
+新手 汽车 驾驶	驾校 培训	1

dssm/data/rank/test.txt

+苹果 六 袋	苹果 6s	新手 汽车 驾驶	1
+新手 汽车 驾驶	驾校 培训	苹果 6s	0

dssm/data/rank/train.txt

+苹果 六 袋	苹果 6s	新手 汽车 驾驶	1
+新手 汽车 驾驶	驾校 培训	苹果 6s	1

dssm/data/vocab.txt

+UNK
+苹果
+六
+袋
+6s
+新手
+汽车
+驾驶
+驾校
+培训
\ No newline at end of file

dssm/network_conf.py

+from paddle import v2 as paddle
+from paddle.v2.attr import ParamAttr
+from utils import TaskType, logger, ModelType, ModelArch
+
+
+class DSSM(object):
+    def __init__(self,
+                 dnn_dims=[],
+                 vocab_sizes=[],
+                 model_type=ModelType.create_classification(),
+                 model_arch=ModelArch.create_cnn(),
+                 share_semantic_generator=False,
+                 class_num=None,
+                 share_embed=False):
+        '''
+        @dnn_dims: list of int
+            dimentions of each layer in semantic vector generator.
+        @vocab_sizes: 2-d tuple
+            size of both left and right items.
+        @model_type: int
+            type of task, should be 'rank: 0', 'regression: 1' or 'classification: 2'
+        @model_arch: int
+            model architecture
+        @share_semantic_generator: bool
+            whether to share the semantic vector generator for both left and right.
+        @share_embed: bool
+            whether to share the embeddings between left and right.
+        @class_num: int
+            number of categories.
+        '''
+        assert len(
+            vocab_sizes
+        ) == 2, "vocab_sizes specify the sizes left and right inputs, and dim should be 2."
+        assert len(dnn_dims) > 1, "more than two layers is needed."
+
+        self.dnn_dims = dnn_dims
+        self.vocab_sizes = vocab_sizes
+        self.share_semantic_generator = share_semantic_generator
+        self.share_embed = share_embed
+        self.model_type = ModelType(model_type)
+        self.model_arch = ModelArch(model_arch)
+        self.class_num = class_num
+        logger.warning("build DSSM model with config of %s, %s" %
+                       (self.model_type, self.model_arch))
+        logger.info("vocabulary sizes: %s" % str(self.vocab_sizes))
+
+        # bind model architecture
+        _model_arch = {
+            'cnn': self.create_cnn,
+            'fc': self.create_fc,
+            'rnn': self.create_rnn,
+        }
+
+        def _model_arch_creater(emb, prefix=''):
+            sent_vec = _model_arch.get(str(model_arch))(emb, prefix)
+            dnn = self.create_dnn(sent_vec, prefix)
+            return dnn
+
+        self.model_arch_creater = _model_arch_creater
+
+        # build model type
+        _model_type = {
+            'classification': self._build_classification_model,
+            'rank': self._build_rank_model,
+            'regression': self._build_regression_model,
+        }
+        print 'model type: ', str(self.model_type)
+        self.model_type_creater = _model_type[str(self.model_type)]
+
+    def __call__(self):
+        # if self.model_type.is_classification():
+        #     return self._build_classification_model()
+        # return self._build_rank_model()
+        return self.model_type_creater()
+
+    def create_embedding(self, input, prefix=''):
+        '''
+        Create an embedding table whose name has a `prefix`.
+        '''
+        logger.info("create embedding table [%s] which dimention is %d" %
+                    (prefix, self.dnn_dims[0]))
+        emb = paddle.layer.embedding(
+            input=input,
+            size=self.dnn_dims[0],
+            param_attr=ParamAttr(name='%s_emb.w' % prefix))
+        return emb
+
+    def create_fc(self, emb, prefix=''):
+        '''
+        A multi-layer fully connected neural networks.
+
+        @emb: paddle.layer
+            output of the embedding layer
+        @prefix: str
+            prefix of layers' names, used to share parameters between more than one `fc` parts.
+        '''
+        _input_layer = paddle.layer.pooling(
+            input=emb, pooling_type=paddle.pooling.Max())
+        fc = paddle.layer.fc(input=_input_layer, size=self.dnn_dims[1])
+        return fc
+
+    def create_rnn(self, emb, prefix=''):
+        '''
+        A GRU sentence vector learner.
+        '''
+        gru = paddle.layer.gru_memory(
+            input=emb, )
+        sent_vec = paddle.layer.last_seq(gru)
+        return sent_vec
+
+    def create_cnn(self, emb, prefix=''):
+        '''
+        A multi-layer CNN.
+
+        @emb: paddle.layer
+            output of the embedding layer
+        @prefix: str
+            prefix of layers' names, used to share parameters between more than one `cnn` parts.
+        '''
+
+        def create_conv(context_len, hidden_size, prefix):
+            key = "%s_%d_%d" % (prefix, context_len, hidden_size)
+            conv = paddle.networks.sequence_conv_pool(
+                input=emb,
+                context_len=context_len,
+                hidden_size=hidden_size,
+                # set parameter attr for parameter sharing
+                context_proj_param_attr=ParamAttr(name=key + 'contex_proj.w'),
+                fc_param_attr=ParamAttr(name=key + '_fc.w'),
+                fc_bias_attr=ParamAttr(name=key + '_fc.b'),
+                pool_bias_attr=ParamAttr(name=key + '_pool.b'))
+            return conv
+
+        logger.info('create a sequence_conv_pool which context width is 3')
+        conv_3 = create_conv(3, self.dnn_dims[1], "cnn")
+        logger.info('create a sequence_conv_pool which context width is 4')
+        conv_4 = create_conv(4, self.dnn_dims[1], "cnn")
+
+        return conv_3, conv_4
+
+    def create_dnn(self, sent_vec, prefix):
+        # if more than three layers, than a fc layer will be added.
+        if len(self.dnn_dims) > 1:
+            _input_layer = sent_vec
+            for id, dim in enumerate(self.dnn_dims[1:]):
+                name = "%s_fc_%d_%d" % (prefix, id, dim)
+                logger.info("create fc layer [%s] which dimention is %d" %
+                            (name, dim))
+                fc = paddle.layer.fc(
+                    name=name,
+                    input=_input_layer,
+                    size=dim,
+                    act=paddle.activation.Tanh(),
+                    param_attr=ParamAttr(name='%s.w' % name),
+                    bias_attr=ParamAttr(name='%s.b' % name))
+                _input_layer = fc
+        return _input_layer
+
+    def _build_classification_model(self):
+        logger.info("build classification model")
+        assert self.model_type.is_classification()
+        return self._build_classification_or_regression_model(
+            is_classification=True)
+
+    def _build_regression_model(self):
+        logger.info("build regression model")
+        assert self.model_type.is_regression()
+        return self._build_classification_or_regression_model(
+            is_classification=False)
+
+    def _build_rank_model(self):
+        '''
+        Build a pairwise rank model, and the cost is returned.
+
+        A pairwise rank model has 3 inputs:
+          - source sentence
+          - left_target sentence
+          - right_target sentence
+          - label, 1 if left_target should be sorted in front of right_target, otherwise 0.
+        '''
+        logger.info("build rank model")
+        assert self.model_type.is_rank()
+        source = paddle.layer.data(
+            name='source_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+        left_target = paddle.layer.data(
+            name='left_target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        right_target = paddle.layer.data(
+            name='right_target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        label = paddle.layer.data(
+            name='label_input', type=paddle.data_type.integer_value(1))
+
+        prefixs = '_ _ _'.split(
+        ) if self.share_semantic_generator else 'source left right'.split()
+        embed_prefixs = '_ _'.split(
+        ) if self.share_embed else 'source target target'.split()
+
+        word_vecs = []
+        for id, input in enumerate([source, left_target, right_target]):
+            x = self.create_embedding(input, prefix=embed_prefixs[id])
+            word_vecs.append(x)
+
+        semantics = []
+        for id, input in enumerate(word_vecs):
+            x = self.model_arch_creater(input, prefix=prefixs[id])
+            semantics.append(x)
+
+        # cossim score of source and left_target
+        left_score = paddle.layer.cos_sim(semantics[0], semantics[1])
+        # cossim score of source and right target
+        right_score = paddle.layer.cos_sim(semantics[0], semantics[2])
+
+        # rank cost
+        cost = paddle.layer.rank_cost(left_score, right_score, label=label)
+        # prediction = left_score - right_score
+        # but this operator is not supported currently.
+        # so AUC will not used.
+        return cost, None, None
+
+    def _build_classification_or_regression_model(self, is_classification):
+        '''
+        Build a classification/regression model, and the cost is returned.
+
+        A Classification has 3 inputs:
+          - source sentence
+          - target sentence
+          - classification label
+
+        '''
+        if is_classification:
+            # prepare inputs.
+            assert self.class_num
+
+        source = paddle.layer.data(
+            name='source_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[0]))
+        target = paddle.layer.data(
+            name='target_input',
+            type=paddle.data_type.integer_value_sequence(self.vocab_sizes[1]))
+        label = paddle.layer.data(
+            name='label_input',
+            type=paddle.data_type.integer_value(self.class_num)
+            if is_classification else paddle.data_type.dense_vector(1))
+
+        prefixs = '_ _'.split(
+        ) if self.share_semantic_generator else 'left right'.split()
+        embed_prefixs = '_ _'.split(
+        ) if self.share_embed else 'left right'.split()
+
+        word_vecs = []
+        for id, input in enumerate([source, target]):
+            x = self.create_embedding(input, prefix=embed_prefixs[id])
+            word_vecs.append(x)
+
+        semantics = []
+        for id, input in enumerate(word_vecs):
+            x = self.model_arch_creater(input, prefix=prefixs[id])
+            semantics.append(x)
+
+        if is_classification:
+            concated_vector = paddle.layer.concat(semantics)
+            prediction = paddle.layer.fc(
+                input=concated_vector,
+                size=self.class_num,
+                act=paddle.activation.Softmax())
+            cost = paddle.layer.classification_cost(
+                input=prediction, label=label)
+        else:
+            prediction = paddle.layer.cos_sim(*semantics)
+            cost = paddle.layer.mse_cost(prediction, label)
+        return cost, prediction, label
+
+
+class RankMetrics(object):
+    '''
+    A custom metrics to calculate AUC.
+
+    Paddle's rank model do not support auc evaluator directly,
+    to make it, infer all the outputs and use python to calculate
+    the metrics.
+    '''
+
+    def __init__(self, model_parameters, left_score_layer, right_score_layer,
+                 label):
+        '''
+        @model_parameters: dict
+            model's parameters
+        @left_score_layer: paddle.layer
+            left part's score
+        @right_score_laeyr: paddle.layer
+            right part's score
+        @label: paddle.data_layer
+            label input
+        '''
+        self.inferer = paddle.inference.Inference(
+            output_layer=[left_score_layer, right_score_layer],
+            parameters=model_parameters)
+
+    def test(self, input):
+        scores = []
+        for id, rcd in enumerate(input()):
+            # output [left_score, right_score, label]
+            res = self.inferer(input=input)
+            scores.append(res)
+        print scores

dssm/reader.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+from utils import UNK, ModelType, TaskType, load_dic, sent2ids, logger, ModelType
+
+
+class Dataset(object):
+    def __init__(self, train_path, test_path, source_dic_path, target_dic_path,
+                 model_type):
+        self.train_path = train_path
+        self.test_path = test_path
+        self.source_dic_path = source_dic_path
+        self.target_dic_path = target_dic_path
+        self.model_type = ModelType(model_type)
+
+        self.source_dic = load_dic(self.source_dic_path)
+        self.target_dic = load_dic(self.target_dic_path)
+
+        _record_reader = {
+            ModelType.CLASSIFICATION_MODE: self._read_classification_record,
+            ModelType.REGRESSION_MODE: self._read_regression_record,
+            ModelType.RANK_MODE: self._read_rank_record,
+        }
+
+        assert isinstance(model_type, ModelType)
+        self.record_reader = _record_reader[model_type.mode]
+
+    def train(self):
+        '''
+        Load trainset.
+        '''
+        logger.info("[reader] load trainset from %s" % self.train_path)
+        with open(self.train_path) as f:
+            for line_id, line in enumerate(f):
+                yield self.record_reader(line)
+
+    def test(self):
+        '''
+        Load testset.
+        '''
+        logger.info("[reader] load testset from %s" % self.test_path)
+        with open(self.test_path) as f:
+            for line_id, line in enumerate(f):
+                yield self.record_reader(line)
+
+    def _read_classification_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <target words> [TAB] <label>
+
+        @line: str
+            a string line which represent a record.
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 3, "wrong format for classification\n" + \
+            "the format shoud be " +\
+            "<source words> [TAB] <target words> [TAB] <label>'"
+        source = sent2ids(fs[0], self.source_dic)
+        target = sent2ids(fs[1], self.target_dic)
+        label = int(fs[2])
+        return (source, target, label, )
+
+    def _read_regression_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <target words> [TAB] <label>
+
+        @line: str
+            a string line which represent a record.
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 3, "wrong format for regression\n" + \
+            "the format shoud be " +\
+            "<source words> [TAB] <target words> [TAB] <label>'"
+        source = sent2ids(fs[0], self.source_dic)
+        target = sent2ids(fs[1], self.target_dic)
+        label = float(fs[2])
+        return (source, target, [label], )
+
+    def _read_rank_record(self, line):
+        '''
+        data format:
+            <source words> [TAB] <left_target words> [TAB] <right_target words> [TAB] <label>
+        '''
+        fs = line.strip().split('\t')
+        assert len(fs) == 4, "wrong format for rank\n" + \
+            "the format should be " +\
+            "<source words> [TAB] <left_target words> [TAB] <right_target words> [TAB] <label>"
+
+        source = sent2ids(fs[0], self.source_dic)
+        left_target = sent2ids(fs[1], self.target_dic)
+        right_target = sent2ids(fs[2], self.target_dic)
+        label = int(fs[3])
+
+        return (source, left_target, right_target, label)
+
+
+if __name__ == '__main__':
+    path = './data/classification/train.txt'
+    test_path = './data/classification/test.txt'
+    source_dic = './data/vocab.txt'
+    dataset = Dataset(path, test_path, source_dic, source_dic,
+                      ModelType.CLASSIFICATION)
+
+    for rcd in dataset.train():
+        print rcd

dssm/train.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import argparse
+import gzip
+
+import paddle.v2 as paddle
+from network_conf import DSSM
+import reader
+from utils import TaskType, load_dic, logger, ModelType, ModelArch
+
+parser = argparse.ArgumentParser(description="PaddlePaddle DSSM example")
+
+parser.add_argument(
+    '-i',
+    '--train_data_path',
+    type=str,
+    required=False,
+    help="path of training dataset")
+parser.add_argument(
+    '-t',
+    '--test_data_path',
+    type=str,
+    required=False,
+    help="path of testing dataset")
+parser.add_argument(
+    '-s',
+    '--source_dic_path',
+    type=str,
+    required=False,
+    help="path of the source's word dic")
+parser.add_argument(
+    '--target_dic_path',
+    type=str,
+    required=False,
+    help="path of the target's word dic, if not set, the `source_dic_path` will be used"
+)
+parser.add_argument(
+    '-b',
+    '--batch_size',
+    type=int,
+    default=10,
+    help="size of mini-batch (default:10)")
+parser.add_argument(
+    '-p',
+    '--num_passes',
+    type=int,
+    default=10,
+    help="number of passes to run(default:10)")
+parser.add_argument(
+    '-y',
+    '--model_type',
+    type=int,
+    required=True,
+    default=ModelType.CLASSIFICATION_MODE,
+    help="model type, %d for classification, %d for pairwise rank, %d for regression (default: classification)"
+    % (ModelType.CLASSIFICATION_MODE, ModelType.RANK_MODE,
+       ModelType.REGRESSION_MODE))
+parser.add_argument(
+    '--model_arch',
+    type=int,
+    required=True,
+    default=ModelArch.CNN_MODE,
+    help="model architecture, %d for CNN, %d for FC, %d for RNN" %
+    (ModelArch.CNN_MODE, ModelArch.FC_MODE, ModelArch.RNN_MODE))
+parser.add_argument(
+    '--share_network_between_source_target',
+    type=bool,
+    default=False,
+    help="whether to share network parameters between source and target")
+parser.add_argument(
+    '--share_embed',
+    type=bool,
+    default=False,
+    help="whether to share word embedding between source and target")
+parser.add_argument(
+    '--dnn_dims',
+    type=str,
+    default='256,128,64,32',
+    help="dimentions of dnn layers, default is '256,128,64,32', which means create a 4-layer dnn, demention of each layer is 256, 128, 64 and 32"
+)
+parser.add_argument(
+    '--num_workers', type=int, default=1, help="num worker threads, default 1")
+parser.add_argument(
+    '--use_gpu',
+    type=bool,
+    default=False,
+    help="whether to use GPU devices (default: False)")
+parser.add_argument(
+    '-c',
+    '--class_num',
+    type=int,
+    default=0,
+    help="number of categories for classification task.")
+
+# arguments check.
+args = parser.parse_args()
+args.model_type = ModelType(args.model_type)
+args.model_arch = ModelArch(args.model_arch)
+if args.model_type.is_classification():
+    assert args.class_num > 1, "--class_num should be set in classification task."
+
+layer_dims = [int(i) for i in args.dnn_dims.split(',')]
+target_dic_path = args.source_dic_path if not args.target_dic_path else args.target_dic_path
+
+model_save_name_prefix = "dssm_pass_%s_%s" % (args.model_type,
+                                              args.model_arch, )
+
+
+def train(train_data_path=None,
+          test_data_path=None,
+          source_dic_path=None,
+          target_dic_path=None,
+          model_type=ModelType.create_classification(),
+          model_arch=ModelArch.create_cnn(),
+          batch_size=10,
+          num_passes=10,
+          share_semantic_generator=False,
+          share_embed=False,
+          class_num=None,
+          num_workers=1,
+          use_gpu=False):
+    '''
+    Train the DSSM.
+    '''
+    default_train_path = './data/rank/train.txt'
+    default_test_path = './data/rank/test.txt'
+    default_dic_path = './data/vocab.txt'
+    if not model_type.is_rank():
+        default_train_path = './data/classification/train.txt'
+        default_test_path = './data/classification/test.txt'
+
+    use_default_data = not train_data_path
+
+    if use_default_data:
+        train_data_path = default_train_path
+        test_data_path = default_test_path
+        source_dic_path = default_dic_path
+        target_dic_path = default_dic_path
+
+    dataset = reader.Dataset(
+        train_path=train_data_path,
+        test_path=test_data_path,
+        source_dic_path=source_dic_path,
+        target_dic_path=target_dic_path,
+        model_type=model_type, )
+
+    train_reader = paddle.batch(
+        paddle.reader.shuffle(dataset.train, buf_size=1000),
+        batch_size=batch_size)
+
+    test_reader = paddle.batch(
+        paddle.reader.shuffle(dataset.test, buf_size=1000),
+        batch_size=batch_size)
+
+    paddle.init(use_gpu=use_gpu, trainer_count=num_workers)
+
+    cost, prediction, label = DSSM(
+        dnn_dims=layer_dims,
+        vocab_sizes=[
+            len(load_dic(path)) for path in [source_dic_path, target_dic_path]
+        ],
+        model_type=model_type,
+        model_arch=model_arch,
+        share_semantic_generator=share_semantic_generator,
+        class_num=class_num,
+        share_embed=share_embed)()
+
+    parameters = paddle.parameters.create(cost)
+
+    adam_optimizer = paddle.optimizer.Adam(
+        learning_rate=1e-3,
+        regularization=paddle.optimizer.L2Regularization(rate=1e-3),
+        model_average=paddle.optimizer.ModelAverage(average_window=0.5))
+
+    trainer = paddle.trainer.SGD(
+        cost=cost,
+        extra_layers=None,
+        parameters=parameters,
+        update_equation=adam_optimizer)
+    # trainer = paddle.trainer.SGD(
+    #     cost=cost,
+    #     extra_layers=paddle.evaluator.auc(input=prediction, label=label)
+    #     if prediction and model_type.is_classification() else None,
+    #     parameters=parameters,
+    #     update_equation=adam_optimizer)
+
+    feeding = {}
+    if model_type.is_classification() or model_type.is_regression():
+        feeding = {'source_input': 0, 'target_input': 1, 'label_input': 2}
+    else:
+        feeding = {
+            'source_input': 0,
+            'left_target_input': 1,
+            'right_target_input': 2,
+            'label_input': 3
+        }
+
+    def _event_handler(event):
+        '''
+        Define batch handler
+        '''
+        if isinstance(event, paddle.event.EndIteration):
+            if event.batch_id % 100 == 0:
+                logger.info("Pass %d, Batch %d, Cost %f, %s\n" % (
+                    event.pass_id, event.batch_id, event.cost, event.metrics))
+
+        if isinstance(event, paddle.event.EndPass):
+            if test_reader is not None:
+                if model_type.is_classification():
+                    result = trainer.test(reader=test_reader, feeding=feeding)
+                    logger.info("Test at Pass %d, %s \n" % (event.pass_id,
+                                                            result.metrics))
+                else:
+                    result = None
+            with gzip.open("dssm_%s_pass_%05d.tar.gz" %
+                           (model_save_name_prefix, event.pass_id), "w") as f:
+                parameters.to_tar(f)
+
+    trainer.train(
+        reader=train_reader,
+        event_handler=_event_handler,
+        feeding=feeding,
+        num_passes=num_passes)
+
+    logger.info("Training has finished.")
+
+
+if __name__ == '__main__':
+    train(
+        train_data_path=args.train_data_path,
+        test_data_path=args.test_data_path,
+        source_dic_path=args.source_dic_path,
+        target_dic_path=args.target_dic_path,
+        model_type=ModelType(args.model_type),
+        model_arch=ModelArch(args.model_arch),
+        batch_size=args.batch_size,
+        num_passes=args.num_passes,
+        share_semantic_generator=args.share_network_between_source_target,
+        share_embed=args.share_embed,
+        class_num=args.class_num,
+        num_workers=args.num_workers,
+        use_gpu=args.use_gpu)

dssm/utils.py

+import paddle
+
+UNK = 0
+
+logger = logging.getLogger("paddle")
+logger.setLevel(logging.INFO)
+
+
+def mode_attr_name(mode):
+    return mode.upper() + '_MODE'
+
+
+def create_attrs(cls):
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, mode_attr_name(mode), id)
+
+
+def make_check_method(cls):
+    '''
+    create methods for classes.
+    '''
+
+    def method(mode):
+        def _method(self):
+            return self.mode == getattr(cls, mode_attr_name(mode))
+
+        return _method
+
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, 'is_' + mode, method(mode))
+
+
+def make_create_method(cls):
+    def method(mode):
+        @staticmethod
+        def _method():
+            key = getattr(cls, mode_attr_name(mode))
+            return cls(key)
+
+        return _method
+
+    for id, mode in enumerate(cls.modes):
+        setattr(cls, 'create_' + mode, method(mode))
+
+
+def make_str_method(cls, type_name='unk'):
+    def _str_(self):
+        for mode in cls.modes:
+            if self.mode == getattr(cls, mode_attr_name(mode)):
+                return mode
+
+    def _hash_(self):
+        return self.mode
+
+    setattr(cls, '__str__', _str_)
+    setattr(cls, '__repr__', _str_)
+    setattr(cls, '__hash__', _hash_)
+    cls.__name__ = type_name
+
+
+def _init_(self, mode, cls):
+    if isinstance(mode, int):
+        self.mode = mode
+    elif isinstance(mode, cls):
+        self.mode = mode.mode
+    else:
+        raise Exception("wrong mode type, get type: %s, value: %s" %
+                        (type(mode), mode))
+
+
+def build_mode_class(cls):
+    create_attrs(cls)
+    make_str_method(cls)
+    make_check_method(cls)
+    make_create_method(cls)
+
+
+class TaskType(object):
+    modes = 'train test infer'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, TaskType)
+
+
+class ModelType:
+    modes = 'classification rank regression'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, ModelType)
+
+
+class ModelArch:
+    modes = 'fc cnn rnn'.split()
+
+    def __init__(self, mode):
+        _init_(self, mode, ModelArch)
+
+
+build_mode_class(TaskType)
+build_mode_class(ModelType)
+build_mode_class(ModelArch)
+
+
+def sent2ids(sent, vocab):
+    '''
+    transform a sentence to a list of ids.
+
+    @sent: str
+        a sentence.
+    @vocab: dict
+        a word dic
+    '''
+    return [vocab.get(w, UNK) for w in sent.split()]
+
+
+def load_dic(path):
+    '''
+    word dic format:
+      each line is a word
+    '''
+    dic = {}
+    with open(path) as f:
+        for id, line in enumerate(f):
+            w = line.strip()
+            dic[w] = id
+    return dic
+
+
+if __name__ == '__main__':
+    t = TaskType(1)
+    t = TaskType.create_train()
+    print t
+    print 'is', t.is_train()

generate_sequence_by_rnn_lm/config.py

@@ -26,7 +26,7 @@ num_passes = 20  # how many passes to train the model
 log_period = 50
 save_period_by_batches = 50
 
-use_gpu = True  # to use gpu or not
+use_gpu = False  # to use gpu or not
 trainer_count = 1  # number of trainer
 
 ##################  for model configuration  ##################

generate_sequence_by_rnn_lm/train.py

@@ -35,8 +35,7 @@ def train(topology,
         os.mkdir(model_save_dir)
 
     # initialize PaddlePaddle
-    paddle.init(
-        use_gpu=conf.use_gpu, gpu_id=3, trainer_count=conf.trainer_count)
+    paddle.init(use_gpu=conf.use_gpu, trainer_count=conf.trainer_count)
 
     # create optimizer
     adam_optimizer = paddle.optimizer.Adam(

image_classification/caffe2paddle/caffe2paddle.py

@@ -72,6 +72,10 @@ class ModelConverter(object):
                 file_name = "_%s.w%s" % (name, str(i))
             param_conf = ParameterConfig()
             param_conf.name = file_name
+            dims = list(data.shape)
+            if len(dims) == 1:
+                dims.insert(1, 1)
+                param_conf.dims.extend(dims)
             param_conf.size = reduce(lambda a, b: a * b, data.shape)
             self.params[file_name] = (param_conf, data.flatten())
 

nmt_without_attention/train.py

@@ -63,4 +63,4 @@ def train(save_dir_path, source_dict_dim, target_dict_dim):
 
 
 if __name__ == '__main__':
-    train(save_dir_path="models", source_dict_dim=3000, target_dict_dim=3000)
+    train(save_dir_path="models", source_dict_dim=30000, target_dict_dim=30000)

sequence_tagging_for_ner/README.md

@@ -23,10 +23,10 @@
 
 序列标注可以分为Sequence Classification、Segment Classification和Temporal Classification三类[[1](#参考文献)]，本例只考虑Segment Classification，即对输入序列中的每个元素在输出序列中给出对应的标签。对于NER任务，由于需要标识边界，一般采用[BIO标注方法](http://book.paddlepaddle.org/07.label_semantic_roles/)定义的标签集，如下是一个NER的标注结果示例：
 
-<div  align="center">
-<img src="images/ner_label_ins.png" width = "80%"  align=center /><br>
+<p align="center">
+<img src="images/ner_label_ins.png" width="80%" align="center"/><br/>
 图1. BIO标注方法示例
-</div>
+</p>
 
 根据序列标注结果可以直接得到实体边界和实体类别。类似的，分词、词性标注、语块识别、[语义角色标注](http://book.paddlepaddle.org/07.label_semantic_roles/index.cn.html)等任务都可通过序列标注来解决。使用神经网络模型解决问题的思路通常是：前层网络学习输入的特征表示，网络的最后一层在特征基础上完成最终的任务；对于序列标注问题，通常：使用基于RNN的网络结构学习特征，将学习到的特征接入CRF完成序列标注。实际上是将传统CRF中的线性模型换成了非线性神经网络。沿用CRF的出发点是：CRF使用句子级别的似然概率，能够更好的解决标记偏置问题[[2](#参考文献)]。本例也将基于此思路建立模型。虽然，这里以NER任务作为示例，但所给出的模型可以应用到其他各种序列标注任务中。
 
@@ -43,10 +43,10 @@ NER任务的输入是"一句话"，目标是识别句子中的实体边界及类
 3. 将步骤2中的2个词向量序列作为双向RNN的输入，学习输入序列的特征表示，得到新的特性表示序列；
 4. CRF以步骤3中模型学习到的特征为输入，以标记序列为监督信号，实现序列标注。
 
-<div  align="center">  
-<img src="images/ner_network.png" width = "40%"  align=center /><br>
+<p align="center">
+<img src="images/ner_network.png" width="40%" align="center"/><br/>
 图2. NER 模型网络结构图
-</div>
+</p>
 
 
 ## 数据说明

sequence_tagging_for_ner/index.html

@@ -65,10 +65,10 @@
 
 序列标注可以分为Sequence Classification、Segment Classification和Temporal Classification三类[[1](#参考文献)]，本例只考虑Segment Classification，即对输入序列中的每个元素在输出序列中给出对应的标签。对于NER任务，由于需要标识边界，一般采用[BIO标注方法](http://book.paddlepaddle.org/07.label_semantic_roles/)定义的标签集，如下是一个NER的标注结果示例：
 
-<div  align="center">
-<img src="images/ner_label_ins.png" width = "80%"  align=center /><br>
+<p align="center">
+<img src="images/ner_label_ins.png" width="80%" align="center"/><br/>
 图1. BIO标注方法示例
-</div>
+</p>
 
 根据序列标注结果可以直接得到实体边界和实体类别。类似的，分词、词性标注、语块识别、[语义角色标注](http://book.paddlepaddle.org/07.label_semantic_roles/index.cn.html)等任务都可通过序列标注来解决。使用神经网络模型解决问题的思路通常是：前层网络学习输入的特征表示，网络的最后一层在特征基础上完成最终的任务；对于序列标注问题，通常：使用基于RNN的网络结构学习特征，将学习到的特征接入CRF完成序列标注。实际上是将传统CRF中的线性模型换成了非线性神经网络。沿用CRF的出发点是：CRF使用句子级别的似然概率，能够更好的解决标记偏置问题[[2](#参考文献)]。本例也将基于此思路建立模型。虽然，这里以NER任务作为示例，但所给出的模型可以应用到其他各种序列标注任务中。
 
@@ -85,10 +85,10 @@ NER任务的输入是"一句话"，目标是识别句子中的实体边界及类
 3. 将步骤2中的2个词向量序列作为双向RNN的输入，学习输入序列的特征表示，得到新的特性表示序列；
 4. CRF以步骤3中模型学习到的特征为输入，以标记序列为监督信号，实现序列标注。
 
-<div  align="center">  
-<img src="images/ner_network.png" width = "40%"  align=center /><br>
+<p align="center">
+<img src="images/ner_network.png" width="40%" align="center"/><br/>
 图2. NER 模型网络结构图
-</div>
+</p>
 
 
 ## 数据说明