Upload mode: Dialogue-PLATO. (#3932)

PaddleNLP/Research/Dialogue-PLATO/.gitignore

+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don’t work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

PaddleNLP/Research/Dialogue-PLATO/README.md

+# PLATO
+**PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable**
+[paper link](http://arxiv.org/abs/1910.07931)
+
+**\*\*\*\*\* Update \*\*\*\*\***
+
+Nov. 14: Support new APIs in paddlepaddle 1.6.0 (model files in the link have been updated accordingly), multi-GPU training and decoding strategy of top-k sampling. Release our baseline model `PLATO w/o latent`.
+
+## Requirements
+```
+- python >= 3.6
+- paddlepaddle >= 1.6.0
+- numpy
+- nltk
+- tqdm
+- visualdl >= 1.3.0 (optional)
+- regex
+```
+
+## Pre-trained dialogue generation model
+A novel pre-training model for dialogue generation is introduced in this work, incorporated with latent discrete variables for one-to-many relationship modeling. Our model is flexible enough to support various kinds of conversations, including chit-chat, knowledge grounded dialogues, and conversational question answering. The pre-training is carried out with Reddit and Twitter corpora. You can download the uncased pre-trained model from:
+* PLATO, uncased [model](https://baidu-nlp.bj.bcebos.com/PLATO/model.tar.gz): 12-layers, 768-hidden, 12-heads, 132M parameters
+* PLATO w/o latent, uncased [model](https://baidu-nlp.bj.bcebos.com/PLATO/model-baseline.tar.gz): 12-layers 768-hidden, 12-heads, 109M parameters
+
+```bash
+mv /path/to/model.tar.gz .
+tar xzf model.tar.gz
+```
+
+## Fine-tuning
+We also provide instructions to fine-tune PLATO on different conversation datasets (chit-chat, knowledge grounded dialogues and conversational question answering).
+
+### Data preparation
+Download data from the [link](https://baidu-nlp.bj.bcebos.com/PLATO/data.tar.gz).
+The tar file contains three processed datasets: `DailyDialog`, `PersonaChat` and `DSTC7_AVSD`.
+```bash
+mv /path/to/data.tar.gz .
+tar xzf data.tar.gz
+```
+
+### Data format
+Our model supports two kinds of data formats for dialogue context: `multi` and `multi_knowledge`.
+* `multi`: multi-turn dialogue context.
+```txt
+u_1 __eou__ u_2 __eou__ ... u_n \t r
+```
+* `multi_knowledge`: multi-turn dialogue context with background knowledges.
+```txt
+k_1 __eou__ k_2 __eou__ ... k_m \t u_1 __eou__ u_2 __eou__ ... u_n \t r
+```
+
+If you want to use this model on other datasets, you can process your data accordingly.
+
+### Train
+Fine-tuning the pre-trained model on different `${DATASET}`.
+```bash
+# DailyDialog / PersonaChat / DSTC7_AVSD
+DATASET=DailyDialog
+sh scripts/${DATASET}/train.sh
+```
+After training, you can find the output folder `outputs/${DATASET}` (by default). It contatins `best.model` (best results on validation dataset), `hparams.json` (hyper-parameters of training script) and `trainer.log` (training log).
+
+
+Fine-tuning the pre-trained model on multiple GPUs.
+
+Note: You need to install NCCL library and set up the environment variable `LD_LIBRARY` properly.
+```bash
+sh scripts/DailyDialog/multi_gpu_train.sh
+```
+
+You can fine-tune PLATO w/o latent on different `${DATASET}`. We provide an example script on DailyDialog dataset.
+```bash
+sh scripts/DailyDialog/baseline_train.sh
+```
+
+#### Recommended settings
+
+For the fine-tuning of our pre-trained model, it usually requires about 10 epochs to reach convergence with learning rate = 1e-5 and about 2-3 epochs to reach convergence with learning rate = 5e-5.
+
+GPU Memory | batch size | max len
+------|------|------
+16G | 6 | 256
+32G | 12 | 256
+
+### Infer
+Running inference on test dataset.
+```bash
+# DailyDialog / PersonaChat / DSTC7_AVSD
+DATASET=DailyDialog
+sh scripts/${DATASET}/infer.sh
+
+# Running inference of PLATO w/o latent
+sh scripts/DailyDialog/baseline_infer.sh
+```
+After inference, you can find the output foler `outputs/${DATASET}.infer` (by default). It contains `infer_0.result.json` (the inference result), `hparams.json` (hyper-parameters of inference scipt) and `trainer.log` (inference log).
+
+If you want to use top-k sampling (beam search by default), you can follow the example script:
+```bash
+sh scripts/DailyDialog/topk_infer.sh
+```
+
+## Result
+
+### DailyDialog
+Model | BLEU-1/2 | Distinct-1/2 | Fluency | Coherence | Informativeness | Overall
+------|------|------|------|------|------|-------
+Seq2Seq | 0.336/0.268 | 0.030/0.128 | 1.85 | 0.37 | 0.44 | 0.33
+iVAE_MI | 0.309/0.249 | 0.029/0.250 | 1.53 | 0.34 | 0.59 | 0.30
+Our w/o Latent | **0.405/0.322** | 0.046/0.246 | 1.91 | **1.58** | 1.03 | 1.44
+Our Method | 0.397/0.311 | **0.053/0.291** | **1.97** | 1.57 | **1.23** | **1.48**
+
+### PersonaChat
+Model | BLEU-1/2 | Distinct-1/2 | Knowledge R/P/F1 | Fluency | Coherence | Informativeness | Overall
+------|------|------|------|------|------|-------|-------
+Seq2Seq | 0.448/0.353 | 0.004/0.016 | 0.004/0.016/0.006 | 1.82 | 0.37 | 0.85 | 0.34
+LIC | 0.405/0.320 | 0.019/0.113 | 0.042/0.154/0.064 | 1.95 | 1.34 | 1.09 | 1.29
+Our w/o Latent | **0.458/0.357** | 0.012/0.064 | 0.085/0.263/0.125 | 1.98 | 1.36 | 1.04 | 1.30
+Our Method | 0.406/0.315 | **0.021/0.121** | **0.142/0.461/0.211** | **1.99** | **1.51** | **1.70** | **1.50**
+
+### DSTC7_AVSD
+Model | BELU-1 | BELU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGH-L | CIDEr
+------|------|------|------|------|------|-------|-------
+Baseline | 0.629 | 0.485 | 0.383 | 0.309 | 0.215 | 0.487 | 0.746
+CMU | 0.718 | 0.584 | 0.478 | 0.394 | 0.267 | 0.563 | 1.094
+Our Method | **0.784** | **0.637** | **0.525** | **0.435** | **0.286** | **0.596** | **1.209**
+Our Method Upper Bound | 0.925 | 0.843 | 0.767 | 0.689 | 0.361 | 0.731 | 1.716
+
+Note: In the experiments on `DSTC7_AVSD`, the response selection of our method is strengthened with an extra ranking step, which ranks the candidates according to the automatic scores and selects the top one as the final answer.
+
+## Citation
+If you find PLATO useful in your work, please cite the following Arxiv paper:
+```
+@article{bao2019plato,
+    title={PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable},
+    author={Bao, Siqi and He, Huang and Wang, Fan and Wu, Hua and Wang, Haifeng},
+    journal={arXiv preprint arXiv:1910.07931},
+    year={2019}
+}
+```
+
+## Disclaimer
+This project aims to facilitate further research progress in dialogue generation. Baidu is not responsible for the 3rd party's generation with the pre-trained system.
+
+## Contact information
+For help or issues using PLATO, please submit a GitHub issue.
+
+For personal communication related to PLATO, please contact Siqi Bao (`baosiqi@baidu.com`), or Huang He (`hehuang@baidu.com`).

PaddleNLP/Research/Dialogue-PLATO/plato/args.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Parse argument.
+"""
+
+import argparse
+import json
+
+
+def str2bool(v):
+    if v.lower() in ('yes', 'true', 't', 'y', '1'):
+        return True
+    elif v.lower() in ('no', 'false', 'f', 'n', '0'):
+        return False
+    else:
+        raise argparse.ArgumentTypeError('Unsupported value encountered.')
+
+
+class HParams(dict):
+    """ Hyper-parameters class
+
+    Store hyper-parameters in training / infer / ... scripts.
+    """
+
+    def __getattr__(self, name):
+        if name in self.keys():
+            return self[name]
+        for v in self.values():
+            if isinstance(v, HParams):
+                if name in v:
+                    return v[name]
+        raise AttributeError(f"'HParams' object has no attribute '{name}'")
+
+    def __setattr__(self, name, value):
+        self[name] = value
+
+    def save(self, filename):
+        with open(filename, "w", encoding="utf-8") as fp:
+            json.dump(self, fp, ensure_ascii=False,
+                      indent=4, sort_keys=False)
+
+    def load(self, filename):
+        with open(filename, "r", encoding="utf-8") as fp:
+            params_dict = json.load(fp)
+        for k, v in params_dict.items():
+            if isinstance(v, dict):
+                self[k].update(HParams(v))
+            else:
+                self[k] = v
+
+
+def parse_args(parser):
+    """ Parse hyper-parameters from cmdline. """
+    parsed = parser.parse_args()
+    args = HParams()
+    optional_args = parser._action_groups[1]
+    for action in optional_args._group_actions[1:]:
+        arg_name = action.dest
+        args[arg_name] = getattr(parsed, arg_name)
+    for group in parser._action_groups[2:]:
+        group_args = HParams()
+        for action in group._group_actions:
+            arg_name = action.dest
+            group_args[arg_name] = getattr(parsed, arg_name)
+        if len(group_args) > 0:
+            args[group.title] = group_args
+    return args

PaddleNLP/Research/Dialogue-PLATO/plato/data/data_loader.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+DataLoader class
+"""
+
+import math
+
+import paddle.fluid as fluid
+import paddle.batch
+
+from plato.args import str2bool
+from plato.data.sampler import RandomSampler
+from plato.data.sampler import SequentialSampler
+from plato.data.sampler import SortedSampler
+import plato.modules.parallel as parallel
+
+
+class DataLoader(object):
+    """ Implement of DataLoader. """
+
+    @classmethod
+    def add_cmdline_argument(cls, group):
+        group.add_argument("--shuffle", type=str2bool, default=True)
+        group.add_argument("--sort_pool_size", type=int, default=0)
+        return group
+
+    def __init__(self, dataset, hparams, collate_fn=None, sampler=None, is_test=False, is_train=False):
+        self.dataset = dataset
+        self.collate_fn = collate_fn
+        self.sort_pool_size = hparams.sort_pool_size
+
+        if sampler is None:
+            if hparams.shuffle and not is_test:
+                sampler = RandomSampler(dataset)
+            else:
+                sampler = SequentialSampler(dataset)
+
+        if self.sort_pool_size > 0 and not is_test:
+            sampler = SortedSampler(sampler, self.sort_pool_size)
+
+        def reader():
+            for idx in sampler:
+                yield idx
+
+        self.reader = paddle.batch(reader, batch_size=hparams.batch_size, drop_last=False)
+        self.num_batches = math.ceil(len(dataset) / hparams.batch_size)
+
+        if hparams.use_data_distributed and parallel.Env().nranks > 1 and is_train:
+            self.reader = fluid.contrib.reader.distributed_batch_reader(self.reader)
+            self.num_batches = self.num_batches // fluid.dygraph.parallel.Env().nranks
+
+        return
+
+    def __len__(self):
+        return self.num_batches
+
+    def __iter__(self):
+        for batch_indices in self.reader():
+            samples = [self.dataset[idx] for idx in batch_indices]
+            yield self.collate_fn(samples)

PaddleNLP/Research/Dialogue-PLATO/plato/data/dataset.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Dataset class
+"""
+
+import json
+
+
+class Dataset(object):
+    """ Basic Dataset interface class. """
+
+    @classmethod
+    def add_cmdline_argument(cls, parser):
+        group = parser.add_argument_group("Dataset")
+        group.add_argument("--data_dir", type=str, required=True,
+                           help="The dataset dir.")
+        group.add_argument("--data_type", type=str, required=True,
+                           choices=["multi", "multi_knowledge"],
+                           help="The type of dataset.")
+        return group
+
+    def __init__(self, data):
+        self.data = data
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+class LazyDataset(Dataset):
+    """
+    Lazy load dataset from disk.
+
+    Each line of data file is a preprocessed example.
+    """
+
+    def __init__(self, data_file, transform=lambda s: json.loads(s)):
+        """
+        Initialize lazy dataset.
+
+        By default, loading .jsonl format.
+
+        :param data_file
+        :type str
+
+        :param transform
+        :type callable
+        """
+        self.data_file = data_file
+        self.transform = transform
+        self.offsets = [0]
+        with open(data_file, "r", encoding="utf-8") as fp:
+            while fp.readline() != "":
+                self.offsets.append(fp.tell())
+        self.offsets.pop()
+        self.fp = open(data_file, "r", encoding="utf-8")
+
+    def __len__(self):
+        return len(self.offsets)
+
+    def __getitem__(self, idx):
+        self.fp.seek(self.offsets[idx], 0)
+        return self.transform(self.fp.readline().strip())

PaddleNLP/Research/Dialogue-PLATO/plato/data/field.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Field class
+"""
+
+from itertools import chain
+import json
+import numpy as np
+import pickle
+import time
+from tqdm import tqdm
+
+from plato.args import str2bool
+from plato.data.tokenizer import Tokenizer
+
+
+def max_lens(X):
+    lens = [len(X)]
+    while isinstance(X[0], list):
+        lens.append(max(map(len, X)))
+        X = [x for xs in X for x in xs]
+    return lens
+
+
+def list2np(X, padding=0, dtype="int64"):
+    shape = max_lens(X)
+    ret = np.full(shape, padding, dtype=np.int32)
+
+    if len(shape) == 1:
+        ret = np.array(X)
+    elif len(shape) == 2:
+        for i, x in enumerate(X):
+            ret[i, :len(x)] = np.array(x)
+    elif len(shape) == 3:
+        for i, xs in enumerate(X):
+            for j, x in enumerate(xs):
+                ret[i, j, :len(x)] = np.array(x)
+    return ret.astype(dtype)
+
+class BPETextField(object):
+
+    pad_token = "[PAD]"
+    bos_token = "[BOS]"
+    eos_token = "[EOS]"
+    unk_token = "[UNK]"
+
+    @classmethod
+    def add_cmdline_argument(cls, parser):
+        group = parser.add_argument_group("BPETextField")
+        group.add_argument("--vocab_path", type=str, required=True,
+                           help="The vocabulary file path.")
+        group.add_argument("--filtered", type=str2bool, default=False,
+                           help="Whether to filter the data with too long utterance/context. "
+                           "If the data is unfiltered, it will be truncated.")
+        group.add_argument("--max_len", type=int, default=256,
+                           help="The maximum length of context or knowledges.")
+        group.add_argument("--min_utt_len", type=int, default=1,
+                           help="The minimum length of utterance.")
+        group.add_argument("--max_utt_len", type=int, default=50,
+                           help="The maximum length of utterance.")
+        group.add_argument("--min_ctx_turn", type=int, default=1,
+                           help="The minimum turn of context.")
+        group.add_argument("--max_ctx_turn", type=int, default=16,
+                           help="The maximum turn of context.")
+        group.add_argument("--max_knowledge_num", type=int, default=16,
+                           help="The maximum number of knowledges.")
+        group.add_argument("--max_knowledge_len", type=int, default=16,
+                           help="The maximum length of each knowledges.")
+        group.add_argument("--tokenizer_type", type=str, default="Bert",
+                           choices=["Bert", "GPT2"],
+                           help="The type of tokenizer.")
+        return group
+
+    def __init__(self, hparams):
+        special_tokens = [self.pad_token, self.bos_token, self.eos_token, self.unk_token]
+        self.tokenizer = Tokenizer(vocab_path=hparams.vocab_path,
+                                   special_tokens=special_tokens,
+                                   tokenizer_type=hparams.tokenizer_type)
+
+        self.filtered = hparams.filtered
+        self.max_len = hparams.max_len
+        self.min_utt_len = hparams.min_utt_len
+        self.max_utt_len = hparams.max_utt_len
+        self.min_ctx_turn = hparams.min_ctx_turn
+        self.max_ctx_turn = hparams.max_ctx_turn - 1 # subtract reply turn
+        self.max_knowledge_num = hparams.max_knowledge_num
+        self.max_knowledge_len = hparams.max_knowledge_len
+        return
+
+    @property
+    def vocab_size(self):
+        return self.tokenizer.vocab_size
+
+    @property
+    def num_specials(self):
+        return len(self.special_tokens)
+
+    @property
+    def pad_id(self):
+        return self.tokenizer.convert_tokens_to_ids([self.pad_token])[0]
+
+    @property
+    def bos_id(self):
+        return self.tokenizer.convert_tokens_to_ids([self.bos_token])[0]
+
+    @property
+    def eos_id(self):
+        return self.tokenizer.convert_tokens_to_ids([self.eos_token])[0]
+
+    @property
+    def unk_id(self):
+        return self.tokenizer.convert_tokens_to_ids([self.unk_token])[0]
+
+    @property
+    def bot_id(self):
+        return 0
+
+    @property
+    def user_id(self):
+        return 1
+
+    @property
+    def knowledge_id(self):
+        return 2
+
+    def numericalize(self, tokens):
+        assert isinstance(tokens, list)
+        if len(tokens) == 0:
+            return []
+        element = tokens[0]
+        if isinstance(element, list):
+            return [self.numericalize(s) for s in tokens]
+        else:
+            return self.tokenizer.convert_tokens_to_ids(tokens)
+
+    def denumericalize(self, numbers):
+        assert isinstance(numbers, list)
+        if len(numbers) == 0:
+            return []
+        element = numbers[0]
+        if isinstance(element, list):
+            return [self.denumericalize(x) for x in numbers]
+        else:
+            return self.tokenizer.decode(
+                numbers, ignore_tokens=[self.bos_token, self.eos_token, self.pad_token])
+
+    def save_examples(self, examples, filename):
+        print(f"Saving examples to '{filename}' ...")
+        start = time.time()
+        if filename.endswith("pkl"):
+            with open(filename, "wb") as fp:
+                pickle.dump(examples, fp)
+        elif filename.endswith("jsonl"):
+            with open(filename, "w", encoding="utf-8") as fp:
+                for ex in examples:
+                    fp.write(json.dumps(ex) + "\n")
+        else:
+            raise ValueError(f"Unsport file format: {filename}")
+        elapsed = time.time() - start
+        print(f"Saved {len(examples)} examples (elapsed {elapsed:.2f}s)")
+
+    def load_examples(self, filename):
+        print(f"Loading examples from '{filename}' ...")
+        start = time.time()
+        if filename.endswith("pkl"):
+            with open(filename, "rb") as fp:
+                examples = pickle.load(fp)
+        else:
+            with open(filename, "r", encoding="utf-8") as fp:
+                examples = list(map(lambda s: json.loads(s.strip()), fp))
+        elapsed = time.time() - start
+        print(f"Loaded {len(examples)} examples (elapsed {elapsed:.2f}s)")
+        return examples
+
+    def utt_filter_pred(self, utt):
+        return self.min_utt_len <= len(utt) \
+            and (not self.filtered or len(utt) <= self.max_utt_len)
+
+    def utts_filter_pred(self, utts):
+        return self.min_ctx_turn <= len(utts) \
+            and (not self.filtered or len(utts) <= self.max_ctx_turn)
+
+    def build_example_multi_turn(self, req):
+        examples = []
+        src = [self.tokenizer.tokenize(s) for s in req["context"]]
+        src = [s[-self.max_utt_len:] for s in src[-self.max_ctx_turn:]]
+        src = [self.numericalize(s) + [self.eos_id] for s in src]
+        ex = {"src": src}
+        examples.append(ex)
+        return examples
+
+    def build_example_multi_turn_with_knowledge(self, req):
+        examples = []
+        src = [self.tokenizer.tokenize(s) for s in req["context"]]
+        src = [s[-self.max_utt_len:] for s in src[-self.max_ctx_turn:]]
+        src = [self.numericalize(s) + [self.eos_id] for s in src]
+        knowledge = [self.tokenizer.tokenize(k) for k in req["knowledge"]]
+        knowledge = [k[:self.max_knowledge_len] for k in knowledge]
+        knowledge = [self.numericalize(k) + [self.eos_id] for k in knowledge]
+        ex = {"src": src, "knowledge": knowledge}
+        examples.append(ex)
+        return examples
+
+    def build_examples_multi_turn(self, data_file, data_type="train"):
+        print(f"Reading examples from '{data_file}' ...")
+        examples = []
+        ignored = 0
+
+        with open(data_file, "r", encoding="utf-8") as f:
+            for line in tqdm(f, total=None):
+                src, tgt = line.strip("\n").split("\t")
+                tgt = self.tokenizer.tokenize(tgt)
+                src = [self.tokenizer.tokenize(s) for s in src.split(" __eou__ ")]
+
+                if (self.utts_filter_pred(src) and all(map(self.utt_filter_pred, src))
+                        and self.utt_filter_pred(tgt)) or data_type == "test":
+                    src = [s[-self.max_utt_len:] for s in src[-self.max_ctx_turn:]]
+                    src = [self.numericalize(s) + [self.eos_id] for s in src]
+                    tgt = [self.bos_id] + self.numericalize(tgt) + [self.eos_id]
+                    if data_type != "test":
+                        tgt = tgt[:self.max_utt_len + 2]
+                    ex = {"src": src, "tgt": tgt}
+                    examples.append(ex)
+                else:
+                    ignored += 1
+        print(f"Built {len(examples)} {data_type.upper()} examples ({ignored} filtered)")
+        return examples
+
+    def build_examples_multi_turn_with_knowledge(self, data_file, data_type="train"):
+        print(f"Reading examples from '{data_file}' ...")
+        examples = []
+        ignored = 0
+
+        with open(data_file, "r", encoding="utf-8") as f:
+            for line in tqdm(f, total=None):
+                knowledge, src, tgt = line.strip("\n").split("\t")
+                tgt = self.tokenizer.tokenize(tgt)
+                knowledge = [self.tokenizer.tokenize(k) for k in knowledge.split(" __eou__ ")]
+                knowledge = [k[:self.max_knowledge_len]
+                             for k in knowledge[-self.max_knowledge_num:]]
+                src = [self.tokenizer.tokenize(s) for s in src.split(" __eou__ ")]
+
+                if (self.utts_filter_pred(src) and all(map(self.utt_filter_pred, src)) 
+                        and self.utt_filter_pred(tgt)) or data_type == "test":
+                    src = [s[-self.max_utt_len:] for s in src[-self.max_ctx_turn:]]
+                    src = [self.numericalize(s) + [self.eos_id] for s in src]
+                    knowledge = [self.numericalize(k) + [self.eos_id] for k in knowledge]
+                    tgt = [self.bos_id] + self.numericalize(tgt) + [self.eos_id]
+                    if data_type != "test":
+                        tgt = tgt[:self.max_utt_len + 2]
+                    ex = {"src": src, "knowledge": knowledge, "tgt": tgt}
+                    examples.append(ex)
+                else:
+                    ignored += 1
+        print(f"Built {len(examples)} {data_type.upper()} examples ({ignored} filtered)")
+        return examples
+
+    def collate_fn_multi_turn(self, samples):
+        batch_size = len(samples)
+
+        src = [sp["src"] for sp in samples]
+
+        src_token, src_pos, src_turn, src_role = [], [], [], []
+        for utts in src:
+            utt_lens = [len(utt) for utt in utts]
+
+            # Token ids
+            src_token.append(list(chain(*utts))[-self.max_len:])
+
+            # Position ids
+            pos = [list(range(l)) for l in utt_lens]
+            src_pos.append(list(chain(*pos))[-self.max_len:])
+
+            # Turn ids
+            turn = [[len(utts) - i] * l for i, l in enumerate(utt_lens)]
+            src_turn.append(list(chain(*turn))[-self.max_len:])
+
+            # Role ids
+            role = [[self.bot_id if (len(utts) - i) % 2 == 0 else self.user_id] * l
+                    for i, l in enumerate(utt_lens)]
+            src_role.append(list(chain(*role))[-self.max_len:])
+
+        src_token = list2np(src_token, padding=self.pad_id)
+        src_pos = list2np(src_pos, padding=self.pad_id)
+        src_turn = list2np(src_turn, padding=self.pad_id)
+        src_role = list2np(src_role, padding=self.pad_id)
+
+        batch = {}
+        batch["src_token"] = src_token
+        batch["src_mask"] = (src_token != self.pad_id).astype("int64")
+        batch["src_pos"] = src_pos
+        batch["src_type"] = src_role
+        batch["src_turn"] = src_turn
+
+        if "tgt" in samples[0]:
+            tgt = [sp["tgt"] for sp in samples]
+
+            # Token ids & Label ids
+            tgt_token = list2np(tgt, padding=self.pad_id)
+
+            # Position ids
+            tgt_pos = np.zeros_like(tgt_token)
+            tgt_pos[:] = np.arange(tgt_token.shape[1], dtype=tgt_token.dtype)
+
+            # Turn ids
+            tgt_turn = np.zeros_like(tgt_token)
+
+            # Role ids
+            tgt_role = np.full_like(tgt_token, self.bot_id)
+
+            batch["tgt_token"] = tgt_token
+            batch["tgt_mask"] = (tgt_token != self.pad_id).astype("int64")
+            batch["tgt_pos"] = tgt_pos
+            batch["tgt_type"] = tgt_role
+            batch["tgt_turn"] = tgt_turn
+
+        return batch, batch_size
+
+    def collate_fn_multi_turn_with_knowledge(self, samples):
+        batch_size = len(samples)
+
+        src = [sp["src"] for sp in samples]
+        knowledge = [sp["knowledge"] for sp in samples]
+
+        src_token, src_pos, src_turn, src_role = [], [], [], []
+        for utts, ks in zip(src, knowledge):
+            utt_lens = [len(utt) for utt in utts]
+            k_lens = [len(k) for k in ks]
+
+            # Token ids
+            token = list(chain(*utts))[-self.max_len:]
+            token.extend(list(chain(*ks))[-self.max_len:])
+            src_token.append(token)
+
+            # Position ids
+            pos = list(chain(*[list(range(l)) for l in utt_lens]))[-self.max_len:]
+            pos.extend(list(chain(*[list(range(l)) for l in k_lens]))[-self.max_len:])
+            src_pos.append(pos)
+
+            # Turn ids
+            turn = list(chain(*[[len(utts) - i] * l for i, l in enumerate(utt_lens)]))[-self.max_len:]
+            turn.extend(list(chain(*[[i] * l for i, l in enumerate(k_lens)]))[-self.max_len:])
+            src_turn.append(turn)
+
+            # Role ids
+            role = list(chain(*[[self.bot_id if (len(utts)-i) % 2 == 0 else self.user_id] * l
+                                for i, l in enumerate(utt_lens)]))[-self.max_len:]
+            role.extend(list(chain(*[[self.knowledge_id] * l for l in k_lens]))[-self.max_len:])
+            src_role.append(role)
+        
+        src_token = list2np(src_token, padding=self.pad_id)
+        src_pos = list2np(src_pos, padding=self.pad_id)
+        src_turn = list2np(src_turn, padding=self.pad_id)
+        src_role = list2np(src_role, padding=self.pad_id)
+
+        batch = {}
+        batch["src_token"] = src_token
+        batch["src_mask"] = (src_token != self.pad_id).astype("int64")
+        batch["src_pos"] = src_pos
+        batch["src_type"] = src_role
+        batch["src_turn"] = src_turn
+
+        if "tgt" in samples[0]:
+            tgt = [sp["tgt"] for sp in samples]
+
+            # Token ids & Label ids
+            tgt_token = list2np(tgt, padding=self.pad_id)
+
+            # Position ids
+            tgt_pos = np.zeros_like(tgt_token)
+            tgt_pos[:] = np.arange(tgt_token.shape[1], dtype=tgt_token.dtype)
+
+            # Turn ids
+            tgt_turn = np.zeros_like(tgt_token)
+
+            # Role ids
+            tgt_role = np.full_like(tgt_token, self.bot_id)
+
+            batch["tgt_token"] = tgt_token
+            batch["tgt_mask"] = (tgt_token != self.pad_id).astype("int64")
+            batch["tgt_pos"] = tgt_pos
+            batch["tgt_type"] = tgt_role
+            batch["tgt_turn"] = tgt_turn
+
+        return batch, batch_size

PaddleNLP/Research/Dialogue-PLATO/plato/data/sampler.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Sampler class.
+"""
+
+import numpy as np
+
+
+class Sampler(object):
+    
+    def __init__(self):
+        return
+
+    def __len__(self):
+        raise NotImplementedError
+
+    def __iter__(self):
+        raise NotImplementedError
+
+
+class SequentialSampler(Sampler):
+
+    def __init__(self, dataset):
+        self.dataset = dataset
+        return
+
+    def __len__(self):
+        return len(self.dataset)
+
+    def __iter__(self):
+        return iter(range(len(self)))
+
+
+class RandomSampler(Sampler):
+
+    def __init__(self, dataset):
+        self.dataset = dataset
+        self.epoch = 0
+        return
+
+    def __len__(self):
+        return len(self.dataset)
+
+    def __iter__(self):
+        np.random.seed(self.epoch)
+        self.epoch += 1
+        return iter(np.random.permutation(len(self)))
+
+
+class SortedSampler(Sampler):
+    """ Sorted Sampler.
+    
+    Sort each block of examples by key.
+    """
+
+    def __init__(self, sampler, sort_pool_size, key="src"):
+        self.sampler = sampler
+        self.sort_pool_size = sort_pool_size
+        self.key = lambda idx: len(self.sampler.dataset[idx][key])
+        return
+
+    def __len__(self):
+        return len(self.sampler)
+
+    def __iter__(self):
+        pool = []
+        for idx in self.sampler:
+            pool.append(idx)
+            if len(pool) == self.sort_pool_size:
+                pool = sorted(pool, key=self.key)
+                for i in pool:
+                    yield i
+                pool = []
+        if len(pool) > 0:
+            pool = sorted(pool, key=self.key)
+            for i in pool:
+                yield i

PaddleNLP/Research/Dialogue-PLATO/plato/metrics/metrics.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Metrics class.
+"""
+
+from collections import Counter
+
+from nltk.translate import bleu_score
+from nltk.translate.bleu_score import SmoothingFunction
+import numpy as np
+
+
+def distinct(seqs):
+    """ Calculate intra/inter distinct 1/2. """
+    batch_size = len(seqs)
+    intra_dist1, intra_dist2 = [], []
+    unigrams_all, bigrams_all = Counter(), Counter()
+    for seq in seqs:
+        unigrams = Counter(seq)
+        bigrams = Counter(zip(seq, seq[1:]))
+        intra_dist1.append((len(unigrams)+1e-12) / (len(seq)+1e-5))
+        intra_dist2.append((len(bigrams)+1e-12) / (max(0, len(seq)-1)+1e-5))
+
+        unigrams_all.update(unigrams)
+        bigrams_all.update(bigrams)
+
+    inter_dist1 = (len(unigrams_all)+1e-12) / (sum(unigrams_all.values())+1e-5)
+    inter_dist2 = (len(bigrams_all)+1e-12) / (sum(bigrams_all.values())+1e-5)
+    intra_dist1 = np.average(intra_dist1)
+    intra_dist2 = np.average(intra_dist2)
+    return intra_dist1, intra_dist2, inter_dist1, inter_dist2
+
+
+def bleu(hyps, refs):
+    """ Calculate bleu 1/2. """
+    bleu_1 = []
+    bleu_2 = []
+    for hyp, ref in zip(hyps, refs):
+        try:
+            score = bleu_score.sentence_bleu(
+                [ref], hyp,
+                smoothing_function=SmoothingFunction().method7,
+                weights=[1, 0, 0, 0])
+        except:
+            score = 0
+        bleu_1.append(score)
+        try:
+            score = bleu_score.sentence_bleu(
+                [ref], hyp,
+                smoothing_function=SmoothingFunction().method7,
+                weights=[0.5, 0.5, 0, 0])
+        except:
+            score = 0
+        bleu_2.append(score)
+    bleu_1 = np.average(bleu_1)
+    bleu_2 = np.average(bleu_2)
+    return bleu_1, bleu_2

PaddleNLP/Research/Dialogue-PLATO/plato/metrics/metrics_tracker.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+MetricsTracker class
+"""
+
+from collections import defaultdict
+import math
+
+
+class MetricsTracker(object):
+    """ Tracking metrics. """
+
+    def __init__(self):
+        self.metrics_val = defaultdict(float)
+        self.metrics_avg = defaultdict(float)
+        self.num_samples = 0
+
+    def update(self, metrics, num_samples):
+        for key, val in metrics.items():
+            if val is not None:
+                val = float(val)
+                self.metrics_val[key] = val
+                avg_val = (self.metrics_avg.get(key, 0) * self.num_samples +
+                           val * num_samples) / (self.num_samples + num_samples)
+                self.metrics_avg[key] = avg_val
+        self.num_samples += num_samples
+
+    def clear(self):
+        self.metrics_val = defaultdict(float)
+        self.metrics_avg = defaultdict(float)
+        self.num_samples = 0
+
+    def items(self):
+        return self.metrics_avg.items()
+
+    def get(self, name):
+        if self.num_samples == 0:
+            raise ValueError("There is no data in Metrics.")
+        return self.metrics_avg.get(name)
+
+    def state_dict(self):
+        return {
+            "metrics_val": self.metrics_val,
+            "metrics_avg": self.metrics_avg,
+            "num_samples": self.num_samples,
+        }
+
+    def load_state_dict(self, state_dict):
+        self.metrics_val = state_dict["metrics_val"]
+        self.metrics_avg = state_dict["metrics_avg"]
+        self.num_samples = state_dict["num_samples"]
+
+    def value(self):
+        metric_strs = []
+        for key, val in self.metrics_val.items():
+            metric_str = f"{key.upper()}-{val:.3f}"
+            metric_strs.append(metric_str)
+        if "token_nll" in self.metrics_val:
+            metric_str = f"TOKEN_PPL-{math.exp(self.metrics_val['token_nll']):.3f}"
+            metric_strs.append(metric_str)
+        metric_strs = "   ".join(metric_strs)
+        return metric_strs
+
+    def summary(self):
+        metric_strs = []
+        for key, val in self.metrics_avg.items():
+            metric_str = f"{key.upper()}-{val:.3f}"
+            metric_strs.append(metric_str)
+        if "token_nll" in self.metrics_avg:
+            metric_str = f"TOKEN_PPL-{math.exp(self.metrics_avg['token_nll']):.3f}"
+            metric_strs.append(metric_str)
+        metric_strs = "   ".join(metric_strs)
+        return metric_strs

PaddleNLP/Research/Dialogue-PLATO/plato/models/__init__.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Loading models.
+"""
+
+import plato.models.unified_transformer

PaddleNLP/Research/Dialogue-PLATO/plato/models/generator.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Generator class.
+"""
+
+import bisect
+import math
+import sys
+
+import numpy as np
+import paddle.fluid as fluid
+import paddle.fluid.layers as layers
+from paddle.fluid.framework import Variable
+
+from plato.args import str2bool
+import plato.modules.functions as F
+
+
+def repeat(var, times):
+    if isinstance(var, list):
+        return [repeat(x, times) for x in var]
+    elif isinstance(var, dict):
+        return {k: repeat(v, times) for k, v in var.items()}
+    elif isinstance(var, Variable):
+        var = F.unsqueeze(var, [1])
+        expand_times = [1] * len(var.shape)
+        expand_times[1] = times
+        dtype = var.dtype
+        var = layers.cast(var, "float32")
+        var = layers.expand(var, expand_times)
+        shape = [var.shape[0] * var.shape[1]] + var.shape[2:]
+        var = layers.reshape(var, shape)
+        var = layers.cast(var, dtype)
+        return var
+    else:
+        return var
+
+
+def gather(var, idx):
+    if isinstance(var, list):
+        return [gather(x, idx) for x in var]
+    elif isinstance(var, dict):
+        return {k: gather(v, idx) for k, v in var.items()}
+    elif isinstance(var, Variable):
+        out = layers.gather(var, idx)
+        return out
+    else:
+        return var
+
+
+class Generator(object):
+    """ Genrator class. """
+
+    _registry = dict()
+
+    @classmethod
+    def register(cls, name):
+        Generator._registry[name] = cls
+        return
+
+    @staticmethod
+    def by_name(name):
+        return Generator._registry[name]
+
+    @staticmethod
+    def create(hparams, *args, **kwargs):
+        """ Create generator. """
+        generator_cls = Generator.by_name(hparams.generator)
+        return generator_cls(hparams, *args, **kwargs)
+
+    @classmethod
+    def add_cmdline_argument(cls, parser):
+        group = parser.add_argument_group("Generator")
+        group.add_argument("--generator", type=str, default="BeamSearch",
+                           choices=["TopKSampling", "TopPSampling", "GreedySampling",
+                                    "BeamSearch"])
+        group.add_argument("--min_gen_len", type=int, default=1,
+                           help="The minimum length of generated response.")
+        group.add_argument("--max_gen_len", type=int, default=30,
+                           help="The maximum length of generated response.")
+        args, _ = parser.parse_known_args()
+        generator_cls = cls.by_name(args.generator)
+        generator_cls.add_cmdline_argument(group)
+        return group
+
+    def __init__(self, hparams, bpe):
+        self.vocab_size = bpe.vocab_size
+        self.bos_id = bpe.bos_id
+        self.eos_id = bpe.eos_id
+        self.unk_id = bpe.unk_id
+        self.pad_id = bpe.pad_id
+        self.min_gen_len = hparams.min_gen_len
+        self.max_gen_len = hparams.max_gen_len
+        assert 1 <= self.min_gen_len <= self.max_gen_len
+        return
+
+    def __call__(self, step_fn, state):
+        """
+        Running generation.
+
+        @param : step_fn : decoding one step
+        @type : function
+
+        @param : state : initial state
+        @type : dict
+        """
+        raise NotImplementedError
+
+
+class Sampling(Generator):
+    """ Sampling Generator. """
+
+    @classmethod
+    def add_cmdline_argument(cls, group):
+        group.add_argument("--ignore_unk", type=str2bool, default=True,
+                           help="Whether to ignore unkown token in generation.")
+        group.add_argument("--sampling_temperature", type=float, default=1.0)
+        return group
+
+    def __init__(self, hparams, bpe):
+        super().__init__(hparams, bpe)
+        self.ignore_unk = hparams.ignore_unk
+        self.temperature = hparams.sampling_temperature
+        return
+
+    def _sampling(self, scores):
+        """ Sampling function. """
+        raise NotImplementedError
+
+    def __call__(self, step_fn, state):
+        """
+        Running generation.
+
+        @param : step_fn : decoding one step
+        @type : function
+
+        @param : state : initial state
+        @type : dict
+        """
+        batch_size = state["batch_size"]
+        vocab_size = self.vocab_size
+
+        pos_index = layers.range(0, batch_size, 1, dtype="int64")
+        pos_index = layers.scale(pos_index, vocab_size)
+
+        # shape: [batch_size, beam_size, 1]
+        predictions = layers.fill_constant(shape=[batch_size, 1],
+                                           dtype="int64",
+                                           value=self.bos_id)
+        sequence_scores = layers.fill_constant(shape=[batch_size],
+                                               dtype="float32",
+                                               value=0.0)
+
+        unk_penalty = np.zeros(vocab_size, dtype="float32")
+        unk_penalty[self.unk_id] = -1e10
+        unk_penalty = layers.assign(unk_penalty)
+
+        eos_penalty = np.zeros(vocab_size, dtype="float32")
+        eos_penalty[self.eos_id] = -1e10
+        eos_penalty = layers.assign(eos_penalty)
+
+        scores_after_end = np.full(vocab_size, -1e10, dtype="float32")
+        scores_after_end[self.pad_id] = 0
+        scores_after_end = layers.assign(scores_after_end)
+
+        # initial input
+        for step in range(1, self.max_gen_len + 1):
+            pre_ids = predictions[:, -1:]
+            state["pred_token"] = F.unsqueeze(pre_ids, [2])
+            if step > 1:
+                state["pred_mask"] = 1 - F.equal(state["pred_token"], self.pad_id)
+                state["pred_pos"] = state["pred_pos"] + 1
+            scores, state = step_fn(state)
+
+            # Generate next
+            # scores shape: [batch_size, vocab_size]
+            if self.ignore_unk:
+                scores = scores + unk_penalty
+
+            if step <= self.min_gen_len:
+                scores = scores + eos_penalty
+
+            # previous token is [PAD] or [EOS]
+            # shape: [batch_size, 1]
+            pre_eos_mask = F.equal(pre_ids, self.eos_id) + F.equal(pre_ids, self.pad_id)
+            scores = scores * (1 - pre_eos_mask) + \
+                layers.expand(pre_eos_mask, [1, vocab_size]) * scores_after_end
+
+            scores = scores / self.temperature
+            preds = self._sampling(scores)
+
+            predictions = layers.concat([predictions, F.unsqueeze(preds, [1])], axis=1)
+
+            scores = layers.reshape(scores, [batch_size * vocab_size])
+            preds = preds + pos_index
+            scores = gather(scores, preds)
+            sequence_scores = sequence_scores + scores
+
+        results = {
+            "preds": predictions,
+            "scores": sequence_scores
+        }
+        return results
+
+
+class GreedySampling(Sampling):
+    """ Greedy sampling. """
+
+    @classmethod
+    def add_cmdline_argument(cls, group):
+        return Sampling.add_cmdline_argument(group)
+
+    def _sampling(self, logits):
+        """ Implement greedy sampling. """
+        preds = layers.argmax(logits, axis=1)
+        return preds
+
+
+class TopKSampling(Sampling):
+    """ Top-k sampling. """
+
+    @classmethod
+    def add_cmdline_argument(cls, group):
+        Sampling.add_cmdline_argument(group)
+        group.add_argument("--top_k_ratio", type=float, default=None)
+        group.add_argument("--top_k_num", type=int, default=None)
+        return group
+
+    def __init__(self, hparams, bpe):
+        super().__init__(hparams, bpe)
+        assert hparams.top_k_ratio is not None or hparams.top_k_num is not None
+        if hparams.top_k_num is not None:
+            self.top_k_num = hparams.top_k_num
+        else:
+            self.top_k_num = math.floor(hparams.top_k_ratio * self.vocab_size)
+        assert self.top_k_num >= 1
+        return
+
+    def _sampling(self, logits):
+        """ Implement top-k sampling. """
+        probs = layers.softmax(logits, axis=1)
+        probs, indices = layers.topk(probs, self.top_k_num)
+        probs = probs / layers.reduce_sum(probs, dim=1, keep_dim=True)
+        preds = []
+        for p, ids in zip(probs.numpy(), indices.numpy()):
+            o = np.random.choice(ids, p=p)
+            preds.append(o)
+        preds = np.array(preds, dtype="int64")
+        return fluid.dygraph.to_variable(preds)
+
+
+class TopPSampling(Sampling):
+    """ Top-p sampling. """
+
+    @classmethod
+    def add_cmdline_argument(cls, group):
+        Sampling.add_cmdline_argument(group)
+        group.add_argument("--top_p_ratio", type=float, default=1.0)
+        return group
+
+    def __init__(self, hparams, bpe):
+        super().__init__(hparams, bpe)
+        self.top_p_ratio = hparams.top_p_ratio
+        return
+
+    def _sampling(self, logits):
+        """ Implement top-k sampling. """
+        probs = layers.softmax(logits, axis=1)
+        preds = []
+        for p in probs.numpy():
+            ids = np.argsort(-p)
+            p = p[ids]
+            c_p = np.cumsum(p)
+            i = bisect.bisect_right(c_p, self.top_p_ratio) + 1
+            o = np.random.choice(ids[:i], p=p[:i]/np.sum(p[:i]))
+            preds.append(o)
+        preds = np.array(preds, dtype="int64")
+        return fluid.dygraph.to_variable(preds)
+
+
+class BeamSearch(Generator):
+    """ BeamSearch generator. """
+
+    @classmethod
+    def add_cmdline_argument(cls, group):
+        group.add_argument("--beam_size", type=int, default=5,
+                           help="The beam size in beam search.")
+        group.add_argument("--length_average", type=str2bool, default=False,
+                           help="Whether to use length average.")
+        group.add_argument("--length_penalty", type=float, default=-1.0,
+                           help="The parameter(alpha) of length penalty.")
+        group.add_argument("--ignore_unk", type=str2bool, default=True,
+                           help="Whether to ignore unkown token in generation.")
+        return group
+
+    def __init__(self, hparams, bpe):
+        super().__init__(hparams, bpe)
+        self.beam_size = hparams.beam_size
+        self.length_average = hparams.length_average
+        self.length_penalty = hparams.length_penalty
+        self.ignore_unk = hparams.ignore_unk
+        return
+
+    def __call__(self, step_fn, state):
+        """
+        Running beam search.
+
+        @param : step_fn : decoding one step
+        @type : function
+
+        @param : state : initial state
+        @type : dict
+        """
+        batch_size = state["batch_size"]
+        beam_size = self.beam_size
+
+        # shape: [batch_size, 1]
+        pos_index = layers.range(0, batch_size, 1, dtype="int64")
+        pos_index = layers.scale(pos_index, beam_size)
+        pos_index = F.unsqueeze(pos_index, [1])
+
+        # shape: [batch_size, beam_size, 1]
+        predictions = layers.fill_constant(shape=[batch_size, beam_size, 1],
+                                           dtype="int64",
+                                           value=self.bos_id)
+
+        # initial input
+        state["pred_token"] = predictions[:, :1]
+        # shape: [batch_size, vocab_size]
+        scores, state = step_fn(state)
+
+        unk_penalty = np.zeros(self.vocab_size, dtype="float32")
+        unk_penalty[self.unk_id] = -1e10
+        unk_penalty = layers.assign(unk_penalty)
+
+        eos_penalty = np.zeros(self.vocab_size, dtype="float32")
+        eos_penalty[self.eos_id] = -1e10
+        eos_penalty = layers.assign(eos_penalty)
+
+        scores_after_end = np.full(self.vocab_size, -1e10, dtype="float32")
+        scores_after_end[self.pad_id] = 0
+        scores_after_end = layers.assign(scores_after_end)
+
+        if self.ignore_unk:
+            scores = scores + unk_penalty
+        scores = scores + eos_penalty
+
+        # shape: [batch_size, beam_size]
+        sequence_scores, preds = layers.topk(scores, self.beam_size)
+
+        predictions = layers.concat([predictions, F.unsqueeze(preds, [2])], axis=2)
+        state = repeat(state, beam_size)
+
+        parent_idx_list = []
+        pred_list = []
+
+        for step in range(2, self.max_gen_len + 1):
+            pre_ids = predictions[:, :, -1:]
+            state["pred_token"] = layers.reshape(pre_ids, shape=[batch_size * beam_size, 1, 1])
+            state["pred_mask"] = 1 - F.equal(state["pred_token"], self.pad_id)
+            state["pred_pos"] = state["pred_pos"] + 1
+            scores, state = step_fn(state)
+
+            # Generate next
+            # scores shape: [batch_size, beam_size, vocab_size]
+            if self.ignore_unk:
+                scores = scores + unk_penalty
+
+            if step <= self.min_gen_len:
+                scores = scores + eos_penalty
+
+            scores = layers.reshape(scores, shape=[batch_size, beam_size, self.vocab_size])
+
+            # previous token is [PAD] or [EOS]
+            pre_eos_mask = F.equal(pre_ids, self.eos_id) + F.equal(pre_ids, self.pad_id)
+
+            scores = scores * (1 - pre_eos_mask) + \
+                layers.expand(pre_eos_mask, [1, 1, self.vocab_size]) * scores_after_end
+            if self.length_average:
+                scaled_value = pre_eos_mask + (1 - pre_eos_mask) * (1 - 1 / step)
+                sequence_scores = F.unsqueeze(sequence_scores, [2]) * scaled_value
+                scaled_value = pre_eos_mask + (1 - pre_eos_mask) * (1 / step)
+                scores = scores * scaled_value
+            elif self.length_penalty >= 0.0:
+                scaled_value = pre_eos_mask + (1 - pre_eos_mask) * \
+                    (math.pow((4 + step) / (5 + step), self.length_penalty))
+                sequence_scores = layers.elementwise_mul(scaled_value, sequence_scores, axis=0)
+                scaled_value = pre_eos_mask + (1 - pre_eos_mask) * \
+                    (math.pow(1 / (5 + step), self.length_penalty))
+                scores = scores * scaled_value
+            scores = layers.elementwise_add(scores, sequence_scores, axis=0)
+            scores = layers.reshape(scores, shape=[batch_size, beam_size * self.vocab_size])
+
+            topk_scores, topk_indices = layers.topk(scores, beam_size)
+            vocab_size = layers.fill_constant(shape=[1], dtype="int64", value=self.vocab_size)
+            parent_idx = layers.elementwise_floordiv(topk_indices, vocab_size)
+            preds = layers.elementwise_mod(topk_indices, vocab_size)
+
+            # Gather state / sequence_scores
+            parent_idx = layers.elementwise_add(parent_idx, pos_index, axis=0)
+            parent_idx = layers.reshape(parent_idx, [batch_size * beam_size])
+            state = gather(state, parent_idx)
+            sequence_scores = topk_scores
+
+            predictions = layers.reshape(predictions, shape=[batch_size * beam_size, step])
+            predictions = gather(predictions, parent_idx)
+            predictions = layers.reshape(predictions, shape=[batch_size, beam_size, step])
+            predictions = layers.concat([predictions, F.unsqueeze(preds, [2])], axis=2)
+
+        pre_ids = predictions[:, :, -1]
+        pre_eos_mask = F.equal(pre_ids, self.eos_id) + F.equal(pre_ids, self.pad_id)
+        sequence_scores = sequence_scores * pre_eos_mask + layers.scale(1 - pre_eos_mask, -1e10)
+
+        _, indices = layers.argsort(sequence_scores, axis=1)
+        indices = indices + pos_index
+        indices = layers.reshape(indices, [-1])
+        sequence_scores = layers.reshape(sequence_scores, [batch_size * beam_size])
+        predictions = layers.reshape(predictions, [batch_size * beam_size, -1])
+        sequence_scores = gather(sequence_scores, indices)
+        predictions = layers.gather(predictions, indices)
+        sequence_scores = layers.reshape(sequence_scores, [batch_size, beam_size])
+        predictions = layers.reshape(predictions, [batch_size, beam_size, -1])
+
+        results = {
+            "preds": predictions[:, -1],
+            "scores": sequence_scores[:, -1]
+        }
+        return results
+
+BeamSearch.register("BeamSearch")
+GreedySampling.register("GreedySampling")
+TopKSampling.register("TopKSampling")
+TopPSampling.register("TopPSampling")

PaddleNLP/Research/Dialogue-PLATO/plato/models/model_base.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Model base
+"""
+
+import paddle.fluid as fluid
+from paddle.fluid.dygraph import parallel_helper
+
+
+class ModelBase(fluid.dygraph.Layer):
+    """
+    Basic model wrapper for static graph and dygrpah.
+    """
+    _registry = dict()
+
+    @classmethod
+    def register(cls, name):
+        ModelBase._registry[name] = cls
+        return
+
+    @staticmethod
+    def by_name(name):
+        return ModelBase._registry[name]
+
+    @staticmethod
+    def create(name_scope, hparams, *args, **kwargs):
+        model_cls = ModelBase.by_name(hparams.model)
+        return model_cls(name_scope, hparams, *args, **kwargs)
+
+    @classmethod
+    def add_cmdline_argument(cls, parser):
+        """ Add cmdline argument. """
+        group = parser.add_argument_group("Model")
+        group.add_argument("--init_checkpoint", type=str, default=None)
+        group.add_argument("--model", type=str, default="UnifiedTransformer",
+                           choices=["UnifiedTransformer"])
+        args, _ = parser.parse_known_args()
+        model_cls = ModelBase.by_name(args.model)
+        model_cls.add_cmdline_argument(group)
+        return group
+
+    def __init__(self, name_scope, hparams):
+        super().__init__(name_scope)
+        self.init_checkpoint = hparams.init_checkpoint
+        return
+
+    def __call__(self, *args, **kwargs):
+        """ Re-implement __call__ function in dygraph mode. """
+        if not self._built:
+            self._build_once(*args, **kwargs)
+            self._built = True
+
+        outputs = self.forward(*args, **kwargs)
+        return outputs
+
+    def _build_once(self, inputs, *args, **kwargs):
+        """
+        Build only once.
+
+        1. Initialize models's parameters.
+        2. Boardcast parameters if in data parallel mode.
+        3. Load saved parameters
+        """
+        # Initial parameters.
+        self._create_parameters()
+
+        if parallel_helper._is_data_parallel_mode():
+            parallel_helper._broadcast_parameters(self._parameters.values())
+
+        # Load persitables
+        self._load_params()
+        return
+
+    def _create_parameters(self):
+        """ Create model's paramters. """
+        raise NotImplementedError
+
+    def _load_params(self):
+        """ Load saved paramters. """
+        raise NotImplementedError
+
+    def _forward(self, inputs, is_training):
+        """ Real forward process of model in different mode(train/test). """
+        raise NotImplementedError
+
+    def _collect_metrics(self, inputs, outputs):
+        """ Calculate loss function by using inputs and outputs. """
+        raise NotImplementedError
+
+    def _optimize(self, loss):
+        """ Optimize loss function and update model. """
+        raise NotImplementedError
+
+    def _infer(self, inputs):
+        """ Real inference process of model. """
+        raise NotImplementedError
+
+    def forward(self, inputs, is_training=False):
+        """
+        Forward process, include real forward, collect metrices and optimize(optional)
+
+        @params : inputs : input data
+        @type : dict of numpy.ndarray/int/float/...
+        """
+        if is_training:
+            self.train()
+        else:
+            self.eval()
+
+        outputs = self._forward(inputs, is_training)
+        metrics = self._collect_metrics(inputs, outputs)
+        loss = metrics["loss"]
+        if is_training:
+            self._optimize(loss)
+
+        metrics = {k: v.numpy() for k, v in metrics.items()}
+        return metrics
+
+    def infer(self, inputs):
+        """
+        Inference process.
+
+        @params : inputs : input data
+        @type : dict of numpy.ndarray/int/float/...
+        """
+        if not self._built:
+            self._build_once(inputs)
+            self._built = True
+
+        self.eval()
+        results = self._infer(inputs)
+        results = {name: results[name].numpy() for name in results}
+        return results

PaddleNLP/Research/Dialogue-PLATO/plato/modules/embedder.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Embedder class.
+"""
+
+import paddle.fluid as fluid
+from paddle.fluid.dygraph import Embedding
+from paddle.fluid.dygraph import Layer
+import paddle.fluid.layers as layers
+
+import plato.modules.functions as F
+
+
+class Embedder(Layer):
+    """
+    Composite embedding layer.
+    """
+
+    def __init__(self,
+                 name_scope,
+                 hidden_dim,
+                 num_token_embeddings,
+                 num_pos_embeddings,
+                 num_type_embeddings,
+                 num_turn_embeddings,
+                 padding_idx=None,
+                 dropout=0.1,
+                 pos_trainable=False):
+        super().__init__(name_scope)
+
+        self.token_embedding = Embedding(name_scope=self.full_name(),
+                                         size=[num_token_embeddings, hidden_dim])
+        self.pos_embedding = Embedding(name_scope=self.full_name(),
+                                       size=[num_pos_embeddings, hidden_dim],
+                                       param_attr=fluid.ParamAttr(trainable=pos_trainable))
+        self.type_embedding = Embedding(name_scope=self.full_name(),
+                                        size=[num_type_embeddings, hidden_dim])
+        self.turn_embedding = Embedding(name_scope=self.full_name(),
+                                        size=[num_turn_embeddings, hidden_dim])
+        self.dropout = dropout
+        return
+
+    def forward(self, token_inp, pos_inp, type_inp, turn_inp):
+        embed = self.token_embedding(token_inp) + \
+            self.pos_embedding(pos_inp) + \
+            self.type_embedding(type_inp) + \
+            self.turn_embedding(turn_inp)
+        embed = F.dropout(embed, self.dropout)
+        return embed
+
+
+def main():
+    import numpy as np
+
+    place = fluid.CPUPlace()
+    with fluid.dygraph.guard(place):
+        model = Embedder("Embedder", 10, 20, 20, 20, 20)
+        token_inp = fluid.dygraph.to_variable(np.random.randint(0, 19, [10, 10, 1]).astype("int64"))
+        pos_inp = fluid.dygraph.to_variable(np.random.randint(0, 19, [10, 10, 1]).astype("int64"))
+        type_inp = fluid.dygraph.to_variable(np.random.randint(0, 19, [10, 10, 1]).astype("int64"))
+        turn_inp = fluid.dygraph.to_variable(np.random.randint(0, 19, [10, 10, 1]).astype("int64"))
+        out = model(token_inp, pos_inp, type_inp, turn_inp)
+        print(out)
+
+
+if __name__ == "__main__":
+    main()

PaddleNLP/Research/Dialogue-PLATO/plato/modules/feedforward.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+FeedForward class.
+"""
+
+import paddle.fluid as fluid
+from paddle.fluid.dygraph import FC
+from paddle.fluid.dygraph import Layer
+import paddle.fluid.layers as layers
+
+import plato.modules.functions as F
+
+
+class FeedForward(Layer):
+    """
+    Positional feed forward layer.
+    """
+
+    def __init__(self, name_scope, hidden_dim, inner_dim, dropout):
+        super().__init__(name_scope)
+
+        self.hidden_dim = hidden_dim
+        self.inner_dim = inner_dim
+        self.linear_hidden = FC(name_scope=self.full_name(),
+                                size=inner_dim,
+                                num_flatten_dims=2,
+                                act="gelu")
+        self.linear_out = FC(name_scope=self.full_name(),
+                             size=hidden_dim,
+                             num_flatten_dims=2)
+        self.dropout = dropout
+        return
+
+    def forward(self, x):
+        out = self.linear_hidden(x)
+        out = F.dropout(out, self.dropout)
+        out = self.linear_out(out)
+        return out
+
+
+def main():
+    import numpy as np
+
+    place = fluid.CPUPlace()
+    with fluid.dygraph.guard(place):
+        model = FeedForward("FeedForward", 10, 20, 0.5)
+        inp = np.random.rand(2, 3, 10).astype("float32")
+        inp = fluid.dygraph.to_variable(inp)
+        out = model(inp)
+        print(out)
+
+
+if __name__ == "__main__":
+    main()

PaddleNLP/Research/Dialogue-PLATO/plato/modules/functions.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Helpful functions.
+"""
+
+import numpy as np
+import paddle.fluid as fluid
+import paddle.fluid.layers as layers
+
+
+def unsqueeze(input, axes):
+    """ Implement unsqueeze in dygraph mode. """
+    # return layers.unsqueeze(input, axes)
+    # op:unsqueeze has bug in dygraph
+    axes = [axis if axis >= 0 else axis + len(input.shape) + 1 for axis in axes]
+    axes = sorted(axes, reverse=True)
+    shape = list(input.shape)
+    for axis in axes:
+        shape.insert(axis, 1)
+    return layers.reshape(input, shape)
+
+
+def gumbel_softmax(input, tau=1, eps=1e-10):
+    """ Basic implement of gumbel_softmax. """
+    U = fluid.dygraph.to_variable(np.random.rand(*input.shape))
+    # U = layers.uniform_random(input.shape, dtype=input.dtype, min=0.0, max=1.0)
+    # U.stop_gradient = True
+    gumbel = 0.0 - layers.log(eps - layers.log(U + eps))
+    y = input + gumbel
+    return layers.softmax(y / tau)
+
+
+def equal(x, y, dtype=None):
+    """ Implement equal in dygraph mode. """
+    # if not isinstance(y, fluid.framework.Variable):
+    #     y = layers.fill_constant(x.shape, x.dtype, y)
+    # return layers.cast(layers.equal(x, y), dtype)
+    if dtype is None:
+        dtype = "float32"
+    if isinstance(x, fluid.framework.Variable):
+        x = x.numpy()
+    if isinstance(y, fluid.framework.Variable):
+        y = y.numpy()
+    out = np.equal(x, y).astype(dtype)
+    return fluid.dygraph.to_variable(out)
+
+
+def not_equal(x, y, dtype=None):
+    """ Implement not_equal in dygraph mode. """
+    return 1 - equal(x, y, dtype)
+
+
+def dropout(x, p):
+    """ Implement dropout function like tensorflow/pytorch. """
+    return layers.dropout(x, p, dropout_implementation="upscale_in_train")

PaddleNLP/Research/Dialogue-PLATO/plato/modules/layer_norm.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+LayerNorm layer.
+"""
+
+# from paddle.fluid.dygraph import LayerNorm
+
+from six.moves import reduce
+
+import paddle.fluid as fluid
+import paddle.fluid.layers as layers
+from paddle.fluid.dygraph import Layer
+import logging
+
+class LayerNorm(Layer):
+    """ Implement LayerNorm in dygraph mode. """
+
+    def __init__(self,
+                 name_scope,
+                 scale=True,
+                 shift=True,
+                 begin_norm_axis=1,
+                 epsilon=1e-05,
+                 param_attr=None,
+                 bias_attr=None,
+                 act=None):
+        super().__init__(name_scope)
+        self._scale = scale
+        self._shift = shift
+        self._begin_norm_axis = begin_norm_axis
+        self._epsilon = epsilon
+        self._param_attr = param_attr
+        self._bias_attr = bias_attr
+        self._act = act
+        return
+
+    def _build_once(self, input):
+        """ Create parameters. """
+        self._dtype = self._helper.input_dtype(input)
+        input_shape = input.shape
+        param_shape = [
+            reduce(lambda x, y: x * y, input_shape[self._begin_norm_axis:])
+        ]
+        if self._scale:
+            self._scale_w = self.create_parameter(
+                attr=self._param_attr,
+                shape=param_shape,
+                dtype=self._dtype,
+                default_initializer=fluid.initializer.Constant(1.0))
+        else:
+            if self._param_attr:
+                logging.warn("param_attr are only avaliable with scale is True")
+
+        if self._shift:
+            assert self._bias_attr is not False
+            self._bias_w = self.create_parameter(
+                attr=self._bias_attr,
+                shape=param_shape,
+                dtype=self._dtype,
+                is_bias=True)
+        else:
+            if self._bias_attr:
+                logging.warn("bias_attr are only avaliable with shift is True")
+        return
+
+    def forward(self, x):
+        """ Forward process of LayerNorm. """
+        mean = layers.reduce_mean(x,
+                                  dim=list(range(self._begin_norm_axis, len(x.shape))),
+                                  keep_dim=True)
+        shift_x = layers.elementwise_sub(x=x, y=mean, axis=0)
+        variance = layers.reduce_mean(layers.square(shift_x),
+                                      dim=list(range(self._begin_norm_axis, len(x.shape))),
+                                      keep_dim=True)
+        r_stdev = layers.rsqrt(variance + self._epsilon)
+        norm_x = layers.elementwise_mul(x=shift_x, y=r_stdev, axis=0)
+        out = layers.elementwise_mul(x=norm_x, y=self._scale_w, axis=-1)
+        out = layers.elementwise_add(x=out, y=self._bias_w, axis=-1)
+        return out

PaddleNLP/Research/Dialogue-PLATO/plato/modules/multihead_attention.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+MultiheadAttention class.
+"""
+
+import paddle.fluid as fluid
+from paddle.fluid.dygraph import Layer
+from paddle.fluid.dygraph import FC
+import paddle.fluid.layers as layers
+
+import plato.modules.functions as F
+
+
+class MultiheadAttention(Layer):
+    """
+    Multi head attention layer.
+    """
+
+    def __init__(self, name_scope, hidden_dim, num_heads, dropout):
+        assert hidden_dim % num_heads == 0
+        super().__init__(name_scope)
+
+        self.hidden_dim = hidden_dim
+        self.num_heads = num_heads
+        self.head_dim = hidden_dim // num_heads
+        self.scale = self.head_dim ** -0.5
+        self.linear_qkv = FC(name_scope=self.full_name(),
+                             size=hidden_dim * 3,
+                             num_flatten_dims=2)
+        self.linear_out = FC(name_scope=self.full_name(),
+                             size=hidden_dim,
+                             num_flatten_dims=2)
+        self.dropout = dropout
+        return
+
+    def _split_heads(self, x, is_key=False):
+        x = layers.reshape(
+            x=x, shape=[0, 0, self.num_heads, self.head_dim]
+        )
+        x = layers.transpose(x=x, perm=[0, 2, 3, 1] if is_key else [0, 2, 1, 3])
+        return x
+
+    def _merge_heads(self, x):
+        x = layers.transpose(x=x, perm=[0, 2, 1, 3])
+        x = layers.reshape(x=x, shape=[0, 0, self.hidden_dim])
+        return x
+
+    def _attn(self, query, key, value, mask):
+        # shape: [batch_size, num_head, seq_len, seq_len]
+        scores = layers.matmul(x=query, y=key, alpha=self.scale)
+
+        if mask is not None:
+            mask = F.unsqueeze(mask, [1])
+            mask = layers.expand(mask, [1, self.num_heads, 1, 1])
+            mask.stop_gradient = True
+            scores = (1 - mask) * scores + layers.scale(mask, scale=-1e10)
+
+        attn = layers.softmax(scores, axis=-1)
+        attn = F.dropout(attn, self.dropout)
+
+        if mask is not None:
+            attn = (1 - mask) * attn
+
+        out = layers.matmul(x=attn, y=value)
+        return out
+
+    def forward(self, inp, mask=None, cache=None):
+        """ Forward process of self attention. """
+        # shape: [batch_size, seq_len, 3 * hidden_dim]
+        qkv = self.linear_qkv(inp)
+        query, key, value = layers.split(qkv, num_or_sections=3, dim=2)
+
+
+        # shape: [batch_size, num_head, seq_len, head_dim]
+        query = self._split_heads(query)
+        # shape: [batch_size, num_head, head_dim, seq_len]
+        key = self._split_heads(key, is_key=True)
+        # shape: [batch_size, num_head, seq_len, head_dim]
+        value = self._split_heads(value)
+
+        if cache is not None:
+            if "key" in cache and "value" in cache:
+                key = layers.concat([cache["key"], key], axis=3)
+                value = layers.concat([cache["value"], value], axis=2)
+            cache["key"] = key
+            cache["value"] = value
+
+        out = self._attn(query, key, value, mask)
+        out = self._merge_heads(out)
+        out = self.linear_out(out)
+        return out
+
+
+def main():
+    import numpy as np
+
+    place = fluid.CPUPlace()
+    with fluid.dygraph.guard(place):
+        model = MultiheadAttention("MultiheadAttention", 10, 2, 0.5)
+        inp = np.random.rand(2, 3, 10).astype("float32")
+        inp = fluid.dygraph.to_variable(inp)
+        out = model(inp, inp, inp)
+        print(out)
+
+
+if __name__ == "__main__":
+    main()

PaddleNLP/Research/Dialogue-PLATO/plato/modules/parallel.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Parallel class.
+"""
+
+from collections import OrderedDict
+import os
+
+import numpy as np
+from paddle.fluid import core
+from paddle.fluid.dygraph import layers
+from paddle.fluid.dygraph import parallel_helper
+import paddle.fluid.framework as framework
+from paddle.fluid.layers import collective
+from paddle.fluid.dygraph.base import to_variable, no_grad
+
+ParallelStrategy = core.ParallelStrategy
+
+
+def prepare_context(strategy=None):
+    """ Copy codes. """
+    if strategy is None:
+        strategy = ParallelStrategy()
+        strategy.nranks = Env().nranks
+        strategy.local_rank = Env().local_rank
+        strategy.trainer_endpoints = Env().trainer_endpoints
+        strategy.current_endpoint = Env().current_endpoint
+    if strategy.nranks < 2:
+        return
+    assert framework.in_dygraph_mode() is True, \
+        "dygraph.parallel.prepare_context should be used with dygrahp mode."
+    place = framework._current_expected_place()
+    assert place is not None, \
+        "dygraph.parallel.prepare_context should be used in fluid.dygraph.guard(place) guard."
+    if isinstance(place, core.CUDAPlace):
+        parallel_helper._set_parallel_ctx(
+            core.NCCLParallelContext(strategy, place))
+    else:
+        # TODO(Yancey1989): add Gloo Parallel Context to support CPU parallel computation
+        assert ("Only support CUDAPlace for now.")
+    parallel_helper._init_parallel_ctx()
+    return strategy
+
+
+class Env(object):
+    """ Copy codes. """
+    def __init__(self):
+        self._nranks = int(os.getenv("PADDLE_TRAINERS_NUM", "1"))
+        self._local_rank = int(os.getenv("PADDLE_TRAINER_ID", "0"))
+        self._dev_id = int(os.getenv("FLAGS_selected_gpus", "0"))
+        self._trainer_endpoints = os.getenv("PADDLE_TRAINER_ENDPOINTS",
+                                            "").split(",")
+        self._current_endpoint = os.getenv("PADDLE_CURRENT_ENDPOINT", "")
+
+    @property
+    def nranks(self):
+        """ Copy codes. """
+        return self._nranks
+
+    @property
+    def local_rank(self):
+        """ Copy codes. """
+        return self._local_rank
+
+    @property
+    def dev_id(self):
+        """ Copy codes. """
+        return self._dev_id
+
+    @property
+    def current_endpoint(self):
+        """ Copy codes. """
+        return self._current_endpoint
+
+    @property
+    def trainer_endpoints(self):
+        """ Copy codes. """
+        return self._trainer_endpoints
+
+
+class DataParallel(layers.Layer):
+    """
+    Runs the module with data parallelism.
+
+    Currently, DataParallel only supports to run the dynamic graph
+    with multi-process. The usage is:
+    `python -m paddle.distributed.launch --gpus 2 dynamic_graph_test.py`.
+    And the content of `dynamic_graph_test.py` is the code of examples.
+
+    Examples:
+        .. code-block:: python
+
+           import numpy as np
+           import paddle.fluid as fluid
+           import paddle.fluid.dygraph as dygraph
+           from paddle.fluid.optimizer import AdamOptimizer
+           from paddle.fluid.dygraph.nn import FC
+           from paddle.fluid.dygraph.base import to_variable
+
+           place = fluid.CUDAPlace(0)
+           with fluid.dygraph.guard(place=place):
+
+               # prepare the data parallel context
+               strategy=dygraph.parallel.prepare_context()
+
+               fc_layer = FC("FC", 10, act="softmax")
+               adam = fluid.optimizer.AdamOptimizer()
+
+               # make the module become the data parallelism module
+               fc_layer = dygraph.parallel.DataParallel(fc_layer, strategy)
+
+               x_data = np.random.random(size=[10, 1]).astype(np.float32)
+               data = to_variable(x_data)
+
+               hidden = fc_layer(data)
+               avg_loss = fluid.layers.mean(hidden)
+
+               # scale the loss according to the number of trainers.
+               avg_loss = fc_layer.scale_loss(avg_loss)
+
+               avg_loss.backward()
+
+               # collect the gradients of trainers.
+               fc_layer.apply_collective_grads()
+
+               adam.minimize(avg_loss)
+               fc_layer.clear_gradients()
+
+    Args:
+        layers(Layer): The module that should be executed by data parallel.
+        strategy(ParallelStrategy): The strategy of data parallelism.
+
+    Returns:
+        Layer: The data paralleled module.
+    """
+
+    def __init__(self, layers, strategy):
+        super(DataParallel,
+              self).__init__(layers.full_name() + "_data_parallel")
+
+        self._layers = layers
+        self._strategy = strategy
+
+    def forward(self, *inputs, **kwargs):
+        return self._layers(*inputs, **kwargs)
+
+    def __call__(self, *args, **kwargs):
+        # Reimplement __call__ function
+        if not self._built:
+            self._built = True
+
+        outputs = self.forward(*args, **kwargs)
+        return outputs
+
+    def scale_loss(self, loss):
+        """
+        Scale the loss. In data parallel mode, the loss should be scale with
+        the number of trainers. If not in data parallel mode, return the loss
+        directly.
+
+        Args:
+            loss(Layer): The loss of the current Model.
+
+        Returns:
+            Layer: the scaled loss.
+        """
+        if not self._is_data_parallel_mode():
+            return loss
+
+        loss_scale = to_variable(
+            np.array([self._strategy.nranks]).astype("float32"))
+        loss_scale.stop_gradient = True
+        loss = loss / loss_scale
+        return loss
+
+    def _coalesce_tensors(self, var_groups):
+        from paddle.fluid.layers import nn
+        coalesced_grads_and_grad_vars = []
+        for group_id, grad_vars in var_groups.items():
+            flattened_vars = []
+            g_var_shapes = []
+            for g_var in grad_vars:
+                g_var_shapes.append(g_var.shape)
+                flattened_vars.append(
+                    nn.reshape(
+                        x=g_var, shape=[np.prod(g_var.shape)], inplace=True))
+            coalesced_grad = nn.concat(flattened_vars)
+            coalesced_grads_and_grad_vars.append(
+                [coalesced_grad, grad_vars, g_var_shapes])
+        return coalesced_grads_and_grad_vars
+
+    def _split_tensors(self, coalesced_grads_and_grad_vars):
+        from paddle.fluid.layers import nn
+        for coalesced_grad, origin_grad_vars, grad_shapes in coalesced_grads_and_grad_vars:
+            grad_var_len = [np.prod(g_shape) for g_shape in grad_shapes]
+            self._helper.main_program.current_block().append_op(
+                type='split',
+                inputs={'X': coalesced_grad},
+                outputs={'Out': origin_grad_vars},
+                attrs={'sections': grad_var_len,
+                       'axis': 0})
+            for g_var, g_shape in zip(origin_grad_vars, grad_shapes):
+                nn.reshape(x=g_var, shape=g_shape, inplace=True)
+
+    @no_grad
+    def apply_collective_grads(self):
+        """
+        AllReduce the Parameters' gradient.
+        """
+        if not self._is_data_parallel_mode():
+            return
+
+        grad_var_set = set()
+        grad_vars = []
+        for param in self._layers.parameters():
+            # NOTE(zcd): The grad_ivar maybe no generated.
+            if param.trainable and param._ivar._grad_ivar():
+                g_var = framework.Variable(
+                    block=self._helper.main_program.current_block(),
+                    name=param._ivar._grad_name(),
+                    stop_gradient=True,
+                    ivar=param._ivar._grad_ivar())
+                grad_vars.append(g_var)
+                assert g_var not in grad_var_set
+                grad_var_set.add(g_var)
+
+        # FIXME(zcd): the type of the var should be LoDTensor, i.e
+        # the gradients should be dense, otherwise, the following
+        # logic should be updated.
+        # 128 MB as a group
+        mega_bytes = 128 * 1024 * 1024
+        group_idx = 0
+        memory_counter = 0
+        grad_var_groups = OrderedDict()
+        dtype = grad_vars[0].dtype
+        for g_var in grad_vars:
+            # Note: the dtype of the same group should be the same.
+            bytes = np.prod(g_var.shape) * core.size_of_dtype(g_var.dtype)
+            if memory_counter < mega_bytes and dtype == g_var.dtype:
+                memory_counter += bytes
+            else:
+                memory_counter = bytes
+                group_idx += 1
+            grad_var_groups.setdefault(group_idx, []).append(g_var)
+
+        coalesced_grads_and_vars = self._coalesce_tensors(grad_var_groups)
+
+        for coalesced_grad, g_vars, g_shapes in coalesced_grads_and_vars:
+            collective._allreduce(
+                coalesced_grad, coalesced_grad, sync_mode=False)
+
+        self._split_tensors(coalesced_grads_and_vars)
+
+    def _is_data_parallel_mode(self):
+        return self._strategy.nranks > 1

PaddleNLP/Research/Dialogue-PLATO/plato/modules/transformer_block.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+TransformerBlock class.
+"""
+
+import paddle.fluid as fluid
+from paddle.fluid.dygraph import FC
+from paddle.fluid.dygraph import Layer
+import paddle.fluid.layers as layers
+
+from plato.modules.feedforward import FeedForward
+from plato.modules.layer_norm import LayerNorm
+from plato.modules.multihead_attention import MultiheadAttention
+import plato.modules.functions as F
+
+
+class TransformerBlock(Layer):
+    """
+    Transformer block module.
+    """
+
+    def __init__(self, name_scope, hidden_dim, num_heads, dropout, attn_dropout, ff_dropout):
+        super().__init__(name_scope)
+
+        self.attn = MultiheadAttention(name_scope=self.full_name(),
+                                       hidden_dim=hidden_dim,
+                                       num_heads=num_heads,
+                                       dropout=attn_dropout)
+        self.attn_norm = LayerNorm(name_scope=self.full_name(),
+                                   begin_norm_axis=2,
+                                   epsilon=1e-12,
+                                   param_attr=fluid.ParamAttr(
+                                       regularizer=fluid.regularizer.L2Decay(0.0)),
+                                   bias_attr=fluid.ParamAttr(
+                                       regularizer=fluid.regularizer.L2Decay(0.0)))
+        self.ff = FeedForward(name_scope=self.full_name(),
+                              hidden_dim=hidden_dim,
+                              inner_dim=4 * hidden_dim,
+                              dropout=ff_dropout)
+        self.ff_norm = LayerNorm(name_scope=self.full_name(),
+                                 begin_norm_axis=2,
+                                 epsilon=1e-12,
+                                 param_attr=fluid.ParamAttr(
+                                     regularizer=fluid.regularizer.L2Decay(0.0)),
+                                 bias_attr=fluid.ParamAttr(
+                                     regularizer=fluid.regularizer.L2Decay(0.0)))
+        self.dropout = dropout
+        return
+
+    def forward(self, inp, mask=None, cache=None):
+        """
+        Forward process on one transformer layer.
+
+        @param : x
+        @type : Variable(shape: [batch_size, seq_len, hidden_size])
+
+        @param : memory
+        @type : Variable(shape: [batch_size, seq_len, hidden_size])
+
+        @param : mask
+
+        @param : cache
+        """
+        attn_out = self.attn(inp, mask, cache)
+        attn_out = F.dropout(attn_out, self.dropout)
+        attn_out = self.attn_norm(attn_out + inp)
+
+        ff_out = self.ff(attn_out)
+        ff_out = F.dropout(ff_out, self.dropout)
+        ff_out = self.ff_norm(ff_out + attn_out)
+
+        return ff_out
+
+
+def main():
+    import numpy as np
+
+    place = fluid.CPUPlace()
+    with fluid.dygraph.guard(place):
+        model = TransformerBlock("TransformerBlock", 10, 2, 0.5, 0.5, 0.5)
+        inp = np.random.rand(2, 3, 10).astype("float32")
+        inp = fluid.dygraph.to_variable(inp)
+        out = model(inp, inp)
+        print(out)
+
+
+if __name__ == "__main__":
+    main()

PaddleNLP/Research/Dialogue-PLATO/plato/trainer.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Trainer class.
+"""
+
+import json
+import logging
+import os
+import sys
+import time
+
+import numpy as np
+import paddle
+import paddle.fluid as fluid
+import paddle.fluid.dygraph as dygraph
+from tqdm import tqdm
+
+from plato.args import str2bool
+from plato.data.data_loader import DataLoader
+from plato.metrics.metrics_tracker import MetricsTracker
+from plato.metrics.metrics import bleu
+from plato.metrics.metrics import distinct
+import plato.modules.parallel as parallel
+
+
+def get_logger(log_path, name="default"):
+    logger = logging.getLogger(name)
+    logger.propagate = False
+    logger.setLevel(logging.DEBUG)
+
+    formatter = logging.Formatter("%(message)s")
+
+    sh = logging.StreamHandler(sys.stdout)
+    sh.setFormatter(formatter)
+    logger.addHandler(sh)
+
+    fh = logging.FileHandler(log_path, mode="w")
+    fh.setFormatter(formatter)
+    logger.addHandler(fh)
+
+    return logger
+
+
+def evaluate_generation_result(results):
+    tgt = [result["tgt"].split(" ") for result in results]
+    pred = [result["preds"][np.argmax(result["scores"])]
+            if isinstance(result["preds"], list)
+            else result["preds"]
+            for result in results]
+    pred = [p.split(" ") for p in pred]
+    metrics = {}
+    metrics_tracker = MetricsTracker()
+
+    bleu1, bleu2 = bleu(pred, tgt)
+    metrics.update({"bleu_1": bleu1, "bleu_2": bleu2})
+
+    intra_dist1, intra_dist2, inter_dist1, inter_dist2 = distinct(pred)
+    metrics.update({"intra_dist_1": intra_dist1,
+                    "intra_dist_2": intra_dist2,
+                    "inter_dist_1": inter_dist1,
+                    "inter_dist_2": inter_dist2})
+
+    avg_len = sum(map(len, pred)) / len(pred)
+    metrics.update({"len": avg_len})
+
+    metrics_tracker.update(metrics, num_samples=1)
+    return metrics_tracker
+
+
+def save(model, model_path):
+    if isinstance(model, parallel.DataParallel):
+        model = model._layers
+    if hasattr(fluid, "save_dygraph"):
+        # >= 1.6.0 compatible
+        fluid.save_dygraph(model.state_dict(), model_path)
+        fluid.save_dygraph(model.optimizer.state_dict(), model_path)
+    else:
+        dygraph.save_persistables(model.state_dict(), model_path, optimizers=model.optimizer)
+    return
+
+
+class Trainer(object):
+
+    @classmethod
+    def add_cmdline_argument(cls, parser):
+        """ Add the cmdline arguments of trainer. """
+        group = parser.add_argument_group("Trainer")
+        group.add_argument("--use_data_distributed", type=str2bool, default=False,
+                           help="Whether to use data distributed for parallel training.")
+        group.add_argument("--valid_metric_name", type=str, default="-loss",
+                           help="The validation metric determining which checkpoint is the best.")
+        group.add_argument("--num_epochs", type=int, default=10,
+                           help="Total number of training epochs to perform.")
+        group.add_argument("--save_dir", type=str, required=True,
+                           help="The output directory where the model will be saved.")
+        group.add_argument("--batch_size", type=int, default=8,
+                           help="Total batch size for training/evaluation/inference.")
+        group.add_argument("--log_steps", type=int, default=100,
+                           help="The number of training steps to output current metrics "
+                           "on past training dataset.")
+        group.add_argument("--valid_steps", type=int, default=2000,
+                           help="The number of training steps to perform a evaluation "
+                           "on validation datasets.")
+        group.add_argument("--save_checkpoint", type=str2bool, default=True,
+                           help="Whether to save one checkpoints for each training epoch.")
+        group.add_argument("--save_summary", type=str2bool, default=False,
+                           help="Whether to save metrics summary for visualDL module.")
+        DataLoader.add_cmdline_argument(group)
+        return group
+
+    def __init__(self, model, to_tensor, hparams, logger=None):
+        # Use data distributed
+        if hparams.use_data_distributed:
+            strategy = parallel.prepare_context()
+            if strategy is not None:
+                parallel_model = parallel.DataParallel(model, strategy)
+                model.before_backward_fn = parallel_model.scale_loss
+                model.after_backward_fn = parallel_model.apply_collective_grads
+                model = parallel_model
+
+        self.model = model
+        self.to_tensor = to_tensor
+
+        self.is_decreased_valid_metric = hparams.valid_metric_name[0] == "-"
+        self.valid_metric_name = hparams.valid_metric_name[1:]
+        self.num_epochs = hparams.num_epochs
+        self.save_dir = hparams.save_dir
+        self.log_steps = hparams.log_steps
+        self.valid_steps = hparams.valid_steps
+        self.save_checkpoint = hparams.save_checkpoint
+        self.save_summary = hparams.save_summary
+
+        if not os.path.exists(self.save_dir):
+            os.makedirs(self.save_dir)
+
+        self.logger = logger or get_logger(os.path.join(self.save_dir, "trainer.log"), "trainer")
+
+        if self.save_summary:
+            from visualdl import LogWriter
+            self.summary_logger = LogWriter(os.path.join(self.save_dir, "summary"), sync_cycle=10000)
+            self.train_summary = {}
+            self.valid_summary = {}
+
+        self.batch_metrics_tracker = MetricsTracker()
+        self.token_metrics_tracker = MetricsTracker()
+
+        self.best_valid_metric = float("inf" if self.is_decreased_valid_metric else "-inf")
+        self.epoch = 0
+        self.batch_num = 0
+
+    def train_epoch(self, train_iter, valid_iter, infer_iter=None, infer_parse_dict=None):
+        """
+        Train an epoch.
+
+        @param train_iter
+        @type : DataLoader
+
+        @param valid_iter
+        @type : DataLoader
+
+        @param infer_iter
+        @type : DataLoader
+
+        @param infer_parse_dict
+        @type : dict of function
+        """
+        self.epoch += 1
+        num_batches = len(train_iter)
+        self.batch_metrics_tracker.clear()
+        self.token_metrics_tracker.clear()
+        times = []
+        for batch_id, (batch, batch_size) in enumerate(train_iter, 1):
+            batch = type(batch)(map(lambda kv: (kv[0], self.to_tensor(kv[1])), batch.items()))
+            batch["epoch"] = self.epoch
+            batch["num_steps"] = self.batch_num
+
+            # Do a training iteration
+            start_time = time.time()
+            metrics = self.model(batch, is_training=True)
+            token_num = metrics.pop("token_num", None)
+            elapsed = time.time() - start_time
+            times.append(elapsed)
+
+            batch_metrics = {k: v for k, v in metrics.items() if "token" not in k}
+            token_metrics = {k: v for k, v in metrics.items() if "token" in k}
+            self.batch_metrics_tracker.update(batch_metrics, batch_size)
+            self.token_metrics_tracker.update(token_metrics, token_num)
+            self.batch_num += 1
+
+            if self.log_steps and batch_id % self.log_steps == 0:
+                batch_metrics_message = self.batch_metrics_tracker.value()
+                token_metrics_message = self.token_metrics_tracker.value()
+                message_prefix = f"[Train][{self.epoch}][{batch_id}/{num_batches}]"
+                avg_time = f"AVG_Time-{sum(times[-self.log_steps:]) / self.log_steps:.3f}"
+                message = "   ".join([message_prefix, batch_metrics_message, token_metrics_message,
+                                      avg_time])
+                self.logger.info(message)
+
+            if self.save_summary:
+                with self.summary_logger.mode("train"):
+                    for k, v in self.batch_metrics_tracker.items():
+                        if k not in self.train_summary:
+                            self.train_summary[k] = self.summary_logger.scalar(k)
+                        scalar = self.train_summary[k]
+                        scalar.add_record(self.batch_num, v)
+                    for k, v in self.token_metrics_tracker.items():
+                        if k not in self.train_summary:
+                            self.train_summary[k] = self.summary_logger.scalar(k)
+                        scalar = self.train_summary[k]
+                        scalar.add_record(self.batch_num, v)
+
+            if self.valid_steps and valid_iter is not None and \
+                    batch_id % self.valid_steps == 0:
+                self.evaluate(valid_iter)
+
+        if valid_iter is not None:
+            self.evaluate(valid_iter)
+
+        if infer_iter is not None and infer_parse_dict is not None:
+            self.infer(infer_iter, infer_parse_dict)
+
+        return
+
+    def infer(self, data_iter, parse_dict, num_batches=None):
+        """
+        Inference interface.
+
+        @param : data_iter
+        @type : DataLoader
+
+        @param : parse_dict
+        @type : dict of function
+
+        @param : num_batches : the number of batch to infer
+        @type : int/None
+        """
+        self.logger.info("Generation starts ...")
+        infer_save_file = os.path.join(self.save_dir, f"infer_{self.epoch}.result.json")
+
+        # Inference
+        infer_results = []
+        batch_cnt = 0
+        begin_time = time.time()
+        for batch, batch_size in tqdm(data_iter, total=num_batches):
+            batch = type(batch)(map(lambda kv: (kv[0], self.to_tensor(kv[1])), batch.items()))
+
+            result = self.model.infer(inputs=batch)
+            batch_result = {}
+
+            def to_list(batch):
+                """ Parse list. """
+                return batch.tolist()
+
+            # parse
+            for k in result:
+                if k in parse_dict:
+                    parse_fn = parse_dict[k]
+                else:
+                    parse_fn = to_list
+                if result[k] is not None:
+                    batch_result[k] = parse_fn(result[k])
+
+            for vs in zip(*batch_result.values()):
+                infer_result = {}
+                for k, v in zip(batch_result.keys(), vs):
+                    infer_result[k] = v
+                infer_results.append(infer_result)
+
+            batch_cnt += 1
+            if batch_cnt == num_batches:
+                break
+
+        self.logger.info(f"Saved inference results to {infer_save_file}")
+        with open(infer_save_file, "w") as fp:
+            json.dump(infer_results, fp, indent=2)
+        infer_metrics_tracker = evaluate_generation_result(infer_results)
+        metrics_message = infer_metrics_tracker.summary()
+        message_prefix = f"[Infer][{self.epoch}]"
+        time_cost = f"TIME-{time.time() - begin_time:.3f}"
+        message = "   ".join([message_prefix, metrics_message, time_cost])
+        self.logger.info(message)
+        return
+
+    def evaluate(self, data_iter, need_save=True):
+        """
+        Evaluation interface
+
+        @param : data_iter
+        @type : DataLoader
+
+        @param : need_save
+        @type : bool
+        """
+        if isinstance(self.model, parallel.DataParallel):
+            need_save = need_save and parallel.Env().local_rank == 0
+
+        # Evaluation
+        begin_time = time.time()
+        batch_metrics_tracker = MetricsTracker()
+        token_metrics_tracker = MetricsTracker()
+        for batch, batch_size in data_iter:
+            batch = type(batch)(map(lambda kv: (kv[0], self.to_tensor(kv[1])), batch.items()))
+            metrics = self.model(batch, is_training=False)
+            token_num = int(metrics.pop("token_num"))
+            batch_metrics = {k: v for k, v in metrics.items() if "token" not in k}
+            token_metrics = {k: v for k, v in metrics.items() if "token" in k}
+            batch_metrics_tracker.update(batch_metrics, batch_size)
+            token_metrics_tracker.update(token_metrics, token_num)
+        batch_metrics_message = batch_metrics_tracker.summary()
+        token_metrics_message = token_metrics_tracker.summary()
+        message_prefix = f"[Valid][{self.epoch}]"
+        time_cost = f"TIME-{time.time() - begin_time:.3f}"
+        message = "   ".join([message_prefix, batch_metrics_message, token_metrics_message, time_cost])
+        self.logger.info(message)
+
+        if need_save:
+            # Check valid metric
+            cur_valid_metric = batch_metrics_tracker.get(self.valid_metric_name)
+            if self.is_decreased_valid_metric:
+                is_best = cur_valid_metric < self.best_valid_metric
+            else:
+                is_best = cur_valid_metric > self.best_valid_metric
+            if is_best:
+                # Save current best model
+                self.best_valid_metric = cur_valid_metric
+                best_model_path = os.path.join(self.save_dir, "best.model")
+                save(self.model, best_model_path)
+                self.logger.info(
+                    f"Saved best model to '{best_model_path}' with new best valid metric "
+                    f"{self.valid_metric_name.upper()}-{self.best_valid_metric:.3f}")
+
+            # Save checkpoint
+            if self.save_checkpoint:
+                model_file = os.path.join(self.save_dir, f"epoch_{self.epoch}.model")
+                save(self.model, model_file)
+
+            if self.save_summary:
+                with self.summary_logger.mode("valid"):
+                    for k, v in self.batch_metrics_tracker.items():
+                        if k not in self.valid_summary:
+                            self.valid_summary[k] = self.summary_logger.scalar(k)
+                        scalar = self.valid_summary[k]
+                        scalar.add_record(self.batch_num, v)
+                    for k, v in self.token_metrics_tracker.items():
+                        if k not in self.valid_summary:
+                            self.valid_summary[k] = self.summary_logger.scalar(k)
+                        scalar = self.valid_summary[k]
+                        scalar.add_record(self.batch_num, v)
+
+        return

PaddleNLP/Research/Dialogue-PLATO/preprocess.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Preprocess script.
+"""
+
+import os
+import argparse
+
+from plato.args import str2bool
+from plato.args import parse_args
+from plato.data.dataset import Dataset
+from plato.data.field import BPETextField
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    
+    BPETextField.add_cmdline_argument(parser)
+    Dataset.add_cmdline_argument(parser)
+    
+    args = parse_args(parser)
+    
+    raw_train_file = os.path.join(args.data_dir, "dial.train")
+    raw_valid_file = os.path.join(args.data_dir, "dial.valid")
+    raw_test_file = os.path.join(args.data_dir, "dial.test")
+    train_file = raw_train_file + f".{args.tokenizer_type}.jsonl"
+    valid_file = raw_valid_file + f".{args.tokenizer_type}.jsonl"
+    test_file = raw_test_file + f".{args.tokenizer_type}.jsonl"
+    
+    bpe = BPETextField(args.BPETextField)
+    
+    BUILD_EXAMPLES_FN = {
+        "multi": bpe.build_examples_multi_turn,
+        "multi_knowledge": bpe.build_examples_multi_turn_with_knowledge
+    }
+    build_examples_fn = BUILD_EXAMPLES_FN[args.data_type]
+    
+    if os.path.exists(raw_valid_file) and not os.path.exists(valid_file):
+        valid_examples = build_examples_fn(raw_valid_file, data_type="valid")
+        bpe.save_examples(valid_examples, valid_file)
+    
+    if os.path.exists(raw_test_file) and not os.path.exists(test_file):
+        test_examples = build_examples_fn(raw_test_file, data_type="test")
+        bpe.save_examples(test_examples, test_file)
+    
+    if os.path.exists(raw_train_file) and not os.path.exists(train_file):
+        train_examples = build_examples_fn(raw_train_file, data_type="train")
+        bpe.save_examples(train_examples, train_file)
+
+    return
+
+
+if __name__ == "__main__":
+    main()

PaddleNLP/Research/Dialogue-PLATO/run.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Running scripts.
+"""
+
+import argparse
+import json
+import os
+
+import numpy as np
+import paddle.fluid as fluid
+
+from plato.args import parse_args
+from plato.args import str2bool
+from plato.data.data_loader import DataLoader
+from plato.data.dataset import Dataset
+from plato.data.dataset import LazyDataset
+from plato.data.field import BPETextField
+from plato.trainer import Trainer
+from plato.models.model_base import ModelBase
+from plato.models.generator import Generator
+import plato.modules.parallel as parallel
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--do_train", type=str2bool, default=False,
+                        help="Whether to run trainning.")
+    parser.add_argument("--do_test", type=str2bool, default=False,
+                        help="Whether to run evaluation on the test dataset.")
+    parser.add_argument("--do_infer", type=str2bool, default=False,
+                        help="Whether to run inference on the test dataset.")
+    parser.add_argument("--num_infer_batches", type=int, default=None,
+                        help="The number of batches need to infer.\n"
+                        "Stay 'None': infer on entrie test dataset.")
+    parser.add_argument("--hparams_file", type=str, default=None,
+                        help="Loading hparams setting from file(.json format).")
+    BPETextField.add_cmdline_argument(parser)
+    Dataset.add_cmdline_argument(parser)
+    Trainer.add_cmdline_argument(parser)
+    ModelBase.add_cmdline_argument(parser)
+    Generator.add_cmdline_argument(parser)
+
+    hparams = parse_args(parser)
+
+    if hparams.hparams_file and os.path.exists(hparams.hparams_file):
+        print(f"Loading hparams from {hparams.hparams_file} ...")
+        hparams.load(hparams.hparams_file)
+        print(f"Loaded hparams from {hparams.hparams_file}")
+
+    print(json.dumps(hparams, indent=2))
+
+    if not os.path.exists(hparams.save_dir):
+        os.makedirs(hparams.save_dir)
+    hparams.save(os.path.join(hparams.save_dir, "hparams.json"))
+    
+    bpe = BPETextField(hparams.BPETextField)
+    hparams.Model.num_token_embeddings = bpe.vocab_size
+
+    generator = Generator.create(hparams.Generator, bpe=bpe)
+
+    COLLATE_FN = {
+        "multi": bpe.collate_fn_multi_turn,
+        "multi_knowledge": bpe.collate_fn_multi_turn_with_knowledge
+    }
+    collate_fn = COLLATE_FN[hparams.data_type]
+
+    # Loading datasets
+    if hparams.do_train:
+        raw_train_file = os.path.join(hparams.data_dir, "dial.train")
+        train_file = raw_train_file + f".{hparams.tokenizer_type}.jsonl"
+        assert os.path.exists(train_file), f"{train_file} isn't exist"
+        train_dataset = LazyDataset(train_file)
+        train_loader = DataLoader(train_dataset, hparams.Trainer, collate_fn=collate_fn, is_train=True)
+        raw_valid_file = os.path.join(hparams.data_dir, "dial.valid")
+        valid_file = raw_valid_file + f".{hparams.tokenizer_type}.jsonl"
+        assert os.path.exists(valid_file), f"{valid_file} isn't exist"
+        valid_dataset = LazyDataset(valid_file)
+        valid_loader = DataLoader(valid_dataset, hparams.Trainer, collate_fn=collate_fn)
+
+    if hparams.do_infer or hparams.do_test:
+        raw_test_file = os.path.join(hparams.data_dir, "dial.test")
+        test_file = raw_test_file + f".{hparams.tokenizer_type}.jsonl"
+        assert os.path.exists(test_file), f"{test_file} isn't exist"
+        test_dataset = LazyDataset(test_file)
+        test_loader = DataLoader(test_dataset, hparams.Trainer, collate_fn=collate_fn, is_test=hparams.do_infer)
+
+    def to_tensor(array):
+        array = np.expand_dims(array, -1)
+        return fluid.dygraph.to_variable(array)
+
+    if hparams.use_data_distributed:
+        place = fluid.CUDAPlace(parallel.Env().dev_id)
+    else:
+        place = fluid.CUDAPlace(0)
+
+    with fluid.dygraph.guard(place):
+        # Construct Model
+        model = ModelBase.create("Model", hparams, generator=generator)
+
+        # Construct Trainer
+        trainer = Trainer(model, to_tensor, hparams.Trainer)
+
+        if hparams.do_train:
+            # Training process
+            for epoch in range(hparams.num_epochs):
+                trainer.train_epoch(train_loader, valid_loader)
+
+        if hparams.do_test:
+            # Validation process
+            trainer.evaluate(test_loader, need_save=False)
+
+        if hparams.do_infer:
+            # Inference process
+            def split(xs, sep, pad):
+                """ Split id list by separator. """
+                out, o = [], []
+                for x in xs:
+                    if x == pad:
+                        continue
+                    if x != sep:
+                        o.append(x)
+                    else:
+                        if len(o) > 0:
+                            out.append(list(o))
+                            o = []
+                if len(o) > 0:
+                    out.append(list(o))
+                assert(all(len(o) > 0 for o in out))
+                return out
+
+            def parse_context(batch):
+                """ Parse context. """
+                return bpe.denumericalize([split(xs, bpe.eos_id, bpe.pad_id)
+                                           for xs in batch.tolist()])
+
+            def parse_text(batch):
+                """ Parse text. """
+                return bpe.denumericalize(batch.tolist())
+
+            infer_parse_dict = {
+                "src": parse_context,
+                "tgt": parse_text,
+                "preds": parse_text
+            }
+            trainer.infer(test_loader, infer_parse_dict, num_batches=hparams.num_infer_batches)
+
+
+if __name__ == "__main__":
+    main()

PaddleNLP/Research/Dialogue-PLATO/scripts/DSTC7_AVSD/infer.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DSTC7_AVSD.infer
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DSTC7_AVSD
+INIT_CHECKPOINT=outputs/DSTC7_AVSD/best.model
+DATA_TYPE=multi_knowledge
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+python -u \
+    ./run.py \
+    --do_infer true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 4 \
+    --num_type_embeddings 3 \
+    --use_discriminator true \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR

PaddleNLP/Research/Dialogue-PLATO/scripts/DSTC7_AVSD/train.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DSTC7_AVSD
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DSTC7_AVSD
+INIT_CHECKPOINT=model/PLATO
+DATA_TYPE=multi_knowledge
+USE_VISUALDL=false
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+if [[ "$USE_VISUALDL" = true ]]; then
+    visualdl --logdir=$SAVE_DIR/summary --port=8083 --host=`hostname` &
+    VISUALDL_PID=$!
+fi
+
+python -u \
+    ./run.py \
+    --do_train true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 4 \
+    --valid_steps 2000 \
+    --num_type_embeddings 3 \
+    --use_discriminator true \
+    --num_epoch 20 \
+    --lr 1e-5 \
+    --save_checkpoint false \
+    --save_summary $USE_VISUALDL \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR
+
+if [[ $USE_VISUALDL = true ]]; then
+    kill $VISUALDL_PID
+fi

PaddleNLP/Research/Dialogue-PLATO/scripts/DailyDialog/baseline_infer.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DailyDialog.baseline.infer
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DailyDialog
+INIT_CHECKPOINT=outputs/DailyDialog.baseline/best.model
+DATA_TYPE=multi
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+python -u \
+    ./run.py \
+    --do_infer true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 48 \
+    --num_latent 0 \
+    --num_type_embeddings 2 \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --length_average true \
+    --save_dir $SAVE_DIR

PaddleNLP/Research/Dialogue-PLATO/scripts/DailyDialog/baseline_train.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DailyDialog.baseline
+VOCAB_PATH=model-baseline/Bert/vocab.txt
+DATA_DIR=data/DailyDialog
+INIT_CHECKPOINT=model-baseline/PLATO.baseline
+DATA_TYPE=multi
+USE_VISUALDL=false
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=2
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+if [[ "$USE_VISUALDL" = true ]]; then
+    visualdl --logdir=$SAVE_DIR/summary --port=8083 --host=`hostname` &
+    VISUALDL_PID=$!
+fi
+
+python -u \
+    ./run.py \
+    --do_train true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 2 \
+    --valid_steps 2000 \
+    --num_type_embeddings 2 \
+    --num_latent 0 \
+    --num_epoch 20 \
+    --lr 1e-5 \
+    --save_checkpoint false \
+    --save_summary $USE_VISUALDL \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR
+
+if [[ $USE_VISUALDL = true ]]; then
+    kill $VISUALDL_PID
+fi

PaddleNLP/Research/Dialogue-PLATO/scripts/DailyDialog/infer.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DailyDialog.infer
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DailyDialog
+INIT_CHECKPOINT=outputs/DailyDialog/best.model
+DATA_TYPE=multi
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+python -u \
+    ./run.py \
+    --do_infer true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 4 \
+    --num_type_embeddings 2 \
+    --num_latent 20 \
+    --use_discriminator true \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR

PaddleNLP/Research/Dialogue-PLATO/scripts/DailyDialog/multi_gpu_train.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DailyDialog
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DailyDialog
+INIT_CHECKPOINT=model/PLATO
+DATA_TYPE=multi
+USE_VISUALDL=false
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0,1
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+if [[ ! -e $DATA_DIR/dial.train.jsonl ]]; then
+    python -u \
+        ./preprocess.py \
+        --vocab_path $VOCAB_PATH \
+        --data_dir $DATA_DIR \
+        --data_type $DATA_TYPE
+fi
+
+if [[ "$USE_VISUALDL" = true ]]; then
+    visualdl --logdir=$SAVE_DIR/summary --port=8083 --host=`hostname` &
+    VISUALDL_PID=$!
+fi
+
+python -m \
+    paddle.distributed.launch \
+    --log_dir $SAVE_DIR \
+    --started_port 8888 \
+    ./run.py \
+    --use_data_distributed true \
+    --do_train true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 6 \
+    --valid_steps 2000 \
+    --num_type_embeddings 2 \
+    --use_discriminator true \
+    --num_epoch 20 \
+    --lr 1e-5 \
+    --save_checkpoint false \
+    --save_summary $USE_VISUALDL \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR
+
+if [[ $USE_VISUALDL = true ]]; then
+    kill $VISUALDL_PID
+fi

PaddleNLP/Research/Dialogue-PLATO/scripts/DailyDialog/topk_infer.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DailyDialog.infer
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DailyDialog
+INIT_CHECKPOINT=outputs/DailyDialog/best.model
+DATA_TYPE=multi
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+if [[ ! -e $DATA_DIR/dial.test.jsonl ]]; then
+    python -u \
+        ./preprocess.py \
+        --vocab_path $VOCAB_PATH \
+        --data_dir $DATA_DIR \
+        --data_type $DATA_TYPE
+fi
+
+python -u \
+    ./run.py \
+    --do_infer true \
+    --generator TopKSampling \
+    --top_k_num 10 \
+    --sampling_temperate 0.8 \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 16 \
+    --num_type_embeddings 2 \
+    --use_discriminator true \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR

PaddleNLP/Research/Dialogue-PLATO/scripts/DailyDialog/train.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/DailyDialog
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/DailyDialog
+INIT_CHECKPOINT=model/PLATO
+DATA_TYPE=multi
+USE_VISUALDL=false
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+if [[ "$USE_VISUALDL" = true ]]; then
+    visualdl --logdir=$SAVE_DIR/summary --port=8083 --host=`hostname` &
+    VISUALDL_PID=$!
+fi
+
+python -u \
+    ./run.py \
+    --do_train true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 6 \
+    --valid_steps 2000 \
+    --num_type_embeddings 2 \
+    --use_discriminator true \
+    --num_epoch 20 \
+    --lr 1e-5 \
+    --save_checkpoint false \
+    --save_summary $USE_VISUALDL \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR
+
+if [[ $USE_VISUALDL = true ]]; then
+    kill $VISUALDL_PID
+fi

PaddleNLP/Research/Dialogue-PLATO/scripts/PersonaChat/infer.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/PersonaChat.infer
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/PersonaChat
+INIT_CHECKPOINT=outputs/PersonaChat/best.model
+DATA_TYPE=multi_knowledge
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+python -u \
+    ./run.py \
+    --do_infer true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 2 \
+    --num_type_embeddings 3 \
+    --use_discriminator true \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR
+
+python -u ./tools/knowledge_f1.py $SAVE_DIR/infer_0.result.json $DATA_DIR/dial.test

PaddleNLP/Research/Dialogue-PLATO/scripts/PersonaChat/train.sh

+#!/bin/bash
+set -ux
+
+SAVE_DIR=outputs/PersonaChat
+VOCAB_PATH=model/Bert/vocab.txt
+DATA_DIR=data/PersonaChat
+INIT_CHECKPOINT=model/PLATO
+DATA_TYPE=multi_knowledge
+USE_VISUALDL=false
+
+# CUDA environment settings.
+export CUDA_VISIBLE_DEVICES=0
+
+# Paddle environment settings.
+export FLAGS_fraction_of_gpu_memory_to_use=0.1
+export FLAGS_eager_delete_scope=True
+export FLAGS_eager_delete_tensor_gb=0.0
+
+python -u \
+    ./preprocess.py \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE
+
+if [[ "$USE_VISUALDL" = true ]]; then
+    visualdl --logdir=$SAVE_DIR/summary --port=8083 --host=`hostname` &
+    VISUALDL_PID=$!
+fi
+
+python -u \
+    ./run.py \
+    --do_train true \
+    --vocab_path $VOCAB_PATH \
+    --data_dir $DATA_DIR \
+    --data_type $DATA_TYPE \
+    --batch_size 4 \
+    --valid_steps 2000 \
+    --num_type_embeddings 3 \
+    --use_discriminator true \
+    --num_epoch 20 \
+    --lr 1e-5 \
+    --save_checkpoint false \
+    --save_summary $USE_VISUALDL \
+    --init_checkpoint $INIT_CHECKPOINT \
+    --save_dir $SAVE_DIR
+
+if [[ $USE_VISUALDL = true ]]; then
+    kill $VISUALDL_PID
+fi

PaddleNLP/Research/Dialogue-PLATO/tools/knowledge_f1.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Calculate Knowledge f1.
+"""
+
+import sys
+import json
+
+import numpy as np
+
+eval_file = sys.argv[1]
+test_file = sys.argv[2]
+
+cnt = 0
+res = 0.0
+r = 0.0
+p = 0.0
+stopwords = set()
+with open("./tools/stopwords.txt") as f:
+    for line in f:
+        word = line.strip()
+        stopwords.add(word)
+
+with open(eval_file) as f:
+    for result, line in zip(json.load(f), open(test_file)):
+        cnt += 1
+        if "scores" in result:
+            pred = result["preds"][np.argmax(result["scores"])]
+        else:
+            pred = result["preds"][0]
+        knowledges, _, reply = line.strip().split('\t')
+
+        words = set()
+        for sent in knowledges.split(" __eou__ "):
+            for word in sent.split():
+                words.add(word)
+        words = words - stopwords
+        k_len = len(words)
+
+        pred1 = set(pred.split())
+        pred1 = pred1 - stopwords
+        pred_len = len(pred1)
+        overlap = len(words & pred1)
+
+        if overlap == 0:
+            continue
+
+        recall = float(overlap) / k_len
+        r += recall
+        precison = float(overlap) / pred_len
+        p += precison
+        res += 2*recall*precison/(recall+precison)
+print(f"Recall:{r/cnt}")
+print(f"Precison:{p/cnt}")
+print(f"F1:{res/cnt}")
+print("Recall/Precision/F1:{:0,.4f}/{:0,.4f}/{:0,.4f}".format(r/cnt, p/cnt, res/cnt))
+