MRQA2019-D-NET (#3413)

* MRQA2019-D-NET

* delete_data_file

PaddleNLP/Research/MRQA2019-D-NET/README.md

+# D-NET
+
+## Introduction
+D-NET is the system Baidu submitted for MRQA (Machine Reading for Question Answering) 2019 Shared Task that focused on generalization of machine reading comprehension (MRC) models. Our system is built on a framework of pre-training and fine-tuning. The techniques of pre-trained language models, multi-task learning and knowledge distillation are employed to improve the generalization of MRC models and the experimental results show the effectiveness of these strategies. Our system is ranked at top 1 of all the participants in terms of averaged F1 score. Additionally, we won the first place for 10 of the 12 test sets and the second place for the other two in terms of F1 scores.
+
+## Framework
+
+<p align="center">
+<img src="./images/D-NET_framework.png" width="500">
+</p>
+
+### D-NET includes 3 parts: 
+#### multi_task_learning
+We use PaddlePaddle PALM multi-task learning library [Link](https://github.com/PaddlePaddle/PALM) to train single model for  MRQA 2019 Shared Task.
+
+#### knowledge_distillation 
+Model ensemble can improve the generalization of MRC models, we leverage the technique of distillation to ensemble multiple models into a single  model, and no loss of accuracy, distillation solves the problem of slow inference process and reduce the use of a huge amount of resource.
+
+#### server
+MRQA2019 submission environment with baidu bert inference model and xlnet inference model.
+
+
+## Copyright and License
+Copyright 2019 Baidu.com, Inc. All Rights Reserved Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/README.md

+# knowledge_distillation
+
+## 1、Introduction
+Model ensemble can improve the generalization of MRC models. However, such approach is not efficient. Because the inference of an ensemble model is slow and a huge amount of resources are required. We leverage the technique of distillation to ensemble multiple models into a single model solves the problem of slow inference process.
+
+## 2、Quick Start
+
+### Environment
+- Python >= 2.7
+- cuda >= 9.0
+- cudnn >= 7.0
+- PaddlePaddle >= 1.5.0 Please refer to Installation Guide [Installation Guide](http://www.paddlepaddle.org/#quick-start)
+
+### Data and Models Preparation
+User can get the data and trained knowledge_distillation models directly we provided: 
+```
+bash wget_models_and_data.sh
+```
+user can get data and models directorys: 
+
+data: 
+
+./data/input/mlm_data: mask language model dataset.
+
+./data/input/mrqa_distill_data: mrqa dataset, it includes two parts: mrqa_distill.json(json data we calculate from teacher models), mrqa-combined.all_dev.raw.json(merge all mrqa dev dataset). 
+
+./data/input/mrqa_evaluation_dataset: mrqa evaluation data(in_domain data and out_of_domain json data).
+
+models: 
+
+./data/pretrain_model/squad2_model: pretrain model(google squad2.0 model as pretrain model [Model Link](https://worksheets.codalab.org/worksheets/0x3852e60a51d2444680606556d404c657)).
+
+./saved_models/knowledge_distillation_model: baidu trained knowledge distillation model.
+
+## 3、Train and Predict
+Train and predict  knowledge distillation model
+```
+bash run_distill.sh
+```
+
+## 4、Evaluation
+To evaluate the result, run
+```
+sh run_evaluation.sh
+```
+Note that we use the evaluation script for SQuAD 1.1 here, which is equivalent to the official one.
+
+## 5、Performance
+
+|  | dev in_domain(Macro-F1)| dev out_of_domain(Macro-F1) |
+| ------------- | ------------ | ------------ |
+| Official baseline | 77.87 | 58.67 |
+| KD(4 teacher model-> student)| 83.67 | 67.34 |
+
+KD: knowledge distillation model(ensemble 4 teacher models to student model)
+
+## Copyright and License
+Copyright 2019 Baidu.com, Inc. All Rights Reserved Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and
+limitations under the License.
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/data/input/input.md

+input data dir: mrqa distillation dataset and mask language model dataset

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/data/output/output.md

+save checkpoints dir

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/data/pretrain_model/pretrain_model.md

+pretrain model dir

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/data/saved_models/saved_models.md

+MRQA2019 baidu trained knowledge distillation model

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/model/bert.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT model."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import six
+import json
+import numpy as np
+import paddle.fluid as fluid
+
+from model.transformer_encoder import encoder as encoder
+from model.transformer_encoder import pre_process_layer as pre_process_layer
+
+
+class BertModel(object):
+    
+    def __init__(self,
+                 src_ids,
+                 position_ids,
+                 sentence_ids,
+                 input_mask,
+                 config,
+                 weight_sharing=True,
+                 use_fp16=False,
+                 model_name = ''):
+
+        self._emb_size = config["hidden_size"]
+        self._n_layer = config["num_hidden_layers"]
+        self._n_head = config["num_attention_heads"]
+        self._voc_size = config["vocab_size"]
+        self._max_position_seq_len = config["max_position_embeddings"]
+        self._sent_types = config["type_vocab_size"]
+        self._hidden_act = config["hidden_act"]
+        self._prepostprocess_dropout = config["hidden_dropout_prob"]
+        self._attention_dropout = config["attention_probs_dropout_prob"]
+        self._weight_sharing = weight_sharing
+
+        self.model_name = model_name
+
+        self._word_emb_name = self.model_name + "word_embedding"
+        self._pos_emb_name = self.model_name + "pos_embedding"
+        self._sent_emb_name = self.model_name + "sent_embedding"
+        self._dtype = "float16" if use_fp16 else "float32"
+
+        # Initialize all weigths by truncated normal initializer, and all biases 
+        # will be initialized by constant zero by default.
+        self._param_initializer = fluid.initializer.TruncatedNormal(
+            scale=config["initializer_range"])
+
+        self._build_model(src_ids, position_ids, sentence_ids, input_mask, config)
+
+    def _build_model(self, src_ids, position_ids, sentence_ids, input_mask, config):
+        # padding id in vocabulary must be set to 0
+        emb_out = fluid.layers.embedding(
+            input=src_ids,
+            size=[self._voc_size, self._emb_size],
+            dtype=self._dtype,
+            param_attr=fluid.ParamAttr(
+                name=self._word_emb_name, initializer=self._param_initializer),
+            is_sparse=False)
+        
+        self.emb_out =emb_out
+        
+        position_emb_out = fluid.layers.embedding(
+            input=position_ids,
+            size=[self._max_position_seq_len, self._emb_size],
+            dtype=self._dtype,
+            param_attr=fluid.ParamAttr(
+                name=self._pos_emb_name, initializer=self._param_initializer))
+    
+        self.position_emb_out = position_emb_out
+
+        sent_emb_out = fluid.layers.embedding(
+            sentence_ids,
+            size=[self._sent_types, self._emb_size],
+            dtype=self._dtype,
+            param_attr=fluid.ParamAttr(
+                name=self._sent_emb_name, initializer=self._param_initializer))
+
+        self.sent_emb_out = sent_emb_out
+
+        emb_out = emb_out + position_emb_out
+        emb_out = emb_out + sent_emb_out
+
+        emb_out = pre_process_layer(
+            emb_out, 'nd', self._prepostprocess_dropout, name='pre_encoder')
+
+        if self._dtype == "float16":
+            input_mask = fluid.layers.cast(x=input_mask, dtype=self._dtype)
+
+        self_attn_mask = fluid.layers.matmul(
+            x = input_mask, y = input_mask, transpose_y = True)
+
+        self_attn_mask = fluid.layers.scale(
+            x = self_attn_mask, scale = 10000.0, bias = -1.0, bias_after_scale = False)
+        
+        n_head_self_attn_mask = fluid.layers.stack(
+            x=[self_attn_mask] * self._n_head, axis=1)
+        
+        n_head_self_attn_mask.stop_gradient = True
+
+        self._enc_out = encoder(
+            enc_input = emb_out,
+            attn_bias = n_head_self_attn_mask,
+            n_layer = self._n_layer,
+            n_head = self._n_head,
+            d_key = self._emb_size // self._n_head,
+            d_value = self._emb_size // self._n_head,
+            d_model = self._emb_size,
+            d_inner_hid = self._emb_size * 4,
+            prepostprocess_dropout = self._prepostprocess_dropout,
+            attention_dropout = self._attention_dropout,
+            relu_dropout = 0,
+            hidden_act = self._hidden_act,
+            preprocess_cmd = "",
+            postprocess_cmd = "dan",
+            param_initializer = self._param_initializer,
+            name = self.model_name + 'encoder')
+
+    def get_sequence_output(self):
+        return self._enc_out
+
+    def get_pooled_output(self):
+        """Get the first feature of each sequence for classification"""
+
+        next_sent_feat = fluid.layers.slice(
+            input = self._enc_out, axes = [1], starts = [0], ends = [1])
+        next_sent_feat = fluid.layers.fc(
+            input = next_sent_feat,
+            size = self._emb_size,
+            act = "tanh",
+            param_attr = fluid.ParamAttr(
+                name = self.model_name + "pooled_fc.w_0", 
+                initializer = self._param_initializer),
+            bias_attr = "pooled_fc.b_0")
+        return next_sent_feat
+
+    def get_pretraining_output(self, mask_label, mask_pos, labels):
+        """Get the loss & accuracy for pretraining"""
+
+        mask_pos = fluid.layers.cast(x=mask_pos, dtype='int32')
+
+        # extract the first token feature in each sentence
+        next_sent_feat = self.get_pooled_output()
+        reshaped_emb_out = fluid.layers.reshape(
+            x=self._enc_out, shape = [-1, self._emb_size])
+        # extract masked tokens' feature
+        mask_feat = fluid.layers.gather(input = reshaped_emb_out, index = mask_pos)
+
+        # transform: fc
+        mask_trans_feat = fluid.layers.fc(
+            input = mask_feat,
+            size = self._emb_size,
+            act = self._hidden_act,
+            param_attr = fluid.ParamAttr(
+                name = self.model_name + 'mask_lm_trans_fc.w_0',
+                initializer = self._param_initializer),
+            bias_attr = fluid.ParamAttr(name = self.model_name + 'mask_lm_trans_fc.b_0'))
+        # transform: layer norm 
+        mask_trans_feat = pre_process_layer(
+            mask_trans_feat, 'n', name = self.model_name + 'mask_lm_trans')
+
+        mask_lm_out_bias_attr = fluid.ParamAttr(
+            name = self.model_name + "mask_lm_out_fc.b_0",
+            initializer = fluid.initializer.Constant(value = 0.0))
+        if self._weight_sharing:
+            fc_out = fluid.layers.matmul(
+                x = mask_trans_feat,
+                y = fluid.default_main_program().global_block().var(
+                    self._word_emb_name),
+                transpose_y = True)
+            fc_out += fluid.layers.create_parameter(
+                shape = [self._voc_size],
+                dtype = self._dtype,
+                attr = mask_lm_out_bias_attr,
+                is_bias = True)
+
+        else:
+            fc_out = fluid.layers.fc(input = mask_trans_feat,
+                                     size = self._voc_size,
+                                     param_attr = fluid.ParamAttr(
+                                         name = self.model_name + "mask_lm_out_fc.w_0",
+                                         initializer = self._param_initializer),
+                                     bias_attr = mask_lm_out_bias_attr)
+
+        mask_lm_loss = fluid.layers.softmax_with_cross_entropy(
+            logits = fc_out, label = mask_label)
+        mean_mask_lm_loss = fluid.layers.mean(mask_lm_loss)
+
+        next_sent_fc_out = fluid.layers.fc(
+            input = next_sent_feat,
+            size = 2,
+            param_attr = fluid.ParamAttr(
+                name = self.model_name + "next_sent_fc.w_0", 
+                initializer = self._param_initializer),
+            bias_attr = self.model_name + "next_sent_fc.b_0")
+
+        next_sent_loss, next_sent_softmax = fluid.layers.softmax_with_cross_entropy(
+            logits = next_sent_fc_out, label = labels, return_softmax = True)
+
+        next_sent_acc = fluid.layers.accuracy(
+            input = next_sent_softmax, label = labels)
+
+        mean_next_sent_loss = fluid.layers.mean(next_sent_loss)
+
+        loss = mean_next_sent_loss + mean_mask_lm_loss
+        return next_sent_acc, mean_mask_lm_loss, loss
+
+
+if __name__ == "__main__":
+    print("hello wolrd!")
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/model/bert_model.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+import argparse
+import collections
+import numpy as np
+import multiprocessing
+from copy import deepcopy as copy
+
+import paddle
+import paddle.fluid as fluid
+
+from model.bert import BertModel
+from utils.configure import JsonConfig
+
+
+class ModelBERT(object):
+
+    def __init__(
+        self, 
+        conf, 
+        name = "", 
+        is_training = False,
+        base_model = None):
+
+        # the name of this task
+        # name is used for identifying parameters
+        self.name = name
+
+        # deep copy the configure of model
+        self.conf = copy(conf)
+
+        self.is_training = is_training
+
+        ## the overall loss of this task
+        self.loss = None
+
+        ## outputs may be useful for the other models
+        self.outputs = {}
+
+        ## the prediction of this task
+        self.predict = []
+
+    def create_model(self, 
+                      args,
+                      reader_input,
+                      base_model = None):
+        """
+            given the base model, reader_input
+            return the create fn for create this model
+        """
+
+        def _create_model():
+        
+            src_ids, pos_ids, sent_ids, input_mask = reader_input
+
+            bert_conf = JsonConfig(self.conf["bert_conf_file"])
+            self.bert = BertModel(
+                src_ids = src_ids,
+                position_ids = pos_ids,
+                sentence_ids = sent_ids,
+                input_mask = input_mask,
+                config = bert_conf,
+                use_fp16 = args.use_fp16,
+                model_name = self.name)
+                    
+            self.loss = None
+            self.outputs = {
+                "sequence_output":self.bert.get_sequence_output(),
+            }
+
+        return _create_model
+
+
+    def get_output(self, name):
+        return self.outputs[name]
+
+
+    def get_outputs(self):
+        return self.outputs
+
+    def get_predict(self):
+        return self.predict
+
+
+if __name__ == "__main__":
+    
+    bert_model = ModelBERT(conf = {"json_conf_path" : "./data/pretrained_models/squad2_model/bert_config.json"})
+
+    
+    
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/model/mlm_net.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import paddle.fluid as fluid
+
+from model.transformer_encoder import pre_process_layer
+from utils.configure import JsonConfig
+
+
+def compute_loss(output_tensors, args=None):
+    """Compute loss for mlm model"""
+    fc_out = output_tensors['mlm_out']
+    mask_label = output_tensors['mask_label']
+    mask_lm_loss = fluid.layers.softmax_with_cross_entropy(
+        logits=fc_out, label=mask_label)
+    mean_mask_lm_loss = fluid.layers.mean(mask_lm_loss)
+    return mean_mask_lm_loss
+
+
+def create_model(reader_input, base_model=None, is_training=True, args=None):
+    """
+        given the base model, reader_input
+        return the output tensors
+    """
+    mask_label, mask_pos = reader_input
+
+    config = JsonConfig(args.bert_config_path)
+
+    _emb_size = config['hidden_size']
+    _voc_size = config['vocab_size']
+    _hidden_act = config['hidden_act']
+
+    _word_emb_name = "word_embedding"
+    _dtype = "float16" if args.use_fp16 else "float32"
+
+    _param_initializer = fluid.initializer.TruncatedNormal(
+        scale=config['initializer_range'])
+
+    mask_pos = fluid.layers.cast(x=mask_pos, dtype='int32')
+
+    enc_out = base_model.get_output("sequence_output")
+
+    # extract the first token feature in each sentence
+    reshaped_emb_out = fluid.layers.reshape(
+        x=enc_out, shape=[-1, _emb_size])
+    # extract masked tokens' feature
+    mask_feat = fluid.layers.gather(input=reshaped_emb_out, index=mask_pos)
+    num_seqs = fluid.layers.fill_constant(shape=[1], value=512, dtype='int64')
+
+    # transform: fc
+    mask_trans_feat = fluid.layers.fc(
+        input=mask_feat,
+        size=_emb_size,
+        act=_hidden_act,
+        param_attr=fluid.ParamAttr(
+            name='mask_lm_trans_fc.w_0',
+            initializer=_param_initializer),
+        bias_attr=fluid.ParamAttr(name='mask_lm_trans_fc.b_0'))
+    # transform: layer norm
+    mask_trans_feat = pre_process_layer(
+        mask_trans_feat, 'n', name='mask_lm_trans')
+
+    mask_lm_out_bias_attr = fluid.ParamAttr(
+        name="mask_lm_out_fc.b_0",
+        initializer=fluid.initializer.Constant(value=0.0))
+
+    fc_out = fluid.layers.matmul(
+        x=mask_trans_feat,
+        y=fluid.default_main_program().global_block().var(
+            _word_emb_name),
+        transpose_y=True)
+    fc_out += fluid.layers.create_parameter(
+        shape=[_voc_size],
+        dtype=_dtype,
+        attr=mask_lm_out_bias_attr,
+        is_bias=True)
+
+    output_tensors = {}
+    output_tensors['num_seqs'] = num_seqs
+    output_tensors['mlm_out'] = fc_out
+    output_tensors['mask_label'] = mask_label
+
+    return output_tensors
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/model/mrqa_net.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import paddle.fluid as fluid
+
+
+def compute_loss(output_tensors, args=None):
+    """Compute loss for mrc model"""
+    def _compute_single_loss(logits, positions):
+        """Compute start/end loss for mrc model"""
+        loss = fluid.layers.softmax_with_cross_entropy(
+            logits=logits, label=positions)
+        loss = fluid.layers.mean(x=loss)
+        return loss
+
+    start_logits = output_tensors['start_logits']
+    end_logits = output_tensors['end_logits']
+    start_positions = output_tensors['start_positions']
+    end_positions = output_tensors['end_positions']
+    start_loss = _compute_single_loss(start_logits, start_positions)
+    end_loss = _compute_single_loss(end_logits, end_positions)
+    total_loss = (start_loss + end_loss) / 2.0
+    if args.use_fp16 and args.loss_scaling > 1.0:
+        total_loss = total_loss * args.loss_scaling
+
+    return total_loss
+
+
+def compute_distill_loss(output_tensors, args=None): 
+    """Compute loss for mrc model"""
+    start_logits = output_tensors['start_logits']
+    end_logits = output_tensors['end_logits']
+    start_logits_truth = output_tensors['start_logits_truth']
+    end_logits_truth = output_tensors['end_logits_truth']
+    input_mask = output_tensors['input_mask']
+    def _mask(logits, input_mask, nan=1e5):
+        input_mask = fluid.layers.reshape(input_mask, [-1, 512])
+        logits = logits - (1.0 - input_mask) * nan
+        return logits
+    start_logits = _mask(start_logits, input_mask)
+    end_logits = _mask(end_logits, input_mask)
+    start_logits_truth = _mask(start_logits_truth, input_mask)
+    end_logits_truth = _mask(end_logits_truth, input_mask)
+    start_logits_truth = fluid.layers.reshape(start_logits_truth, [-1, 512])
+    end_logits_truth = fluid.layers.reshape(end_logits_truth, [-1, 512])
+    T = 1.0
+    start_logits_softmax = fluid.layers.softmax(input=start_logits/T)
+    end_logits_softmax = fluid.layers.softmax(input=end_logits/T)
+    start_logits_truth_softmax = fluid.layers.softmax(input=start_logits_truth/T)
+    end_logits_truth_softmax = fluid.layers.softmax(input=end_logits_truth/T)
+    start_logits_truth_softmax.stop_gradient = True
+    end_logits_truth_softmax.stop_gradient = True
+    start_loss = fluid.layers.cross_entropy(start_logits_softmax, start_logits_truth_softmax, soft_label=True)
+    end_loss = fluid.layers.cross_entropy(end_logits_softmax, end_logits_truth_softmax, soft_label=True)
+    start_loss = fluid.layers.mean(x=start_loss)
+    end_loss = fluid.layers.mean(x=end_loss)
+    total_loss = (start_loss + end_loss) / 2.0
+    return total_loss
+
+
+def create_model(reader_input, base_model=None, is_training=True, args=None):
+    """
+        given the base model, reader_input
+        return the output tensors
+    """
+
+    if is_training: 
+        if args.do_distill: 
+            src_ids, pos_ids, sent_ids, input_mask, \
+                start_logits_truth, end_logits_truth, start_positions, end_positions = reader_input
+        else: 
+            src_ids, pos_ids, sent_ids, input_mask, \
+                start_positions, end_positions = reader_input
+    else:
+        src_ids, pos_ids, sent_ids, input_mask, unique_id = reader_input
+    enc_out = base_model.get_output("sequence_output")
+    logits = fluid.layers.fc(
+        input=enc_out,
+        size=2,
+        num_flatten_dims=2,
+        param_attr=fluid.ParamAttr(
+            name="cls_squad_out_w",
+            initializer=fluid.initializer.TruncatedNormal(scale=0.02)),
+        bias_attr=fluid.ParamAttr(
+            name="cls_squad_out_b", initializer=fluid.initializer.Constant(0.)))
+
+    logits = fluid.layers.transpose(x=logits, perm=[2, 0, 1])
+    start_logits, end_logits = fluid.layers.unstack(x=logits, axis=0)
+
+    batch_ones = fluid.layers.fill_constant_batch_size_like(
+        input=start_logits, dtype='int64', shape=[1], value=1)
+    num_seqs = fluid.layers.reduce_sum(input=batch_ones)
+
+    output_tensors = {}
+    output_tensors['start_logits'] = start_logits
+    output_tensors['end_logits'] = end_logits
+    output_tensors['num_seqs'] = num_seqs
+    output_tensors['input_mask'] = input_mask
+    if is_training:
+        output_tensors['start_positions'] = start_positions
+        output_tensors['end_positions'] = end_positions
+        if args.do_distill: 
+            output_tensors['start_logits_truth'] = start_logits_truth
+            output_tensors['end_logits_truth'] = end_logits_truth
+
+    else:
+        output_tensors['unique_id'] = unique_id
+        output_tensors['start_logits'] = start_logits
+        output_tensors['end_logits'] = end_logits
+
+    return output_tensors

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/model/transformer_encoder.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Transformer encoder."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from functools import partial
+import numpy as np
+
+import paddle.fluid as fluid
+import paddle.fluid.layers as layers
+from paddle.fluid.layer_helper import LayerHelper
+
+def layer_norm(x, begin_norm_axis=1, epsilon=1e-6, param_attr=None, bias_attr=None):
+    helper = LayerHelper('layer_norm', **locals())
+    mean = layers.reduce_mean(x, dim=begin_norm_axis, keep_dim=True)
+    shift_x = layers.elementwise_sub(x=x, y=mean, axis=0)
+    variance = layers.reduce_mean(layers.square(shift_x), dim=begin_norm_axis, keep_dim=True)
+    r_stdev = layers.rsqrt(variance + epsilon)
+    norm_x = layers.elementwise_mul(x=shift_x, y=r_stdev, axis=0)
+
+    param_shape = [reduce(lambda x, y: x * y, norm_x.shape[begin_norm_axis:])]
+    param_dtype = norm_x.dtype
+    scale = helper.create_parameter(
+        attr=param_attr,
+        shape=param_shape,
+        dtype=param_dtype,
+        default_initializer=fluid.initializer.Constant(1.))
+    bias = helper.create_parameter(
+        attr=bias_attr,
+        shape=param_shape,
+        dtype=param_dtype,
+        is_bias=True,
+        default_initializer=fluid.initializer.Constant(0.))
+
+    out = layers.elementwise_mul(x=norm_x, y=scale, axis=-1)
+    out = layers.elementwise_add(x=out, y=bias, axis=-1)
+
+    return out
+
+def multi_head_attention(queries,
+                         keys,
+                         values,
+                         attn_bias,
+                         d_key,
+                         d_value,
+                         d_model,
+                         n_head=1,
+                         dropout_rate=0.,
+                         cache=None,
+                         param_initializer=None,
+                         name='multi_head_att'):
+    """
+    Multi-Head Attention. Note that attn_bias is added to the logit before
+    computing softmax activiation to mask certain selected positions so that
+    they will not considered in attention weights.
+    """
+    keys = queries if keys is None else keys
+    values = keys if values is None else values
+
+    if not (len(queries.shape) == len(keys.shape) == len(values.shape) == 3):
+        raise ValueError(
+            "Inputs: quries, keys and values should all be 3-D tensors.")
+
+    def __compute_qkv(queries, keys, values, n_head, d_key, d_value):
+        """
+        Add linear projection to queries, keys, and values.
+        """
+        q = layers.fc(input = queries,
+                      size = d_key * n_head,
+                      num_flatten_dims = 2,
+                      param_attr = fluid.ParamAttr(
+                          name = name + '_query_fc.w_0',
+                          initializer = param_initializer),
+                      bias_attr = name + '_query_fc.b_0')
+        k = layers.fc(input = keys,
+                      size = d_key * n_head,
+                      num_flatten_dims = 2,
+                      param_attr = fluid.ParamAttr(
+                          name = name + '_key_fc.w_0',
+                          initializer = param_initializer),
+                      bias_attr = name + '_key_fc.b_0')
+        v = layers.fc(input = values,
+                      size = d_value * n_head,
+                      num_flatten_dims = 2,
+                      param_attr = fluid.ParamAttr(
+                          name = name + '_value_fc.w_0',
+                          initializer = param_initializer),
+                      bias_attr = name + '_value_fc.b_0')
+        return q, k, v
+
+    def __split_heads(x, n_head):
+        """
+        Reshape the last dimension of inpunt tensor x so that it becomes two
+        dimensions and then transpose. Specifically, input a tensor with shape
+        [bs, max_sequence_length, n_head * hidden_dim] then output a tensor
+        with shape [bs, n_head, max_sequence_length, hidden_dim].
+        """
+        hidden_size = x.shape[-1]
+        # The value 0 in shape attr means copying the corresponding dimension
+        # size of the input as the output dimension size.
+        reshaped = layers.reshape(
+            x = x, shape = [0, 0, n_head, hidden_size // n_head], inplace=False)
+
+        # permuate the dimensions into:
+        # [batch_size, n_head, max_sequence_len, hidden_size_per_head]
+        return layers.transpose(x=reshaped, perm=[0, 2, 1, 3])
+
+    def __combine_heads(x):
+        """
+        Transpose and then reshape the last two dimensions of inpunt tensor x
+        so that it becomes one dimension, which is reverse to __split_heads.
+        """
+        if len(x.shape) == 3: return x
+        if len(x.shape) != 4:
+            raise ValueError("Input(x) should be a 4-D Tensor.")
+
+        trans_x = layers.transpose(x, perm=[0, 2, 1, 3])
+        # The value 0 in shape attr means copying the corresponding dimension
+        # size of the input as the output dimension size.
+        return layers.reshape(
+            x = trans_x,
+            shape = [0, 0, trans_x.shape[2] * trans_x.shape[3]],
+            inplace = False)
+
+    def scaled_dot_product_attention(q, k, v, attn_bias, d_key, dropout_rate):
+        """
+        Scaled Dot-Product Attention
+        """
+        scaled_q = layers.scale(x = q, scale = d_key**-0.5)
+        product = layers.matmul(x = scaled_q, y = k, transpose_y = True)
+        if attn_bias:
+            product += attn_bias
+        weights = layers.softmax(product)
+        if dropout_rate:
+            weights = layers.dropout(
+                weights,
+                dropout_prob=dropout_rate,
+                dropout_implementation="upscale_in_train",
+                is_test=False)
+        out = layers.matmul(weights, v)
+        return out
+
+    q, k, v = __compute_qkv(queries, keys, values, n_head, d_key, d_value)
+
+    if cache is not None:  # use cache and concat time steps
+        # Since the inplace reshape in __split_heads changes the shape of k and
+        # v, which is the cache input for next time step, reshape the cache
+        # input from the previous time step first.
+        k = cache["k"] = layers.concat(
+            [layers.reshape(
+                cache["k"], shape=[0, 0, d_model]), k], axis=1)
+        v = cache["v"] = layers.concat(
+            [layers.reshape(
+                cache["v"], shape=[0, 0, d_model]), v], axis=1)
+
+    q = __split_heads(q, n_head)
+    k = __split_heads(k, n_head)
+    v = __split_heads(v, n_head)
+
+    ctx_multiheads = scaled_dot_product_attention(q, k, v, attn_bias, d_key,
+                                                  dropout_rate)
+
+    out = __combine_heads(ctx_multiheads)
+
+    # Project back to the model size.
+    proj_out = layers.fc(input = out,
+                         size = d_model,
+                         num_flatten_dims = 2,
+                         param_attr=fluid.ParamAttr(
+                             name = name + '_output_fc.w_0',
+                             initializer = param_initializer),
+                         bias_attr = name + '_output_fc.b_0')
+    return proj_out
+
+
+def positionwise_feed_forward(x,
+                              d_inner_hid,
+                              d_hid,
+                              dropout_rate,
+                              hidden_act,
+                              param_initializer=None,
+                              name='ffn'):
+    """
+    Position-wise Feed-Forward Networks.
+    This module consists of two linear transformations with a ReLU activation
+    in between, which is applied to each position separately and identically.
+    """
+    hidden = layers.fc(input=x,
+                       size=d_inner_hid,
+                       num_flatten_dims=2,
+                       act=hidden_act,
+                       param_attr=fluid.ParamAttr(
+                           name=name + '_fc_0.w_0',
+                           initializer=param_initializer),
+                       bias_attr=name + '_fc_0.b_0')
+    if dropout_rate:
+        hidden = layers.dropout(
+            hidden,
+            dropout_prob=dropout_rate,
+            dropout_implementation="upscale_in_train",
+            is_test = False)
+
+    out = layers.fc(input = hidden,
+                    size = d_hid,
+                    num_flatten_dims = 2,
+                    param_attr=fluid.ParamAttr(
+                        name = name + '_fc_1.w_0', 
+                        initializer = param_initializer),
+                    bias_attr = name + '_fc_1.b_0')
+    return out
+
+
+def pre_post_process_layer(prev_out, out, process_cmd, dropout_rate=0.,
+                           name=''):
+    """
+    Add residual connection, layer normalization and droput to the out tensor
+    optionally according to the value of process_cmd.
+    This will be used before or after multi-head attention and position-wise
+    feed-forward networks.
+    """
+    for cmd in process_cmd:
+        if cmd == "a":  # add residual connection
+            out = out + prev_out if prev_out else out
+        elif cmd == "n":  # add layer normalization
+            out_dtype = out.dtype
+            if out_dtype == fluid.core.VarDesc.VarType.FP16:
+                out = layers.cast(x = out, dtype = "float32")
+            out = layer_norm(
+                out,
+                begin_norm_axis=len(out.shape) - 1,
+                param_attr=fluid.ParamAttr(
+                    name = name + '_layer_norm_scale',
+                    initializer = fluid.initializer.Constant(1.)),
+                bias_attr=fluid.ParamAttr(
+                    name = name + '_layer_norm_bias',
+                    initializer = fluid.initializer.Constant(0.)))
+            if out_dtype == fluid.core.VarDesc.VarType.FP16:
+                out = layers.cast(x = out, dtype = "float16")
+        elif cmd == "d":  # add dropout
+            if dropout_rate:
+                out = layers.dropout(
+                    out,
+                    dropout_prob = dropout_rate,
+                    dropout_implementation = "upscale_in_train",
+                    is_test = False)
+    return out
+
+
+pre_process_layer = partial(pre_post_process_layer, None)
+post_process_layer = pre_post_process_layer
+
+def encoder_layer(enc_input,
+                  attn_bias,
+                  n_head,
+                  d_key,
+                  d_value,
+                  d_model,
+                  d_inner_hid,
+                  prepostprocess_dropout,
+                  attention_dropout,
+                  relu_dropout,
+                  hidden_act,
+                  preprocess_cmd="n",
+                  postprocess_cmd="da",
+                  param_initializer=None,
+                  name=''):
+    """The encoder layers that can be stacked to form a deep encoder.
+    This module consits of a multi-head (self) attention followed by
+    position-wise feed-forward networks and both the two components companied
+    with the post_process_layer to add residual connection, layer normalization
+    and droput.
+    """
+    attn_output = multi_head_attention(
+        pre_process_layer(
+            enc_input,
+            preprocess_cmd,
+            prepostprocess_dropout,
+            name=name + '_pre_att'),
+        None,
+        None,
+        attn_bias,
+        d_key,
+        d_value,
+        d_model,
+        n_head,
+        attention_dropout,
+        param_initializer = param_initializer,
+        name = name + '_multi_head_att')
+    attn_output = post_process_layer(
+        enc_input,
+        attn_output,
+        postprocess_cmd,
+        prepostprocess_dropout,
+        name = name + '_post_att')
+    ffd_output = positionwise_feed_forward(
+        pre_process_layer(
+            attn_output,
+            preprocess_cmd,
+            prepostprocess_dropout,
+            name = name + '_pre_ffn'),
+        d_inner_hid,
+        d_model,
+        relu_dropout,
+        hidden_act,
+        param_initializer = param_initializer,
+        name = name + '_ffn')
+    return post_process_layer(
+        attn_output,
+        ffd_output,
+        postprocess_cmd,
+        prepostprocess_dropout,
+        name = name + '_post_ffn')
+
+
+def encoder(enc_input,
+            attn_bias,
+            n_layer,
+            n_head,
+            d_key,
+            d_value,
+            d_model,
+            d_inner_hid,
+            prepostprocess_dropout,
+            attention_dropout,
+            relu_dropout,
+            hidden_act,
+            preprocess_cmd="n",
+            postprocess_cmd="da",
+            param_initializer=None,
+            name='',
+            return_all = False):
+    """
+    The encoder is composed of a stack of identical layers returned by calling
+    encoder_layer.
+    """
+    enc_outputs = []
+    for i in range(n_layer):
+        enc_output = encoder_layer(
+            enc_input,
+            attn_bias,
+            n_head,
+            d_key,
+            d_value,
+            d_model,
+            d_inner_hid,
+            prepostprocess_dropout,
+            attention_dropout,
+            relu_dropout,
+            hidden_act,
+            preprocess_cmd,
+            postprocess_cmd,
+            param_initializer = param_initializer,
+            name = name + '_layer_' + str(i))
+        enc_input = enc_output
+        if i < n_layer - 1:
+            enc_outputs.append(enc_output)
+
+    enc_output = pre_process_layer(
+        enc_output, preprocess_cmd, prepostprocess_dropout, name="post_encoder")
+    enc_outputs.append(enc_output)
+
+    if not return_all:
+        return enc_output
+    else:
+        return enc_output, enc_outputs

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/optimizer/optimization.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Optimization and learning rate scheduling."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+import paddle.fluid as fluid
+from utils.fp16 import create_master_params_grads, master_param_to_train_param
+
+
+def linear_warmup_decay(learning_rate, warmup_steps, num_train_steps):
+    """ Applies linear warmup of learning rate from 0 and decay to 0."""
+    with fluid.default_main_program()._lr_schedule_guard():
+        lr = fluid.layers.tensor.create_global_var(
+            shape=[1],
+            value=0.0,
+            dtype='float32',
+            persistable=True,
+            name="scheduled_learning_rate")
+
+        global_step = fluid.layers.learning_rate_scheduler._decay_step_counter()
+
+        with fluid.layers.control_flow.Switch() as switch:
+            with switch.case(global_step < warmup_steps):
+                warmup_lr = learning_rate * (global_step / warmup_steps)
+                fluid.layers.tensor.assign(warmup_lr, lr)
+            with switch.default():
+                decayed_lr = fluid.layers.learning_rate_scheduler.polynomial_decay(
+                    learning_rate=learning_rate,
+                    decay_steps=num_train_steps,
+                    end_learning_rate=0.0,
+                    power=1.0,
+                    cycle=False)
+                fluid.layers.tensor.assign(decayed_lr, lr)
+
+        return lr
+
+
+def optimization(loss,
+                 warmup_steps,
+                 num_train_steps,
+                 learning_rate,
+                 train_program,
+                 startup_prog,
+                 weight_decay,
+                 scheduler='linear_warmup_decay',
+                 use_fp16=False,
+                 loss_scaling=1.0):
+    if warmup_steps > 0:
+        if scheduler == 'noam_decay':
+            scheduled_lr = fluid.layers.learning_rate_scheduler\
+             .noam_decay(1/(warmup_steps *(learning_rate ** 2)),
+                         warmup_steps)
+        elif scheduler == 'linear_warmup_decay':
+            scheduled_lr = linear_warmup_decay(learning_rate, warmup_steps,
+                                               num_train_steps)
+        else:
+            raise ValueError("Unkown learning rate scheduler, should be "
+                             "'noam_decay' or 'linear_warmup_decay'")
+        optimizer = fluid.optimizer.Adam(learning_rate=scheduled_lr)
+    else:
+        optimizer = fluid.optimizer.Adam(learning_rate=learning_rate)
+        scheduled_lr = learning_rate
+
+    clip_norm_thres = 1.0
+    # When using mixed precision training, scale the gradient clip threshold
+    # by loss_scaling
+    if use_fp16 and loss_scaling > 1.0:
+        clip_norm_thres *= loss_scaling
+    fluid.clip.set_gradient_clip(
+        clip=fluid.clip.GradientClipByGlobalNorm(clip_norm=clip_norm_thres))
+
+    def exclude_from_weight_decay(name):
+        if name.find("layer_norm") > -1:
+            return True
+        bias_suffix = ["_bias", "_b", ".b_0"]
+        for suffix in bias_suffix:
+            if name.endswith(suffix):
+                return True
+        return False
+
+    param_list = dict()
+
+    if use_fp16:
+        param_grads = optimizer.backward(loss)
+        master_param_grads = create_master_params_grads(
+            param_grads, train_program, startup_prog, loss_scaling)
+
+        for param, _ in master_param_grads:
+            param_list[param.name] = param * 1.0
+            param_list[param.name].stop_gradient = True
+
+        optimizer.apply_gradients(master_param_grads)
+
+        if weight_decay > 0:
+            for param, grad in master_param_grads:
+                if exclude_from_weight_decay(param.name.rstrip(".master")):
+                    continue
+                with param.block.program._optimized_guard(
+                    [param, grad]), fluid.framework.name_scope("weight_decay"):
+                    updated_param = param - param_list[
+                        param.name] * weight_decay * scheduled_lr
+                    fluid.layers.assign(output=param, input=updated_param)
+
+        master_param_to_train_param(master_param_grads, param_grads,
+                                    train_program)
+
+    else:
+        for param in train_program.global_block().all_parameters():
+            param_list[param.name] = param * 1.0
+            param_list[param.name].stop_gradient = True
+
+        _, param_grads = optimizer.minimize(loss)
+
+        if weight_decay > 0:
+            for param, grad in param_grads:
+                if exclude_from_weight_decay(param.name):
+                    continue
+                with param.block.program._optimized_guard(
+                    [param, grad]), fluid.framework.name_scope("weight_decay"):
+                    updated_param = param - param_list[
+                        param.name] * weight_decay * scheduled_lr
+                    fluid.layers.assign(output=param, input=updated_param)
+
+    return scheduled_lr

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/reader/joint_reader.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+import random
+import numpy as np
+
+import paddle
+import paddle.fluid as fluid
+
+from utils.placeholder import Placeholder
+
+
+def repeat(reader):
+    """Repeat a generator forever"""
+    generator = reader()
+    while True:
+        try:
+            yield next(generator)
+        except StopIteration:
+            generator = reader()
+            yield next(generator)
+
+
+def create_joint_generator(input_shape, generators, do_distill, is_multi_task=True):
+
+    def empty_output(input_shape, batch_size=1):
+        results = []
+        for i in range(len(input_shape)):
+            if input_shape[i][1] == 'int32':
+                dtype = np.int32
+            if input_shape[i][1] == 'int64':
+                dtype = np.int64
+            if input_shape[i][1] == 'float32':
+                dtype = np.float32
+            if input_shape[i][1] == 'float64':
+                dtype = np.float64
+            shape = input_shape[i][0]
+            shape[0] = batch_size
+            pad_tensor = np.zeros(shape=shape, dtype=dtype)
+            results.append(pad_tensor)
+        return results
+
+    def wrapper(): 
+        """wrapper data"""
+        generators_inst = [repeat(gen[0]) for gen in generators]
+
+        generators_ratio = [gen[1] for gen in generators]
+        weights = [ratio/sum(generators_ratio) for ratio in generators_ratio]
+        run_task_id = range(len(generators))
+        while True:
+            idx = np.random.choice(run_task_id, p=weights)
+            gen_results = next(generators_inst[idx])
+            if not gen_results:
+                break
+            batch_size = gen_results[0].shape[0]
+            results = empty_output(input_shape, batch_size)
+
+            task_id_tensor = np.array([[idx]]).astype("int64")
+            results[0] = task_id_tensor
+            for i in range(4):
+                results[i+1] = gen_results[i]
+            if do_distill: 
+                if idx == 0: 
+                    results[5] = gen_results[4]
+                    results[6] = gen_results[5]
+                    results[7] = gen_results[6]
+                    results[8] = gen_results[7]
+                else: 
+                    results[9] = gen_results[4]
+                    results[10] = gen_results[5]
+
+            else: 
+                if idx == 0:
+                    # mrc batch
+                    results[5] = gen_results[4]
+                    results[6] = gen_results[5]
+                elif idx == 1:
+                    # mlm batch
+                    results[7] = gen_results[4]
+                    results[8] = gen_results[5]
+            # idx stands for the task index
+            yield results
+
+    return wrapper
+
+
+def create_reader(reader_name, input_shape, is_multi_task, do_distill, *gens):
+    """
+    build reader for multi_task_learning
+    """
+    placeholder = Placeholder(input_shape)
+    pyreader, model_inputs = placeholder.build(capacity=100, reader_name=reader_name)
+    joint_generator = create_joint_generator(input_shape, gens[0], do_distill, is_multi_task=is_multi_task)
+
+    return joint_generator, pyreader, model_inputs
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/reader/mlm_reader.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+from __future__ import division
+
+import os
+import re
+import six
+import gzip
+import types
+import logging
+import numpy as np
+import collections
+
+import paddle
+import paddle.fluid as fluid
+
+from utils import tokenization
+from utils.batching import prepare_batch_data
+
+
+class DataReader(object):
+    def __init__(self,
+                 data_dir,
+                 vocab_path,
+                 batch_size=4096,
+                 in_tokens=True,
+                 max_seq_len=512,
+                 shuffle_files=True,
+                 epoch=100,
+                 voc_size=0,
+                 is_test=False,
+                 generate_neg_sample=False):
+
+        self.vocab = self.load_vocab(vocab_path)
+        self.data_dir = data_dir
+        self.batch_size = batch_size
+        self.in_tokens = in_tokens
+        self.shuffle_files = shuffle_files
+        self.epoch = epoch
+        self.current_epoch = 0
+        self.current_file_index = 0
+        self.total_file = 0
+        self.current_file = None
+        self.voc_size = voc_size
+        self.max_seq_len = max_seq_len
+        self.pad_id = self.vocab["[PAD]"]
+        self.cls_id = self.vocab["[CLS]"]
+        self.sep_id = self.vocab["[SEP]"]
+        self.mask_id = self.vocab["[MASK]"]
+        self.is_test = is_test
+        self.generate_neg_sample = generate_neg_sample
+        if self.in_tokens:
+            assert self.batch_size >= self.max_seq_len, "The number of " \
+                   "tokens in batch should not be smaller than max seq length."
+
+        if self.is_test:
+            self.epoch = 1
+            self.shuffle_files = False
+
+    def get_progress(self):
+        """return current progress of traning data
+        """
+        return self.current_epoch, self.current_file_index, self.total_file, self.current_file
+
+    def parse_line(self, line, max_seq_len=512):
+        """ parse one line to token_ids, sentence_ids, pos_ids, label
+        """
+        line = line.strip().decode().split(";")
+        assert len(line) == 4, "One sample must have 4 fields!"
+        (token_ids, sent_ids, pos_ids, label) = line
+        token_ids = [int(token) for token in token_ids.split(" ")]
+        sent_ids = [int(token) for token in sent_ids.split(" ")]
+        pos_ids = [int(token) for token in pos_ids.split(" ")]
+        assert len(token_ids) == len(sent_ids) == len(
+            pos_ids
+        ), "[Must be true]len(token_ids) == len(sent_ids) == len(pos_ids)"
+        label = int(label)
+        if len(token_ids) > max_seq_len:
+            return None
+        return [token_ids, sent_ids, pos_ids, label]
+
+    def read_file(self, file):
+        assert file.endswith('.gz'), "[ERROR] %s is not a gzip file" % file
+        file_path = self.data_dir + "/" + file
+        with gzip.open(file_path, "rb") as f:
+            for line in f:
+                parsed_line = self.parse_line(
+                    line, max_seq_len=self.max_seq_len)
+                if parsed_line is None:
+                    continue
+                yield parsed_line
+
+    def convert_to_unicode(self, text):
+        """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
+        if six.PY3:
+            if isinstance(text, str):
+                return text
+            elif isinstance(text, bytes):
+                return text.decode("utf-8", "ignore")
+            else:
+                raise ValueError("Unsupported string type: %s" % (type(text)))
+        elif six.PY2:
+            if isinstance(text, str):
+                return text.decode("utf-8", "ignore")
+            elif isinstance(text, unicode):
+                return text
+            else:
+                raise ValueError("Unsupported string type: %s" % (type(text)))
+        else:
+            raise ValueError("Not running on Python2 or Python 3?")
+
+    def load_vocab(self, vocab_file):
+        """Loads a vocabulary file into a dictionary."""
+        vocab = collections.OrderedDict()
+        fin = open(vocab_file)
+        for num, line in enumerate(fin):
+            items = self.convert_to_unicode(line.strip()).split("\t")
+            if len(items) > 2:
+                break
+            token = items[0]
+            index = items[1] if len(items) == 2 else num
+            token = token.strip()
+            vocab[token] = int(index)
+        return vocab
+
+    def random_pair_neg_samples(self, pos_samples):
+        """ randomly generate negtive samples using pos_samples
+
+            Args:
+                pos_samples: list of positive samples
+            
+            Returns:
+                neg_samples: list of negtive samples
+        """
+        np.random.shuffle(pos_samples)
+        num_sample = len(pos_samples)
+        neg_samples = []
+        miss_num = 0
+
+        for i in range(num_sample):
+            pair_index = (i + 1) % num_sample
+            origin_src_ids = pos_samples[i][0]
+            origin_sep_index = origin_src_ids.index(2)
+            pair_src_ids = pos_samples[pair_index][0]
+            pair_sep_index = pair_src_ids.index(2)
+
+            src_ids = origin_src_ids[:origin_sep_index + 1] + pair_src_ids[
+                pair_sep_index + 1:]
+            if len(src_ids) >= self.max_seq_len:
+                miss_num += 1
+                continue
+            sent_ids = [0] * len(origin_src_ids[:origin_sep_index + 1]) + [
+                1
+            ] * len(pair_src_ids[pair_sep_index + 1:])
+            pos_ids = list(range(len(src_ids)))
+            neg_sample = [src_ids, sent_ids, pos_ids, 0]
+            assert len(src_ids) == len(sent_ids) == len(
+                pos_ids
+            ), "[ERROR]len(src_id) == lne(sent_id) == len(pos_id) must be True"
+            neg_samples.append(neg_sample)
+        return neg_samples, miss_num
+
+    def mixin_negtive_samples(self, pos_sample_generator, buffer=1000):
+        """ 1. generate negtive samples by randomly group sentence_1 and sentence_2 of positive samples
+            2. combine negtive samples and positive samples
+            
+            Args:
+                pos_sample_generator: a generator producing a parsed positive sample, which is a list: [token_ids, sent_ids, pos_ids, 1]
+
+            Returns:
+                sample: one sample from shuffled positive samples and negtive samples
+        """
+        pos_samples = []
+        num_total_miss = 0
+        pos_sample_num = 0
+        try:
+            while True:
+                while len(pos_samples) < buffer:
+                    pos_sample = next(pos_sample_generator)
+                    label = pos_sample[3]
+                    assert label == 1, "positive sample's label must be 1"
+                    pos_samples.append(pos_sample)
+                    pos_sample_num += 1
+
+                neg_samples, miss_num = self.random_pair_neg_samples(
+                    pos_samples)
+                num_total_miss += miss_num
+                samples = pos_samples + neg_samples
+                pos_samples = []
+                np.random.shuffle(samples)
+                for sample in samples:
+                    yield sample
+        except StopIteration:
+            print("stopiteration: reach end of file")
+            if len(pos_samples) == 1:
+                yield pos_samples[0]
+            elif len(pos_samples) == 0:
+                yield None
+            else:
+                neg_samples, miss_num = self.random_pair_neg_samples(
+                    pos_samples)
+                num_total_miss += miss_num
+                samples = pos_samples + neg_samples
+                pos_samples = []
+                np.random.shuffle(samples)
+                for sample in samples:
+                    yield sample
+            print("miss_num:%d\tideal_total_sample_num:%d\tmiss_rate:%f" %
+                  (num_total_miss, pos_sample_num * 2,
+                   num_total_miss / (pos_sample_num * 2)))
+
+    def data_generator(self):
+        """
+        data_generator
+        """
+        files = os.listdir(self.data_dir)
+        self.total_file = len(files)
+        assert self.total_file > 0, "[Error] data_dir is empty"
+
+        def wrapper():
+            def reader():
+                for epoch in range(self.epoch):
+                    self.current_epoch = epoch + 1
+                    if self.shuffle_files:
+                        np.random.shuffle(files)
+                    for index, file in enumerate(files):
+                        self.current_file_index = index + 1
+                        self.current_file = file
+                        sample_generator = self.read_file(file)
+                        if not self.is_test and self.generate_neg_sample:
+                            sample_generator = self.mixin_negtive_samples(
+                                sample_generator)
+                        for sample in sample_generator:
+                            if sample is None:
+                                continue
+                            yield sample
+
+            def batch_reader(reader, batch_size, in_tokens):
+                batch, total_token_num, max_len = [], 0, 0
+                for parsed_line in reader():
+                    token_ids, sent_ids, pos_ids, label = parsed_line
+                    max_len = max(max_len, len(token_ids))
+                    if in_tokens:
+                        to_append = (len(batch) + 1) * max_len <= batch_size
+                    else:
+                        to_append = len(batch) < batch_size
+                    if to_append:
+                        batch.append(parsed_line)
+                        total_token_num += len(token_ids)
+                    else:
+                        yield batch, total_token_num
+                        batch, total_token_num, max_len = [parsed_line], len(
+                            token_ids), len(token_ids)
+
+                if len(batch) > 0:
+                    yield batch, total_token_num
+
+            for batch_data, total_token_num in batch_reader(
+                    reader, self.batch_size, self.in_tokens):
+                yield prepare_batch_data(
+                    batch_data,
+                    total_token_num,
+                    voc_size=self.voc_size,
+                    pad_id=self.pad_id,
+                    cls_id=self.cls_id,
+                    sep_id=self.sep_id,
+                    mask_id=self.mask_id,
+                    max_len=self.max_seq_len,
+                    return_input_mask=True,
+                    return_max_len=False,
+                    return_num_token=False)
+
+        return wrapper
+
+
+if __name__ == "__main__":
+    pass

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/reader/mrqa_distill_reader.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run MRQA"""
+
+import six
+import math
+import json
+import random
+import collections
+import numpy as np
+
+from utils import tokenization
+from utils.batching import prepare_batch_data
+
+
+class DataProcessorDistill(object): 
+    def __init__(self): 
+        self.num_examples = -1
+        self.current_train_example = -1
+        self.current_train_epoch = -1
+
+    def get_features(self, data_path): 
+        with open(data_path, 'r') as fr: 
+            for line in fr: 
+                yield line.strip()
+
+    def data_generator(self, 
+                       data_file, 
+                       batch_size, 
+                       max_len, 
+                       in_tokens, 
+                       dev_count,
+                       epochs,
+                       shuffle): 
+        self.num_examples = len([ "" for line in open(data_file,"r")])
+        def batch_reader(data_file, in_tokens, batch_size): 
+            batch = []
+            index = 0
+            for feature in self.get_features(data_file): 
+                to_append = len(batch) < batch_size
+                if to_append: 
+                    batch.append(feature)
+                else: 
+                    yield batch
+                    batch = []
+            if len(batch) > 0: 
+                yield batch
+
+        def wrapper(): 
+            for epoch in range(epochs): 
+                all_batches = []
+                for batch_data in batch_reader(data_file, in_tokens, batch_size): 
+                    batch_data_segment = []
+                    for feature in batch_data: 
+                        data = json.loads(feature.strip())
+                        example_index = data['example_index']
+                        unique_id = data['unique_id']
+                        input_ids = data['input_ids']
+                        position_ids = data['position_ids']
+                        input_mask = data['input_mask']
+                        segment_ids = data['segment_ids']
+                        start_position = data['start_position']
+                        end_position = data['end_position']
+                        start_logits = data['start_logits']
+                        end_logits = data['end_logits']
+                        instance = [input_ids, position_ids, segment_ids, input_mask, start_logits, end_logits, start_position, end_position]
+                        batch_data_segment.append(instance)
+                    batch_data = batch_data_segment
+                    src_ids = [inst[0] for inst in batch_data]
+                    pos_ids = [inst[1] for inst in batch_data]
+                    sent_ids = [inst[2] for inst in batch_data]
+                    input_mask = [inst[3] for inst in batch_data]
+                    start_logits = [inst[4] for inst in batch_data]
+                    end_logits = [inst[5] for inst in batch_data]
+                    src_ids = np.array(src_ids).astype("int64").reshape([-1, max_len, 1])
+                    pos_ids = np.array(pos_ids).astype("int64").reshape([-1, max_len, 1])
+                    sent_ids = np.array(sent_ids).astype("int64").reshape([-1, max_len, 1])
+                    input_mask = np.array(input_mask).astype("float32").reshape([-1, max_len, 1])
+                    start_logits = np.array(start_logits).astype("float32").reshape([-1, max_len])
+                    end_logits = np.array(end_logits).astype("float32").reshape([-1, max_len])
+                    start_positions = [inst[6] for inst in batch_data]
+                    end_positions = [inst[7] for inst in batch_data]
+                    start_positions = np.array(start_positions).astype("int64").reshape([-1, 1])
+                    end_positions = np.array(end_positions).astype("int64").reshape([-1, 1])
+                    batch_data = [src_ids, pos_ids, sent_ids, input_mask, start_logits, end_logits, start_positions, end_positions]
+
+                    if len(all_batches) < dev_count:
+                        all_batches.append(batch_data)
+
+                    if len(all_batches) == dev_count: 
+                        for batch in all_batches: 
+                            yield batch
+                        all_batches = []
+        return wrapper

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/run_distill.sh

+#!/bin/bash
+
+export FLAGS_sync_nccl_allreduce=0
+export FLAGS_eager_delete_tensor_gb=1
+
+export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+if  [ ! "$CUDA_VISIBLE_DEVICES" ]
+then
+    export CPU_NUM=1
+    use_cuda=false
+else
+    use_cuda=true
+fi
+
+# path of pre_train model
+INPUT_PATH="data/input"
+PRETRAIN_MODEL_PATH="data/pretrain_model/squad2_model"
+# path to save checkpoint
+CHECKPOINT_PATH="data/output/output_mrqa"
+mkdir -p $CHECKPOINT_PATH
+
+python -u train.py --use_cuda ${use_cuda}\
+        --batch_size 8 \
+        --in_tokens false \
+        --init_pretraining_params ${PRETRAIN_MODEL_PATH}/params \
+        --checkpoints $CHECKPOINT_PATH \
+        --vocab_path ${PRETRAIN_MODEL_PATH}/vocab.txt \
+        --do_distill true \
+        --do_train true \
+        --do_predict true \
+        --save_steps 10000 \
+        --warmup_proportion 0.1 \
+        --weight_decay  0.01 \
+        --sample_rate 0.02 \
+        --epoch 2 \
+        --max_seq_len 512 \
+        --bert_config_path ${PRETRAIN_MODEL_PATH}/bert_config.json \
+        --predict_file ${INPUT_PATH}/mrqa_distill_data/mrqa-combined.all_dev.raw.json \
+        --do_lower_case false \
+        --doc_stride 128 \
+        --train_file ${INPUT_PATH}/mrqa_distill_data/mrqa_distill.json \
+        --mlm_path ${INPUT_PATH}/mlm_data \
+        --mix_ratio 2.0 \
+        --learning_rate 3e-5 \
+        --lr_scheduler linear_warmup_decay \
+        --skip_steps 100 
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/run_evaluation.sh

+#!/usr/bin/env bash
+# ==============================================================================
+# Copyright 2017 Baidu.com, Inc. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+# path of dev data
+PATH_dev=./data/input/mrqa_evaluation_dataset
+
+
+# path of dev predict
+KD_prediction=./prediction_results/KD_ema_predictions.json
+
+files=$(ls ./prediction_results/*.log 2> /dev/null | wc -l)
+if [ "$files" != "0" ];
+then
+    rm prediction_results/*.log
+fi
+
+# evaluation KD model
+echo "evaluate knowledge distillation model........................................."
+for dataset in `ls $PATH_dev/in_domain_dev/*.raw.json`;do
+    echo $dataset >> prediction_results/KD.log
+    python ../multi_task_learning/scripts/evaluate-v1.1.py $dataset $KD_prediction >> prediction_results/KD.log
+done
+
+for dataset in `ls $PATH_dev/out_of_domain_dev/*.raw.json`;do
+    echo $dataset >> prediction_results/KD.log
+    python ../multi_task_learning/scripts/evaluate-v1.1.py $dataset $KD_prediction >> prediction_results/KD.log
+done
+python ../multi_task_learning/scripts/macro_avg.py prediction_results/KD.log
+
+
+
+
+
+
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/utils/batching.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Mask, padding and batching."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+import numpy as np
+
+
+def mask(batch_tokens, total_token_num, vocab_size, CLS=1, SEP=2, MASK=3):
+    """
+    Add mask for batch_tokens, return out, mask_label, mask_pos;
+    Note: mask_pos responding the batch_tokens after padded;
+    """
+    max_len = max([len(sent) for sent in batch_tokens])
+    mask_label = []
+    mask_pos = []
+    prob_mask = np.random.rand(total_token_num)
+    # Note: the first token is [CLS], so [low=1]
+    replace_ids = np.random.randint(1, high=vocab_size, size=total_token_num)
+    pre_sent_len = 0
+    prob_index = 0
+    for sent_index, sent in enumerate(batch_tokens):
+        mask_flag = False
+        prob_index += pre_sent_len
+        for token_index, token in enumerate(sent):
+            prob = prob_mask[prob_index + token_index]
+            if prob > 0.15:
+                continue
+            elif 0.03 < prob <= 0.15:
+                # mask
+                if token != SEP and token != CLS:
+                    mask_label.append(sent[token_index])
+                    sent[token_index] = MASK
+                    mask_flag = True
+                    mask_pos.append(sent_index * max_len + token_index)
+            elif 0.015 < prob <= 0.03:
+                # random replace
+                if token != SEP and token != CLS:
+                    mask_label.append(sent[token_index])
+                    sent[token_index] = replace_ids[prob_index + token_index]
+                    mask_flag = True
+                    mask_pos.append(sent_index * max_len + token_index)
+            else:
+                # keep the original token
+                if token != SEP and token != CLS:
+                    mask_label.append(sent[token_index])
+                    mask_pos.append(sent_index * max_len + token_index)
+        pre_sent_len = len(sent)
+        # ensure at least mask one word in a sentence
+        while not mask_flag:
+            token_index = int(np.random.randint(1, high=len(sent) - 1, size=1))
+            if sent[token_index] != SEP and sent[token_index] != CLS:
+                mask_label.append(sent[token_index])
+                sent[token_index] = MASK
+                mask_flag = True
+                mask_pos.append(sent_index * max_len + token_index)
+    mask_label = np.array(mask_label).astype("int64").reshape([-1, 1])
+    mask_pos = np.array(mask_pos).astype("int64").reshape([-1, 1])
+    return batch_tokens, mask_label, mask_pos
+
+
+def prepare_batch_data(insts,
+                       total_token_num,
+                       max_len=None,
+                       voc_size=0,
+                       pad_id=None,
+                       cls_id=None,
+                       sep_id=None,
+                       mask_id=None,
+                       return_input_mask=True,
+                       return_max_len=True,
+                       return_num_token=False):
+    """
+    1. generate Tensor of data
+    2. generate Tensor of position
+    3. generate self attention mask, [shape: batch_size *  max_len * max_len]
+    """
+    batch_src_ids = [inst[0] for inst in insts]
+    batch_sent_ids = [inst[1] for inst in insts]
+    batch_pos_ids = [inst[2] for inst in insts]
+    labels_list = []
+    # compatible with mrqa, whose example includes start/end positions, 
+    # or unique id
+    for i in range(3, len(insts[0]), 1):
+        labels = [inst[i] for inst in insts]
+        labels = np.array(labels).astype("int64").reshape([-1, 1])
+        labels_list.append(labels)
+    # First step: do mask without padding
+    if mask_id >= 0:
+        out, mask_label, mask_pos = mask(
+            batch_src_ids,
+            total_token_num,
+            vocab_size=voc_size,
+            CLS=cls_id,
+            SEP=sep_id,
+            MASK=mask_id)
+    else:
+        out = batch_src_ids
+    # Second step: padding
+    src_id, self_input_mask = pad_batch_data(
+        out, 
+        max_len=max_len,
+        pad_idx=pad_id, return_input_mask=True)
+    pos_id = pad_batch_data(
+        batch_pos_ids,
+        max_len=max_len,
+        pad_idx=pad_id,
+        return_pos=False,
+        return_input_mask=False)
+    sent_id = pad_batch_data(
+        batch_sent_ids,
+        max_len=max_len,
+        pad_idx=pad_id,
+        return_pos=False,
+        return_input_mask=False)
+    if mask_id >= 0:
+        return_list = [
+            src_id, pos_id, sent_id, self_input_mask, mask_label, mask_pos
+        ] + labels_list
+    else:
+        return_list = [src_id, pos_id, sent_id, self_input_mask] + labels_list
+    return return_list if len(return_list) > 1 else return_list[0]
+
+
+def pad_batch_data(insts,
+                   max_len=None,
+                   pad_idx=0,
+                   return_pos=False,
+                   return_input_mask=False,
+                   return_max_len=False,
+                   return_num_token=False):
+    """
+    Pad the instances to the max sequence length in batch, and generate the
+    corresponding position data and input mask.
+    """
+    return_list = []
+    if max_len is None:
+        max_len = max(len(inst) for inst in insts)
+    # Any token included in dict can be used to pad, since the paddings' loss
+    # will be masked out by weights and make no effect on parameter gradients.
+    inst_data = np.array([
+        list(inst) + list([pad_idx] * (max_len - len(inst))) for inst in insts
+    ])
+    return_list += [inst_data.astype("int64").reshape([-1, max_len, 1])]
+    # position data
+    if return_pos:
+        inst_pos = np.array([
+            list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))
+            for inst in insts
+        ])
+        return_list += [inst_pos.astype("int64").reshape([-1, max_len, 1])]
+    if return_input_mask:
+        # This is used to avoid attention on paddings.
+        input_mask_data = np.array([[1] * len(inst) + [0] *
+                                    (max_len - len(inst)) for inst in insts])
+        input_mask_data = np.expand_dims(input_mask_data, axis=-1)
+        return_list += [input_mask_data.astype("float32")]
+    if return_max_len:
+        return_list += [max_len]
+    if return_num_token:
+        num_token = 0
+        for inst in insts:
+            num_token += len(inst)
+        return_list += [num_token]
+    return return_list if len(return_list) > 1 else return_list[0]
+
+
+if __name__ == "__main__":
+    pass
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/utils/configure.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import sys
+import argparse
+import six
+import logging
+import json
+
+logging_only_message = "%(message)s"
+logging_details = "%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s: %(message)s"
+
+
+class JsonConfig(object):
+    def __init__(self, config_path):
+        self._config_dict = self._parse(config_path)
+
+    def _parse(self, config_path):
+        try:
+            with open(config_path) as json_file:
+                config_dict = json.load(json_file)
+        except:
+            raise IOError("Error in parsing bert model config file '%s'" %
+                config_path)
+        else:
+            return config_dict
+
+    def __getitem__(self, key):
+        return self._config_dict[key]
+
+    def print_config(self):
+        for arg, value in sorted(six.iteritems(self._config_dict)):
+            print('%s: %s' % (arg, value))
+        print('------------------------------------------------')
+
+
+class ArgumentGroup(object):
+    def __init__(self, parser, title, des):
+        self._group = parser.add_argument_group(title=title, description=des)
+
+    def add_arg(self, name, type, default, help, **kwargs):
+        type = str2bool if type == bool else type
+        self._group.add_argument(
+            "--" + name,
+            default=default,
+            type=type,
+            help=help + ' Default: %(default)s.',
+            **kwargs)
+
+
+class ArgConfig(object):
+    
+    def __init__(self):
+        parser = argparse.ArgumentParser()
+
+        train_g = ArgumentGroup(parser, "training", "training options.")
+        train_g.add_arg("epoch",             int,    3,      "Number of epoches for fine-tuning.")
+        train_g.add_arg("learning_rate",     float,  5e-5,   "Learning rate used to train with warmup.")
+        train_g.add_arg("lr_scheduler",      str,    "linear_warmup_decay",
+                        "scheduler of learning rate.", choices=['linear_warmup_decay', 'noam_decay'])
+        train_g.add_arg("weight_decay",      float,  0.01,   "Weight decay rate for L2 regularizer.")
+        train_g.add_arg("warmup_proportion", float,  0.1,
+                        "Proportion of training steps to perform linear learning rate warmup for.")
+        train_g.add_arg("save_steps",        int,    1000,   "The steps interval to save checkpoints.")
+        train_g.add_arg("use_fp16",          bool,   False,  "Whether to use fp16 mixed precision training.")
+        train_g.add_arg("loss_scaling",      float,  1.0,
+                        "Loss scaling factor for mixed precision training, only valid when use_fp16 is enabled.")
+        train_g.add_arg("pred_dir",   str,    None,   "Path to save the prediction results")
+
+        log_g = ArgumentGroup(parser, "logging", "logging related.")
+        log_g.add_arg("skip_steps",          int,    10,    "The steps interval to print loss.")
+        log_g.add_arg("verbose",             bool,   False, "Whether to output verbose log.")
+
+        run_type_g = ArgumentGroup(parser, "run_type", "running type options.")
+        run_type_g.add_arg("use_cuda",                     bool,   True,  "If set, use GPU for training.")
+        run_type_g.add_arg("use_fast_executor",            bool,   False, "If set, use fast parallel executor (in experiment).")
+        run_type_g.add_arg("num_iteration_per_drop_scope", int,    1,     "Ihe iteration intervals to clean up temporary variables.")
+        run_type_g.add_arg("do_train",                     bool,   True,  "Whether to perform training.")
+        run_type_g.add_arg("do_predict",                   bool,   True,  "Whether to perform prediction.")
+
+        custom_g = ArgumentGroup(parser, "customize", "customized options.")
+
+        self.custom_g = custom_g
+
+        self.parser = parser
+
+    def add_arg(self, name, dtype, default, descrip):
+        self.custom_g.add_arg(name, dtype, default, descrip)
+
+    def build_conf(self):
+        return self.parser.parse_args()
+
+
+def str2bool(v):
+    # because argparse does not support to parse "true, False" as python
+    # boolean directly
+    return v.lower() in ("true", "t", "1")
+
+
+def print_arguments(args, log = None):
+    if not log:
+        print('-----------  Configuration Arguments -----------')
+        for arg, value in sorted(six.iteritems(vars(args))):
+            print('%s: %s' % (arg, value))
+        print('------------------------------------------------')
+    else:
+        log.info('-----------  Configuration Arguments -----------')
+        for arg, value in sorted(six.iteritems(vars(args))):
+            log.info('%s: %s' % (arg, value))
+        log.info('------------------------------------------------')
+
+
+if __name__ == "__main__":
+
+    args = ArgConfig()
+    args = args.build_conf()
+
+    # using print()
+    print_arguments(args)
+
+    logging.basicConfig(
+        level=logging.INFO,
+        format=logging_details,
+        datefmt='%Y-%m-%d %H:%M:%S')
+
+    # using logging
+    print_arguments(args, logging)
+
+    json_conf = JsonConfig("../../data/pretrained_models/uncased_L-12_H-768_A-12/bert_config.json")
+    json_conf.print_config()
+
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/utils/fp16.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import paddle
+import paddle.fluid as fluid
+
+
+def cast_fp16_to_fp32(i, o, prog):
+    prog.global_block().append_op(
+        type="cast",
+        inputs={"X": i},
+        outputs={"Out": o},
+        attrs={
+            "in_dtype": fluid.core.VarDesc.VarType.FP16,
+            "out_dtype": fluid.core.VarDesc.VarType.FP32
+        })
+
+
+def cast_fp32_to_fp16(i, o, prog):
+    prog.global_block().append_op(
+        type="cast",
+        inputs={"X": i},
+        outputs={"Out": o},
+        attrs={
+            "in_dtype": fluid.core.VarDesc.VarType.FP32,
+            "out_dtype": fluid.core.VarDesc.VarType.FP16
+        })
+
+
+def copy_to_master_param(p, block):
+    v = block.vars.get(p.name, None)
+    if v is None:
+        raise ValueError("no param name %s found!" % p.name)
+    new_p = fluid.framework.Parameter(
+        block=block,
+        shape=v.shape,
+        dtype=fluid.core.VarDesc.VarType.FP32,
+        type=v.type,
+        lod_level=v.lod_level,
+        stop_gradient=p.stop_gradient,
+        trainable=p.trainable,
+        optimize_attr=p.optimize_attr,
+        regularizer=p.regularizer,
+        gradient_clip_attr=p.gradient_clip_attr,
+        error_clip=p.error_clip,
+        name=v.name + ".master")
+    return new_p
+
+
+def create_master_params_grads(params_grads, main_prog, startup_prog,
+                               loss_scaling):
+    master_params_grads = []
+    tmp_role = main_prog._current_role
+    OpRole = fluid.core.op_proto_and_checker_maker.OpRole
+    main_prog._current_role = OpRole.Backward
+    for p, g in params_grads:
+        # create master parameters
+        master_param = copy_to_master_param(p, main_prog.global_block())
+        startup_master_param = startup_prog.global_block()._clone_variable(
+            master_param)
+        startup_p = startup_prog.global_block().var(p.name)
+        cast_fp16_to_fp32(startup_p, startup_master_param, startup_prog)
+        # cast fp16 gradients to fp32 before apply gradients
+        if g.name.find("layer_norm") > -1:
+            if loss_scaling > 1:
+                scaled_g = g / float(loss_scaling)
+            else:
+                scaled_g = g
+            master_params_grads.append([p, scaled_g])
+            continue
+        master_grad = fluid.layers.cast(g, "float32")
+        if loss_scaling > 1:
+            master_grad = master_grad / float(loss_scaling)
+        master_params_grads.append([master_param, master_grad])
+    main_prog._current_role = tmp_role
+    return master_params_grads
+
+
+def master_param_to_train_param(master_params_grads, params_grads, main_prog):
+    for idx, m_p_g in enumerate(master_params_grads):
+        train_p, _ = params_grads[idx]
+        if train_p.name.find("layer_norm") > -1:
+            continue
+        with main_prog._optimized_guard([m_p_g[0], m_p_g[1]]):
+            cast_fp32_to_fp16(m_p_g[0], train_p, main_prog)

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/utils/init.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import os
+import six
+import ast
+import copy
+
+import numpy as np
+import paddle.fluid as fluid
+
+
+def cast_fp32_to_fp16(exe, main_program):
+    print("Cast parameters to float16 data format.")
+    for param in main_program.global_block().all_parameters():
+        if not param.name.endswith(".master"):
+            param_t = fluid.global_scope().find_var(param.name).get_tensor()
+            data = np.array(param_t)
+            if param.name.find("layer_norm") == -1:
+                param_t.set(np.float16(data).view(np.uint16), exe.place)
+            master_param_var = fluid.global_scope().find_var(param.name +
+                                                             ".master")
+            if master_param_var is not None:
+                master_param_var.get_tensor().set(data, exe.place)
+
+
+def init_checkpoint(exe, init_checkpoint_path, main_program, use_fp16=False, skip_list = []):
+    assert os.path.exists(
+        init_checkpoint_path), "[%s] cann't be found." % init_checkpoint_path
+
+    def existed_persitables(var):
+        if not fluid.io.is_persistable(var):
+            return False
+        if var.name in skip_list:
+            return False
+        return os.path.exists(os.path.join(init_checkpoint_path, var.name))
+
+    fluid.io.load_vars(
+        exe,
+        init_checkpoint_path,
+        main_program=main_program,
+        predicate=existed_persitables)
+    print("Load model from {}".format(init_checkpoint_path))
+
+    if use_fp16:
+        cast_fp32_to_fp16(exe, main_program)
+
+
+def init_pretraining_params(exe,
+                            pretraining_params_path,
+                            main_program,
+                            use_fp16=False):
+    assert os.path.exists(pretraining_params_path
+                          ), "[%s] cann't be found." % pretraining_params_path
+
+    def existed_params(var):
+        if not isinstance(var, fluid.framework.Parameter):
+            return False
+        return os.path.exists(os.path.join(pretraining_params_path, var.name))
+
+    fluid.io.load_vars(
+        exe,
+        pretraining_params_path,
+        main_program=main_program,
+        predicate=existed_params)
+    print("Load pretraining parameters from {}.".format(
+        pretraining_params_path))
+
+    if use_fp16:
+        cast_fp32_to_fp16(exe, main_program)

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/utils/placeholder.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import six
+import ast
+import copy
+
+import numpy as np
+import paddle.fluid as fluid
+
+
+class Placeholder(object):
+
+    def __init__(self):
+        self.shapes = []
+        self.dtypes = []
+        self.lod_levels = []
+        self.names = []
+
+    def __init__(self, input_shapes):
+
+        self.shapes = []
+        self.dtypes = []
+        self.lod_levels = []
+        self.names = []
+
+        for new_holder in input_shapes:
+            shape = new_holder[0]
+            dtype = new_holder[1]
+            lod_level = new_holder[2] if len(new_holder) >= 3 else 0
+            name = new_holder[3] if len(new_holder) >= 4 else ""
+
+            self.append_placeholder(shape, dtype, lod_level = lod_level, name = name)
+
+    def append_placeholder(self, shape, dtype, lod_level = 0, name = ""):
+        self.shapes.append(shape)
+        self.dtypes.append(dtype)
+        self.lod_levels.append(lod_level)
+        self.names.append(name)
+
+    def build(self, capacity, reader_name, use_double_buffer = False):
+        pyreader = fluid.layers.py_reader(
+            capacity = capacity,
+            shapes = self.shapes,
+            dtypes = self.dtypes,
+            lod_levels = self.lod_levels,
+            name = reader_name, 
+            use_double_buffer = use_double_buffer)
+
+        return [pyreader, fluid.layers.read_file(pyreader)]
+
+    def __add__(self, new_holder):
+        assert isinstance(new_holder, tuple) or isinstance(new_holder, list) 
+        assert len(new_holder) >= 2
+
+        shape = new_holder[0]
+        dtype = new_holder[1]
+        lod_level = new_holder[2] if len(new_holder) >= 3 else 0
+        name = new_holder[3] if len(new_holder) >= 4 else ""
+
+        self.append_placeholder(shape, dtype, lod_level = lod_level, name = name)
+
+
+if __name__ == "__main__":
+    print("hello world!")
+
+
+

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/utils/tokenization.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import unicodedata
+import six
+
+
+def convert_to_unicode(text):
+    """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
+    if six.PY3:
+        if isinstance(text, str):
+            return text
+        elif isinstance(text, bytes):
+            return text.decode("utf-8", "ignore")
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    elif six.PY2:
+        if isinstance(text, str):
+            return text.decode("utf-8", "ignore")
+        elif isinstance(text, unicode):
+            return text
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    else:
+        raise ValueError("Not running on Python2 or Python 3?")
+
+
+def printable_text(text):
+    """Returns text encoded in a way suitable for print or `tf.logging`."""
+
+    # These functions want `str` for both Python2 and Python3, but in one case
+    # it's a Unicode string and in the other it's a byte string.
+    if six.PY3:
+        if isinstance(text, str):
+            return text
+        elif isinstance(text, bytes):
+            return text.decode("utf-8", "ignore")
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    elif six.PY2:
+        if isinstance(text, str):
+            return text
+        elif isinstance(text, unicode):
+            return text.encode("utf-8")
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    else:
+        raise ValueError("Not running on Python2 or Python 3?")
+
+
+def load_vocab(vocab_file):
+    """Loads a vocabulary file into a dictionary."""
+    vocab = collections.OrderedDict()
+    fin = open(vocab_file)
+    for num, line in enumerate(fin):
+        items = convert_to_unicode(line.strip()).split("\t")
+        if len(items) > 2:
+            break
+        token = items[0]
+        index = items[1] if len(items) == 2 else num
+        token = token.strip()
+        vocab[token] = int(index)
+    return vocab
+
+
+def convert_by_vocab(vocab, items):
+    """Converts a sequence of [tokens|ids] using the vocab."""
+    output = []
+    for item in items:
+        output.append(vocab[item])
+    return output
+
+
+def convert_tokens_to_ids(vocab, tokens):
+    return convert_by_vocab(vocab, tokens)
+
+
+def convert_ids_to_tokens(inv_vocab, ids):
+    return convert_by_vocab(inv_vocab, ids)
+
+
+def whitespace_tokenize(text):
+    """Runs basic whitespace cleaning and splitting on a peice of text."""
+    text = text.strip()
+    if not text:
+        return []
+    tokens = text.split()
+    return tokens
+
+
+class FullTokenizer(object):
+    """Runs end-to-end tokenziation."""
+
+    def __init__(self, vocab_file, do_lower_case=True):
+        self.vocab = load_vocab(vocab_file)
+        self.inv_vocab = {v: k for k, v in self.vocab.items()}
+        self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
+        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
+
+    def tokenize(self, text):
+        split_tokens = []
+        for token in self.basic_tokenizer.tokenize(text):
+            for sub_token in self.wordpiece_tokenizer.tokenize(token):
+                split_tokens.append(sub_token)
+
+        return split_tokens
+
+    def convert_tokens_to_ids(self, tokens):
+        return convert_by_vocab(self.vocab, tokens)
+
+    def convert_ids_to_tokens(self, ids):
+        return convert_by_vocab(self.inv_vocab, ids)
+
+
+class CharTokenizer(object):
+    """Runs end-to-end tokenziation."""
+
+    def __init__(self, vocab_file, do_lower_case=True):
+        self.vocab = load_vocab(vocab_file)
+        self.inv_vocab = {v: k for k, v in self.vocab.items()}
+        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
+
+    def tokenize(self, text):
+        split_tokens = []
+        for token in text.lower().split(" "):
+            for sub_token in self.wordpiece_tokenizer.tokenize(token):
+                split_tokens.append(sub_token)
+
+        return split_tokens
+
+    def convert_tokens_to_ids(self, tokens):
+        return convert_by_vocab(self.vocab, tokens)
+
+    def convert_ids_to_tokens(self, ids):
+        return convert_by_vocab(self.inv_vocab, ids)
+
+
+class BasicTokenizer(object):
+    """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
+
+    def __init__(self, do_lower_case=True):
+        """Constructs a BasicTokenizer.
+
+        Args:
+            do_lower_case: Whether to lower case the input.
+        """
+        self.do_lower_case = do_lower_case
+        self._never_lowercase = ['[UNK]', '[SEP]', '[PAD]', '[CLS]', '[MASK]']
+
+    def tokenize(self, text):
+        """Tokenizes a piece of text."""
+        text = convert_to_unicode(text)
+        text = self._clean_text(text)
+
+        # This was added on November 1st, 2018 for the multilingual and Chinese
+        # models. This is also applied to the English models now, but it doesn't
+        # matter since the English models were not trained on any Chinese data
+        # and generally don't have any Chinese data in them (there are Chinese
+        # characters in the vocabulary because Wikipedia does have some Chinese
+        # words in the English Wikipedia.).
+        text = self._tokenize_chinese_chars(text)
+
+        orig_tokens = whitespace_tokenize(text)
+        split_tokens = []
+        for token in orig_tokens:
+            if self.do_lower_case and token not in self._never_lowercase:
+                token = token.lower()
+                token = self._run_strip_accents(token)
+            if token in self._never_lowercase:
+                split_tokens.extend([token])
+            else:
+                split_tokens.extend(self._run_split_on_punc(token))
+
+        output_tokens = whitespace_tokenize(" ".join(split_tokens))
+        return output_tokens
+
+    def _run_strip_accents(self, text):
+        """Strips accents from a piece of text."""
+        text = unicodedata.normalize("NFD", text)
+        output = []
+        for char in text:
+            cat = unicodedata.category(char)
+            if cat == "Mn":
+                continue
+            output.append(char)
+        return "".join(output)
+
+    def _run_split_on_punc(self, text):
+        """Splits punctuation on a piece of text."""
+        chars = list(text)
+        i = 0
+        start_new_word = True
+        output = []
+        while i < len(chars):
+            char = chars[i]
+            if _is_punctuation(char):
+                output.append([char])
+                start_new_word = True
+            else:
+                if start_new_word:
+                    output.append([])
+                start_new_word = False
+                output[-1].append(char)
+            i += 1
+
+        return ["".join(x) for x in output]
+
+    def _tokenize_chinese_chars(self, text):
+        """Adds whitespace around any CJK character."""
+        output = []
+        for char in text:
+            cp = ord(char)
+            if self._is_chinese_char(cp):
+                output.append(" ")
+                output.append(char)
+                output.append(" ")
+            else:
+                output.append(char)
+        return "".join(output)
+
+    def _is_chinese_char(self, cp):
+        """Checks whether CP is the codepoint of a CJK character."""
+        # This defines a "chinese character" as anything in the CJK Unicode block:
+        #     https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
+        #
+        # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
+        # despite its name. The modern Korean Hangul alphabet is a different block,
+        # as is Japanese Hiragana and Katakana. Those alphabets are used to write
+        # space-separated words, so they are not treated specially and handled
+        # like the all of the other languages.
+        if ((cp >= 0x4E00 and cp <= 0x9FFF) or  #
+            (cp >= 0x3400 and cp <= 0x4DBF) or  #
+            (cp >= 0x20000 and cp <= 0x2A6DF) or  #
+            (cp >= 0x2A700 and cp <= 0x2B73F) or  #
+            (cp >= 0x2B740 and cp <= 0x2B81F) or  #
+            (cp >= 0x2B820 and cp <= 0x2CEAF) or
+            (cp >= 0xF900 and cp <= 0xFAFF) or  #
+            (cp >= 0x2F800 and cp <= 0x2FA1F)):  #
+            return True
+
+        return False
+
+    def _clean_text(self, text):
+        """Performs invalid character removal and whitespace cleanup on text."""
+        output = []
+        for char in text:
+            cp = ord(char)
+            if cp == 0 or cp == 0xfffd or _is_control(char):
+                continue
+            if _is_whitespace(char):
+                output.append(" ")
+            else:
+                output.append(char)
+        return "".join(output)
+
+
+class WordpieceTokenizer(object):
+    """Runs WordPiece tokenziation."""
+
+    def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=100):
+        self.vocab = vocab
+        self.unk_token = unk_token
+        self.max_input_chars_per_word = max_input_chars_per_word
+
+    def tokenize(self, text):
+        """Tokenizes a piece of text into its word pieces.
+
+        This uses a greedy longest-match-first algorithm to perform tokenization
+        using the given vocabulary.
+
+        For example:
+            input = "unaffable"
+            output = ["un", "##aff", "##able"]
+
+        Args:
+            text: A single token or whitespace separated tokens. This should have
+                already been passed through `BasicTokenizer.
+
+        Returns:
+            A list of wordpiece tokens.
+        """
+
+        text = convert_to_unicode(text)
+
+        output_tokens = []
+        for token in whitespace_tokenize(text):
+            chars = list(token)
+            if len(chars) > self.max_input_chars_per_word:
+                output_tokens.append(self.unk_token)
+                continue
+
+            is_bad = False
+            start = 0
+            sub_tokens = []
+            while start < len(chars):
+                end = len(chars)
+                cur_substr = None
+                while start < end:
+                    substr = "".join(chars[start:end])
+                    if start > 0:
+                        substr = "##" + substr
+                    if substr in self.vocab:
+                        cur_substr = substr
+                        break
+                    end -= 1
+                if cur_substr is None:
+                    is_bad = True
+                    break
+                sub_tokens.append(cur_substr)
+                start = end
+
+            if is_bad:
+                output_tokens.append(self.unk_token)
+            else:
+                output_tokens.extend(sub_tokens)
+        return output_tokens
+
+
+def _is_whitespace(char):
+    """Checks whether `chars` is a whitespace character."""
+    # \t, \n, and \r are technically contorl characters but we treat them
+    # as whitespace since they are generally considered as such.
+    if char == " " or char == "\t" or char == "\n" or char == "\r":
+        return True
+    cat = unicodedata.category(char)
+    if cat == "Zs":
+        return True
+    return False
+
+
+def _is_control(char):
+    """Checks whether `chars` is a control character."""
+    # These are technically control characters but we count them as whitespace
+    # characters.
+    if char == "\t" or char == "\n" or char == "\r":
+        return False
+    cat = unicodedata.category(char)
+    if cat.startswith("C"):
+        return True
+    return False
+
+
+def _is_punctuation(char):
+    """Checks whether `chars` is a punctuation character."""
+    cp = ord(char)
+    # We treat all non-letter/number ASCII as punctuation.
+    # Characters such as "^", "$", and "`" are not in the Unicode
+    # Punctuation class but we treat them as punctuation anyways, for
+    # consistency.
+    if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
+        (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
+        return True
+    cat = unicodedata.category(char)
+    if cat.startswith("P"):
+        return True
+    return False

PaddleNLP/Research/MRQA2019-D-NET/knowledge_distillation/wget_models_and_data.sh

+# wget pretrain model
+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/squad2_model.tar.gz
+tar -xvf squad2_model.tar.gz
+rm squad2_model.tar.gz
+mv squad2_model ./data/pretrain_model/
+
+# wget knowledge_distillation dataset
+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/d_net_knowledge_distillation_dataset.tar.gz
+tar -xvf d_net_knowledge_distillation_dataset.tar.gz
+rm d_net_knowledge_distillation_dataset.tar.gz
+mv mlm_data ./data/input
+mv mrqa_distill_data ./data/input
+
+# wget evaluation dev dataset
+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/mrqa_evaluation_dataset.tar.gz 
+tar -xvf mrqa_evaluation_dataset.tar.gz 
+rm mrqa_evaluation_dataset.tar.gz 
+mv mrqa_evaluation_dataset ./data/input
+
+# wget predictions results
+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/kd_prediction_results.tar.gz
+tar -xvf kd_prediction_results.tar.gz
+rm kd_prediction_results.tar.gz
+
+# wget MRQA baidu trained knowledge distillation model
+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/knowledge_distillation_model.tar.gz
+tar -xvf knowledge_distillation_model.tar.gz
+rm knowledge_distillation_model.tar.gz
+mv knowledge_distillation_model ./data/saved_models
+
+
+
+

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/README.md

+# Multi_task_learning 
+
+## 1、Introduction
+The pretraining is usually performed on corpus with restricted domains, it is expected that increasing the domain diversity by further pre-training on other corpus may improve the generalization capability. Hence, we incorporate masked language model and domain classify model by using corpus from various domains as an auxiliary tasks in the fine-tuning phase, along with MRC. Additionally, we explore multi-task learning by incorporating the supervised dataset from other NLP tasks to learn better language representation.
+
+## 2、Quick Start
+We use PaddlePaddle PALM(multi-task Learning Library) to train MRQA2019 MRC multi-task baseline model, download PALM:
+```
+git clone https://github.com/PaddlePaddle/PALM.git
+```
+
+### Environment
+- Python >= 2.7
+- cuda >= 9.0
+- cudnn >= 7.0
+- PaddlePaddle >= 1.5.0 Please refer to Installation Guide [Installation Guide](http://www.paddlepaddle.org/#quick-start)
+
+### Data Preparation
+#### Get data directly: 
+User can get the data directly we provided: 
+```
+bash wget_data.sh
+```
+
+#### Convert MRC dataset to squad format data: 
+To download the MRQA datasets, run
+```
+cd scripts && bash download_data.sh && cd ..
+```
+The training and prediction datasets will be saved in `./data/train/` and `./data/dev/`, respectively.
+
+The Multi_task_learning model only supports dataset files in SQuAD format. Before running the model on MRQA datasets, one need to convert the official MRQA data to SQuAD format. To do the conversion, run
+```
+cd scripts && bash convert_mrqa2squad.sh && cd ..
+```
+The output files will be named as `xxx.raw.json`.
+
+For convenience, we provide a script to combine all the training and development data into a single file respectively.
+```
+cd scripts && bash combine.sh && cd ..
+```
+The combined files will be saved in `./data/train/mrqa-combined.raw.json` and `./data/dev/mrqa-combined.raw.json`.
+
+### Models Preparation
+In this competition, We use google squad2.0 model as pretrain model [Model Link](https://worksheets.codalab.org/worksheets/0x3852e60a51d2444680606556d404c657)
+we provide script to convert tensorflow model to paddle model
+```
+cd scripts && python convert_model_params.py  --init_tf_checkpoint tf_model --fluid_params_dir paddle_model && cd ..
+```
+or user can get the pretrain model and multi-task learning trained models we provided: 
+```
+bash wget_models.sh
+```
+## 3、Train and Predict
+Preparing data, models, and task profiles for PALM
+```
+bash run_build_palm.sh
+```
+
+Start training: 
+```
+cd PALM
+bash run_multi_task.sh
+```
+
+## 4、Evaluation
+To evaluate the result, run
+```
+bash run_evaluation.sh
+```
+Note that we use the evaluation script for SQuAD 1.1 here, which is equivalent to the official one.
+
+## 5、Performance
+|  | dev in_domain(Macro-F1)| dev out_of_domain(Macro-F1) |
+| ------------- | ------------ | ------------ |
+| Official baseline | 77.87 | 58.67 |
+| BERT | 82.40 | 66.35 |
+| BERT + MLM | 83.19 | 67.45 |
+| BERT + MLM + ParaRank | 83.51 | 66.83 |
+
+BERT: reading comprehension single model.
+
+BERT + MLM: reading comprehension single model as main task, mask language model as auxiliary task.
+
+BERT + MLM + ParaRank: reading comprehension single model as main task, mask language model and paragraph classify rank as auxiliary tasks.
+
+BERT config: configs/reading_comprehension.yaml 
+
+MLM config: configs/mask_language_model.yaml
+
+ParaRank config: configs/answer_matching.yaml
+
+## Copyright and License
+Copyright 2019 Baidu.com, Inc. All Rights Reserved Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and
+limitations under the License.
+
+

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/configs/answer_matching.yaml

+train_file: "data/am4mrqa/train.txt"
+mix_ratio: 0.8
+batch_size: 4
+in_tokens: False
+generate_neg_sample: False

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/configs/mask_language_model.yaml

+train_file: "data/mlm4mrqa"
+mix_ratio: 2.0
+batch_size: 4
+in_tokens: False
+generate_neg_sample: False

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/configs/mtl_config.yaml

+main_task: "reading_comprehension"
+auxiliary_task: "mask_language_model answer_matching"
+
+do_train: True
+do_predict: True
+
+checkpoint_path: "output"
+
+backbone_model: "bert_model"
+pretrain_model_path: "pretrain_model/squad2_model"
+pretrain_config_path: "pretrain_model/squad2_model/bert_config.json"
+vocab_path: "pretrain_model/squad2_model/vocab.txt"
+
+optimizer: "bert_optimizer"
+learning_rate: 3e-5
+lr_scheduler: "linear_warmup_decay"
+skip_steps: 100
+save_steps: 10000
+epoch: 2
+use_cuda: True
+warmup_proportion: 0.1
+weight_decay: 0.1
+do_lower_case: False
+max_seq_len: 512
+use_ema: True
+ema_decay: 0.9999
+random_seed: 0
+use_fp16: False
+loss_scaling: 1.0
+

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/configs/reading_comprehension.yaml

+train_file: "data/mrqa/mrqa-combined.train.raw.json"
+predict_file: "data/mrqa/mrqa-combined.dev.raw.json"
+sample_rate: 0.02
+mix_ratio: 1.0
+batch_size: 4
+in_tokens: false
+doc_stride: 128
+with_negative: false
+max_query_length: 64
+max_answer_length: 30
+n_best_size: 20
+null_score_diff_threshold: 0.0
+verbose: False

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/run_build_palm.sh

+#!/bin/bash
+
+cp -r configs/* PALM/config/
+cp configs/mtl_config.yaml PALM/
+rm -rf PALM/data
+mv data PALM/
+mv squad2_model PALM/pretrain_model
+mv mrqa_multi_task_models PALM/
+cp run_multi_task.sh PALM/

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/run_evaluation.sh

+#!/usr/bin/env bash
+# ==============================================================================
+# Copyright 2017 Baidu.com, Inc. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+# path of dev data
+PATH_dev=./PALM/data/mrqa_dev
+# path of dev prediction
+BERT_MLM_PATH_prediction=./prediction_results/BERT_MLM_ema_predictions.json 
+BERT_MLM_ParaRank_PATH_prediction=./prediction_results/BERT_MLM_ParaRank_ema_predictions.json
+
+files=$(ls ./prediction_results/*.log 2> /dev/null | wc -l)
+if [ "$files" != "0" ];
+then
+    rm prediction_results/BERT_MLM*.log
+fi
+
+# evaluation BERT_MLM
+echo "evaluate BERT_MLM model........................................."
+for dataset in `ls $PATH_dev/in_domain_dev/*.raw.json`;do
+    echo $dataset >> prediction_results/BERT_MLM.log
+    python scripts/evaluate-v1.1.py $dataset $BERT_MLM_PATH_prediction >> prediction_results/BERT_MLM.log
+done
+
+for dataset in `ls $PATH_dev/out_of_domain_dev/*.raw.json`;do
+    echo $dataset >> prediction_results/BERT_MLM.log
+    python scripts/evaluate-v1.1.py $dataset $BERT_MLM_PATH_prediction >> prediction_results/BERT_MLM.log
+done
+python scripts/macro_avg.py prediction_results/BERT_MLM.log
+
+# evaluation BERT_MLM_ParaRank_PATH_prediction
+echo "evaluate BERT_MLM_ParaRank model................................"
+for dataset in `ls $PATH_dev/in_domain_dev/*.raw.json`;do
+    echo $dataset >> prediction_results/BERT_MLM_ParaRank.log
+    python scripts/evaluate-v1.1.py $dataset $BERT_MLM_ParaRank_PATH_prediction >> prediction_results/BERT_MLM_ParaRank.log
+done
+
+
+for dataset in `ls $PATH_dev/out_of_domain_dev/*.raw.json`;do
+    echo $dataset >> prediction_results/BERT_MLM_ParaRank.log
+    python scripts/evaluate-v1.1.py $dataset $BERT_MLM_ParaRank_PATH_prediction >> prediction_results/BERT_MLM_ParaRank.log
+done
+python scripts/macro_avg.py prediction_results/BERT_MLM_ParaRank.log
+
+
+
+
+
+
+

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/run_multi_task.sh

+#!/bin/bash
+
+# for gpu memory optimization
+export FLAGS_sync_nccl_allreduce=0
+export FLAGS_eager_delete_tensor_gb=1
+
+export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+
+python -u mtl_run.py
+

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/combine.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# ==============================================================================
+# Copyright 2017 Baidu.com, Inc. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""
+This module add all train/dev data to a file named "mrqa-combined.raw.json".
+"""
+
+import json
+import argparse
+import glob
+
+# path of train/dev data
+parser = argparse.ArgumentParser()
+parser.add_argument('path', help='the path of train/dev data')
+args = parser.parse_args()
+path = args.path
+
+# all train/dev data files
+files = glob.glob(path + '/*.raw.json')
+print ('files:', files)
+
+# add all train/dev data to "datasets"
+with open(files[0]) as fin:
+    datasets = json.load(fin)
+for i in range(1, len(files)):
+    with open(files[i]) as fin:
+        dataset = json.load(fin)
+    datasets['data'].extend(dataset['data'])
+
+# save to "mrqa-combined.raw.json"
+with open(path + '/mrqa-combined.raw.json', 'w') as fout:
+    json.dump(datasets, fout, indent=4)

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/combine.sh

+#!/usr/bin/env bash
+# ==============================================================================
+# Copyright 2017 Baidu.com, Inc. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+# path of train and dev data
+PATH_train=train
+PATH_dev=dev
+
+# add all train data to a file "$PATH_train/mrqa-combined.raw.json".
+python combine.py $PATH_train
+
+# add all dev data to a file "$PATH_dev/mrqa-combined.raw.json".
+python combine.py $PATH_dev
\ No newline at end of file

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/convert_model_params.py

+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Convert Google n-gram mask reading comprehension models to Fluid parameters."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+import argparse
+import collections
+from utils.args import print_arguments
+import tensorflow as tf
+import paddle.fluid as fluid
+from tensorflow.python import pywrap_tensorflow
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(__doc__)
+    parser.add_argument(
+        "--init_tf_checkpoint",
+        type=str,
+        required=True,
+        help="Initial TF checkpoint (a pre-trained BERT model).")
+
+    parser.add_argument(
+        "--fluid_params_dir",
+        type=str,
+        required=True,
+        help="The directory to store converted Fluid parameters.")
+    args = parser.parse_args()
+    return args
+
+
+def parse(init_checkpoint):
+    tf_fluid_param_name_map = collections.OrderedDict()
+    tf_param_name_shape_map = collections.OrderedDict()
+
+    init_vars = tf.train.list_variables(init_checkpoint)
+    for (var_name, var_shape) in init_vars: 
+        print("%s\t%s" % (var_name, var_shape))
+        fluid_param_name = ''
+        if var_name.startswith('bert/'):
+            key = var_name[5:]
+            if (key.startswith('embeddings/')):
+                if (key.endswith('LayerNorm/gamma')):
+                    fluid_param_name = 'pre_encoder_layer_norm_scale'
+                elif (key.endswith('LayerNorm/beta')):
+                    fluid_param_name = 'pre_encoder_layer_norm_bias'
+                elif (key.endswith('position_embeddings')):
+                    fluid_param_name = 'pos_embedding'
+                elif (key.endswith('word_embeddings')):
+                    fluid_param_name = 'word_embedding'
+                elif (key.endswith('token_type_embeddings')):
+                    fluid_param_name = 'sent_embedding'
+                else:
+                    print("ignored param: %s" % var_name)
+            elif (key.startswith('encoder/')):
+                key = key[8:]
+                layer_num = int(key[key.find('_') + 1:key.find('/')])
+                suffix = "encoder_layer_" + str(layer_num)
+                if key.endswith('attention/output/LayerNorm/beta'):
+                    fluid_param_name = suffix + '_post_att_layer_norm_bias'
+                elif key.endswith('attention/output/LayerNorm/gamma'):
+                    fluid_param_name = suffix + '_post_att_layer_norm_scale'
+                elif key.endswith('attention/output/dense/bias'):
+                    fluid_param_name = suffix + '_multi_head_att_output_fc.b_0'
+                elif key.endswith('attention/output/dense/kernel'):
+                    fluid_param_name = suffix + '_multi_head_att_output_fc.w_0'
+                elif key.endswith('attention/self/key/bias'):
+                    fluid_param_name = suffix + '_multi_head_att_key_fc.b_0'
+                elif key.endswith('attention/self/key/kernel'):
+                    fluid_param_name = suffix + '_multi_head_att_key_fc.w_0'
+                elif key.endswith('attention/self/query/bias'):
+                    fluid_param_name = suffix + '_multi_head_att_query_fc.b_0'
+                elif key.endswith('attention/self/query/kernel'):
+                    fluid_param_name = suffix + '_multi_head_att_query_fc.w_0'
+                elif key.endswith('attention/self/value/bias'):
+                    fluid_param_name = suffix + '_multi_head_att_value_fc.b_0'
+                elif key.endswith('attention/self/value/kernel'):
+                    fluid_param_name = suffix + '_multi_head_att_value_fc.w_0'
+                elif key.endswith('intermediate/dense/bias'):
+                    fluid_param_name = suffix + '_ffn_fc_0.b_0'
+                elif key.endswith('intermediate/dense/kernel'):
+                    fluid_param_name = suffix + '_ffn_fc_0.w_0'
+                elif key.endswith('output/LayerNorm/beta'):
+                    fluid_param_name = suffix + '_post_ffn_layer_norm_bias'
+                elif key.endswith('output/LayerNorm/gamma'):
+                    fluid_param_name = suffix + '_post_ffn_layer_norm_scale'
+                elif key.endswith('output/dense/bias'):
+                    fluid_param_name = suffix + '_ffn_fc_1.b_0'
+                elif key.endswith('output/dense/kernel'):
+                    fluid_param_name = suffix + '_ffn_fc_1.w_0'
+                else:
+                    print("ignored param: %s" % var_name)
+            elif (key.startswith('pooler/')):
+                if key.endswith('dense/bias'):
+                    fluid_param_name = 'pooled_fc.b_0'
+                elif key.endswith('dense/kernel'):
+                    fluid_param_name = 'pooled_fc.w_0'
+                else:
+                    print("ignored param: %s" % var_name)
+            else:
+                print("ignored param: %s" % var_name)
+
+        elif var_name.startswith('output/'):
+            if var_name == 'output/passage_regression/weights':
+                fluid_param_name = 'passage_regression_weights'
+            elif var_name == 'output/span/start/weights':
+                fluid_param_name = 'span_start_weights'
+            elif var_name == "output/span/end/conditional/dense/kernel": 
+                fluid_param_name = 'conditional_fc_weights'
+            elif var_name == "output/span/end/conditional/dense/bias": 
+                fluid_param_name = 'conditional_fc_bias'
+            elif var_name == "output/span/end/conditional/LayerNorm/beta": 
+                fluid_param_name = 'conditional_layernorm_beta'
+            elif var_name == "output/span/end/conditional/LayerNorm/gamma": 
+                fluid_param_name = 'conditional_layernorm_gamma'
+            elif var_name == "output/span/end/weights": 
+                fluid_param_name = 'span_end_weights'
+            else:
+                print("ignored param: %s" % var_name)
+        else:
+            print("ignored param: %s" % var_name)
+
+        if fluid_param_name != '':
+            tf_fluid_param_name_map[var_name] = fluid_param_name
+            tf_param_name_shape_map[var_name] = var_shape
+            fluid_param_name = ''
+
+    return tf_fluid_param_name_map, tf_param_name_shape_map
+
+
+def convert(args):
+    tf_fluid_param_name_map, tf_param_name_shape_map = parse(
+        args.init_tf_checkpoint)
+    program = fluid.Program()
+    global_block = program.global_block()
+    for param in tf_fluid_param_name_map:
+        global_block.create_parameter(
+            name=tf_fluid_param_name_map[param],
+            shape=tf_param_name_shape_map[param],
+            dtype='float32',
+            initializer=fluid.initializer.Constant(value=0.0))
+
+    place = fluid.core.CPUPlace()
+    exe = fluid.Executor(place)
+    exe.run(program)
+
+    print('---------------------- Converted Parameters -----------------------')
+    print('###### [TF param name] --> [Fluid param name]  [param shape] ######')
+    print('-------------------------------------------------------------------')
+
+    reader = pywrap_tensorflow.NewCheckpointReader(args.init_tf_checkpoint)
+    for param in tf_fluid_param_name_map:
+        value = reader.get_tensor(param)
+        fluid.global_scope().find_var(tf_fluid_param_name_map[
+            param]).get_tensor().set(value, place)
+        print(param, ' --> ', tf_fluid_param_name_map[param], '  ', value.shape)
+
+    fluid.io.save_params(exe, args.fluid_params_dir, main_program=program)
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    print_arguments(args)
+    convert(args)

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/convert_mrqa2squad.sh

+#!/usr/bin/env bash
+# ==============================================================================
+# Copyright 2017 Baidu.com, Inc. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+# path of train and dev data
+PATH_train=train
+PATH_dev=dev
+
+# Convert train data from MRQA format to SQuAD format
+NAME_LIST_train="SQuAD NewsQA TriviaQA SearchQA HotpotQA NaturalQuestions"
+for name in $NAME_LIST_train;do
+    echo "Converting training data from MRQA format to SQuAD format: ""$name"
+    python convert_mrqa2squad.py $PATH_train/$name.jsonl
+done
+
+# Convert dev data from MRQA format to SQuAD format
+NAME_LIST_dev="SQuAD NewsQA TriviaQA SearchQA HotpotQA NaturalQuestions BioASQ TextbookQA RelationExtraction DROP DuoRC RACE"
+for name in $NAME_LIST_dev;do
+    echo "Converting development data from MRQA format to SQuAD format: ""$name"
+    python convert_mrqa2squad.py --dev $PATH_dev/$name.jsonl
+done

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/dev/md5sum_dev.txt

+05f3f16c5c31ba8e46ff5fa80647ac46  SQuAD.jsonl.gz
+5c188c92a84ddffe2ab590ac7598bde2  NewsQA.jsonl.gz
+a7a3bd90db58524f666e757db659b047  TriviaQA.jsonl.gz
+bfcb304f1b3167693b627cbf0f98bc9e  SearchQA.jsonl.gz
+675de35c3605353ec039ca4d2854072d  HotpotQA.jsonl.gz
+c0347eebbca02d10d1b07b9a64efe61d  NaturalQuestions.jsonl.gz
+6408dc4fcf258535d0ea8b125bba5fbb  BioASQ.jsonl.gz
+76ca9cc16625dd8da75758d64676e6a1  TextbookQA.jsonl.gz
+128d318ea1391bf77234d8c1b69a45df  RelationExtraction.jsonl.gz
+8b03867e4da2817ef341707040d99785  DROP.jsonl.gz
+9e66769a70fdfdec4906a4bcef5f3d71  DuoRC.jsonl.gz
+94a7ef9b9ea9402671e5b0248b6a5395  RACE.jsonl.gz
\ No newline at end of file

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/download_data.sh

+#!/usr/bin/env bash
+# ==============================================================================
+# Copyright 2017 Baidu.com, Inc. All Rights Reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+
+# path to save data
+OUTPUT_train=train
+OUTPUT_dev=dev
+
+DATA_URL="https://s3.us-east-2.amazonaws.com/mrqa/release/v2"
+alias wget="wget -c --no-check-certificate"
+# download training datasets
+wget $DATA_URL/train/SQuAD.jsonl.gz -O $OUTPUT_train/SQuAD.jsonl.gz
+wget $DATA_URL/train/NewsQA.jsonl.gz -O $OUTPUT_train/NewsQA.jsonl.gz
+wget $DATA_URL/train/TriviaQA-web.jsonl.gz -O $OUTPUT_train/TriviaQA.jsonl.gz
+wget $DATA_URL/train/SearchQA.jsonl.gz -O $OUTPUT_train/SearchQA.jsonl.gz
+wget $DATA_URL/train/HotpotQA.jsonl.gz -O $OUTPUT_train/HotpotQA.jsonl.gz
+wget $DATA_URL/train/NaturalQuestionsShort.jsonl.gz -O $OUTPUT_train/NaturalQuestions.jsonl.gz
+
+# download the in-domain development data
+wget $DATA_URL/dev/SQuAD.jsonl.gz -O $OUTPUT_dev/SQuAD.jsonl.gz
+wget $DATA_URL/dev/NewsQA.jsonl.gz -O $OUTPUT_dev/NewsQA.jsonl.gz
+wget $DATA_URL/dev/TriviaQA-web.jsonl.gz -O $OUTPUT_dev/TriviaQA.jsonl.gz
+wget $DATA_URL/dev/SearchQA.jsonl.gz -O $OUTPUT_dev/SearchQA.jsonl.gz
+wget $DATA_URL/dev/HotpotQA.jsonl.gz -O $OUTPUT_dev/HotpotQA.jsonl.gz
+wget $DATA_URL/dev/NaturalQuestionsShort.jsonl.gz -O $OUTPUT_dev/NaturalQuestions.jsonl.gz
+
+# download the out-of-domain development data
+wget http://participants-area.bioasq.org/MRQA2019/ -O $OUTPUT_dev/BioASQ.jsonl.gz
+wget $DATA_URL/dev/TextbookQA.jsonl.gz -O $OUTPUT_dev/TextbookQA.jsonl.gz
+wget $DATA_URL/dev/RelationExtraction.jsonl.gz -O $OUTPUT_dev/RelationExtraction.jsonl.gz
+wget $DATA_URL/dev/DROP.jsonl.gz -O $OUTPUT_dev/DROP.jsonl.gz
+wget $DATA_URL/dev/DuoRC.ParaphraseRC.jsonl.gz -O $OUTPUT_dev/DuoRC.jsonl.gz
+wget $DATA_URL/dev/RACE.jsonl.gz -O $OUTPUT_dev/RACE.jsonl.gz
+
+# check md5sum for training datasets
+cd $OUTPUT_train
+if md5sum --status -c md5sum_train.txt; then
+    echo  "finish download training data"
+else
+    echo  "md5sum check failed!"
+fi
+cd ..
+
+# check md5sum for development data
+cd $OUTPUT_dev
+if md5sum --status -c md5sum_dev.txt; then
+    echo  "finish download development data"
+else
+    echo  "md5sum check failed!"
+fi
+cd ..
+
+# gzip training datasets
+echo "unzipping train data"
+NAME_LIST_train="SQuAD NewsQA TriviaQA SearchQA HotpotQA NaturalQuestions"
+for name in $NAME_LIST_train;do
+    gzip -d $OUTPUT_train/$name.jsonl.gz
+done
+
+# gzip development data
+echo "unzipping dev data"
+NAME_LIST_dev="SQuAD NewsQA TriviaQA SearchQA HotpotQA NaturalQuestions BioASQ TextbookQA RelationExtraction DROP DuoRC RACE"
+for name in $NAME_LIST_dev;do
+    gzip -d $OUTPUT_dev/$name.jsonl.gz
+done

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/scripts/train/md5sum_train.txt

+efd6a551d2697c20a694e933210489f8  SQuAD.jsonl.gz
+182f4e977b849cb1dbfb796030b91444  NewsQA.jsonl.gz
+e18f586152612a9358c22f5536bfd32a  TriviaQA.jsonl.gz
+612245315e6e7c4d8446e5fcc3dc1086  SearchQA.jsonl.gz
+d212c7b3fc949bd0dc47d124e8c34907  HotpotQA.jsonl.gz
+e27d27bf7c49eb5ead43cef3f41de6be  NaturalQuestions.jsonl.gz
\ No newline at end of file

PaddleNLP/Research/MRQA2019-D-NET/multi_task_learning/wget_models.sh

+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/squad2_model.tar.gz
+tar -xvf squad2_model.tar.gz
+rm squad2_model.tar.gz
+
+wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/D-Net/mrqa_multi_task_models.tar.gz
+tar -xvf mrqa_multi_task_models.tar.gz
+rm mrqa_multi_task_models.tar.gz