update r0.3-api

.gitignore

 *.pyc
 __pycache__
 pretrain_model
+pretrain
+output*
 output_model
 build
 dist

api_README.md

+# PaddlePALM
+
+PaddlePALM (Paddle for Multi-task) 是一个强大快速、灵活易用的NLP大规模多任务学习框架。通过PaddlePALM，用户可以轻松完成复杂的多任务学习与参数复用，无缝集成「**单任务训练**」、「**多任务辅助训练**」和「**多目标任务联合训练**」这 *3* 种训练方式和灵活的保存与预测机制，且仅需书写极少量代码即可”一键启动”高性能单机单卡和分布式训练与推理。
+
+框架中内置了丰富的[主干网络](#附录b内置主干网络backbone)及其[预训练模型](#预训练模型)（BERT、ERNIE等）、常见的[任务范式](#附录c内置任务范式paradigm)（分类、匹配、机器阅读理解等）和相应的[数据集读取与处理工具](#附录a内置数据集载入与处理工具reader)。同时框架提供了用户自定义接口，若内置工具、主干网络和任务无法满足需求，开发者可以轻松完成相关组件的自定义。各个组件均为零耦合设计，用户仅需完成组件本身的特性开发即可完成与框架的融合。
+
+## 目录
+
+- [安装](#安装)
+- [前期准备](#前期准备)
+    - [理论准备](#理论准备)
+    - [框架原理](#框架原理)
+    - [预训练模型](#预训练模型)
+- [X行代码实现文本分类](#三个demo入门paddlepalm)
+    - 
+- []
+    - [DEMO1：单任务训练](#demo1单任务训练)
+    - [DEMO2：多任务辅助训练与目标任务预测](#demo2多任务辅助训练与目标任务预测)
+    - [DEMO3：多目标任务联合训练与任务层参数复用](#demo3多目标任务联合训练与任务层参数复用)
+- [进阶篇](#进阶篇)
+    - [配置广播机制](#配置广播机制)
+    - [reader、backbone与paradigm的选择](#readerbackbone与paradigm的选择)
+    - [多目标任务下的训练终止条件与预期训练步数](#多目标任务下的训练终止条件与预期训练步数)
+        - [多个目标任务](#多个目标任务)
+        - [训练终止条件](#训练终止条件)
+        - [任务采样概率与预期训练步数](#任务采样概率与预期训练步数)
+        - [多个目标任务时预期训练步数的计算](#多个目标任务时预期训练步数的计算)
+    - [模型保存与预测机制](#模型保存与预测机制)
+    - [分布式训练](#分布式训练)
+- [附录A：内置数据集载入与处理工具（reader）](#附录a内置数据集载入与处理工具reader)
+- [附录B：内置主干网络（backbone）](#附录b内置主干网络backbone)
+- [附录C：内置任务范式（paradigm）](#附录c内置任务范式paradigm)
+- [附录D：可配置的全局参数列表](#附录d可配置的全局参数列表)
+
+
+## Package Overview
+
+| **paddlepalm** | an open source NLP pretraining and multitask learning framework, built on paddlepaddle. |
+| **paddlepalm.reader** | a collection of elastic task-specific dataset readers. |
+| **paddlepalm.backbone** | a collection of classic NLP representation models, e.g., BERT. |
+| **paddlepalm.head** | a collection of task-specific output layers. |
+| **paddlepalm.lr_sched** | a collection of learning rate schedualers. |
+| **paddlepalm.optimizer** | a collection of optimizers. |
+| **paddlepalm.downloader** | a download module for pretrained models with configure and vocab files. |
+| **paddlepalm.Trainer** | the core unit to start a single task training/predicting session. A trainer is to build computation graph, manage training and evaluation process, achieve model/checkpoint saving and pretrain_model/checkpoint loading.|
+| **paddlepalm.MultiHeadTrainer** | the core unit to start a multi-task training/predicting session. A MultiHeadTrainer is built based on several Trainers. Beyond the inheritance of Trainer, it additionally achieves model backbone reuse across tasks, trainer sampling for multi-task learning, and multi-head inference for effective evaluation and prediction. |
+
+
+## Installation
+
+PaddlePALM support both python2 and python3, linux and windows, CPU and GPU. The preferred way to install PaddlePALM is via `pip`. Just run following commands in your shell environment.
+
+```bash
+pip install paddlepalm
+```
+
+We incorporate many pretrained models to initialize model backbone parameters. Training big NLP model, e.g., 12-layer transformers, with pretrained models is practically much more superior than that with randomly initialized parameters. To see all the available pretrained models and download, run following code in python interpreter (input command `python` in shell):
+
+```python
+>>> from paddlepalm import downloader
+>>> downloader.ls('pretrain')
+Available pretrain items:
+  => roberta-cn-base
+  => roberta-cn-large
+  => bert-cn-base
+  => bert-cn-large
+  => bert-en-uncased-base
+  => bert-en-uncased-large
+  => bert-en-cased-base
+  => bert-en-cased-large
+  => ernie-en-uncased-base
+  => ernie-en-uncased-large
+  ...
+
+>>> downloader.download('pretrain', 'bert-en-uncased-base', './pretrain_models')
+...
+```
+
+Then 
+
+## Usage
+
+8 steps to start a typical NLP training task.
+
+1. use `paddlepalm.reader` to create a *reader* for dataset loading and input features generation, then call `reader.load_data` method to load your training data.
+2. use `paddlepalm.backbone` to create a model *backbone* to extract text features (e.g., contextual word embedding, sentence embedding).
+3. register your *reader* with your *backbone* through `reader.register_with` method. After this step, your reader is able to yield input features used by backbone.
+4. use `paddlepalm.head` to create a task output *head*. This head can provide task loss for training and predicting results for model inference.
+5. create a task *trainer* with `paddlepalm.Trainer`, then build forward graph with backbone and task head (created in step 2 and 4) through `trainer.build_forward`.
+6. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `trainer.build_backward`.
+7. fit prepared reader and data (achieved in step 1) to trainer with `trainer.fit_reader` method.
+8. randomly initialize model parameters (and `trainer.load_pretrain` if needed), then do training with `trainer.train`.
+
+More implementation details see following demos: [Sentiment Classification](), [Quora Question Pairs matching](), [Tagging](), [SQuAD machine Reading Comprehension]().
+
+To save models/checkpoints during training, just call `trainer.set_saver` method. More implementation details see [this]().
+
+To do predict/evaluation after a training stage, just create another three reader, backbone and head instance with `phase='predict'` (repeat step 1~4 above). Then do predicting with `predict` method in trainer (no need to create another trainer). More implementation details see [this]().
+
+To run with multi-task learning mode:
+
+1. repeatedly create components (i.e., reader, backbone and head) for each task followed with step 1~5 above. 
+2. create empty trainers (each trainer is corresponded to one task) and pass them to create a `MultiHeadTrainer`. 
+3. build multi-task forward graph with `multi_head_trainer.build_forward` method.
+4. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `multi_head_trainer.build_backward`.
+5. fit all prepared readers and data to multi_head_trainer with `multi_head_trainer.fit_readers` method.
+6. randomly initialize model parameters with `multi_head_trainer.random_init_params` (and `multi_head_trainer.load_pretrain` if needed), then do training with `multi_head_trainer.train`.
+
+The save/load and predict operations of a multi_head_trainer is the same as a trainer.
+
+### reader
+
+### backbone
+
+### 
+
+
+
+
+推荐使用pip安装paddlepalm框架：
+
+```shell
+pip install paddlepalm
+```
+
+对于离线机器，可以使用基于源码的安装方式：
+
+```shell
+git clone https://github.com/PaddlePaddle/PALM.git
+cd PALM && python setup.py install
+```
+
+
+
+**环境依赖**
+
+- Python >= 2.7 (即将支持python3)
+- cuda >= 9.0
+- cudnn >= 7.0
+- PaddlePaddle >= 1.6.1 (请参考[安装指南](http://www.paddlepaddle.org/#quick-start)进行安装)
+
+## 框架代码结构
+
+```text
+.
+├── mtl_controller.py                 # 任务控制器，负责创建和调度各个任务实例来完成多任务学习
+├── task_instance.py                  # 任务实例类，完成任务实例的配置管理、训练进程管理、保存与载入等
+├── default_settings.py               # 默认的环境变量和框架配置
+├── utils                             # 框架核心工具集
+│   ├── config_helper.py                  # 配置工具类，完成命令行与json、yaml的联合解析
+│   ├── reader_helper.py                  # 完成多任务数据集iterators的合并、采样、调度和归一化，连接python生成器与计算图
+│   ├── saver.py                          # 模型保存与载入
+│   ├── print_helper.py                   # 日志打印规范化工具 
+│   ├── plot_helper.py                    # 命令行绘图工具   
+│   └── textprocess_helper.py             # 文本数据处理工具函数
+├── backbone                          # 框架预置的主干网络
+│   ├── ernie.py                          # ERNIE模型
+│   ├── bert.py                           # BERT模型
+│   └── utils                             # 实现主干网络的一些可复用的工具函数
+├── reader                            # 框架内置的数据集载入与处理工具
+│   ├── cls.py                            # 文本分类数据集工具
+│   ├── match.py                          # 文本匹配数据集工具
+│   ├── mrc.py                            # 机器阅读理解数据集工具
+│   └── mlm.py                            # 掩码语言模型（mask language model）数据集生成与处理工具
+└── paradigm                          # 任务范式
+    ├── cls.py                            # 文本分类
+    ├── match.py                          # 文本匹配
+    ├── mrc.py                            # 机器阅读理解
+    └── mlm.py                            # 掩码语言模型（mask language model）
+```
+
+
+## 前期准备
+
+### 理论准备
+
+框架默认采用一对多（One-to-Many）的参数共享方式，如图
+
+
+![image-20191022194400259](https://tva1.sinaimg.cn/large/006y8mN6ly1g88ajvpqmgj31hu07wn5s.jpg)
+
+
+例如我们有一个目标任务MRC和两个辅助任务MLM和MATCH，我们希望通过MLM和MATCH来提高目标任务MRC的测试集表现（即提升模型泛化能力），那么我们可以令三个任务共享相同的文本特征抽取模型（例如BERT、ERNIE等），然后分别为每个任务定义输出层，计算各自的loss值。
+
+框架默认采用任务采样+mini-batch采样的方式（alternating mini-batches optimization）进行模型训练，即对于每个训练step，首先对候选任务进行采样（采样权重取决于用户设置的`mix_ratio`），而后从该任务的训练集中采样出一个mini-batch（采样出的样本数取决于用户设置的`batch_size`）。
+
+### 框架原理
+paddlepalm框架的运行原理图如图所示
+
+![PALM原理图](https://tva1.sinaimg.cn/large/006y8mN6ly1g8j1isf3fcj31ne0tyqbd.jpg)
+
+首先用户为数据集载入与处理、主干网络和任务编写配置文件（框架实现了灵活易用的[配置广播机制](#配置广播机制)），而后用其创建多任务学习的控制器（`Controller`）。进而控制器创建任务实例，并根据用户调用的训练和预测接口对其参数和各个任务实例进行管理和调度。下面我们通过三个DEMO和进阶篇来快速入门并更深入的理解paddlepalm。
+
+### 预训练模型
+
+#### 下载
+我们提供了BERT、ERNIE等主干网络的相关预训练模型。为了加速模型收敛，获得更佳的测试集表现，我们强烈建议用户在多任务学习时尽量在预训练模型的基础上进行（而不是从参数随机初始化开始）。用户可通过运行`script/download_pretrain_models <model_name>`下载需要的预训练模型，例如，下载预训练BERT模型（uncased large）的命令如下
+
+```shell
+bash script/download_pretrain_backbone.sh bert
+```
+
+脚本会自动在**当前文件夹**中创建一个pretrain_model目录（注：运行DEMO时，需保证pretrain_model文件夹在PALM项目目录下），并在其中创建bert子目录，里面存放预训练模型(`params`文件夹内)、相关的网络参数(`bert_config.json`)和字典(`vocab.txt`)。除了BERT模型，脚本还提供了ERNIE预训练模型（uncased large）的一键下载，将`<model_name>`改成`ernie`即可。全部可用的预训练模型列表见[paddlenlp/lark](https://github.com/PaddlePaddle/models/tree/develop/PaddleNLP/PaddleLARK)
+
+
+#### 转换
+注意，预训练模型不能直接被框架使用。我们提供了转换脚本可以将其转换成paddlepalm的模型格式。如下，通过运行`script/convert_params.sh`可将预训练模型bert转换成框架的模型格式。
+
+```shell
+bash script/convert_params.sh pretrain_model/bert/params
+```
+
+注意，以下恢复操作在执行后述DEMO流程中**无需执行**。
+若用户需将转换成的paddlepalm模型恢复为原始的预训练模型，可以运行`script/recover_params.sh`进行恢复。
+
+```shell
+bash script/recover_params.sh pretrain_model/bert/params
+```
+
+
+## 三个DEMO入门PaddlePALM
+
+### DEMO1：单任务训练
+
+框架支持对任何一个内置任务进行传统的单任务训练。接下来我们启动一个复杂的机器阅读理解任务的训练，我们在`data/mrqa`文件夹中提供了[EMNLP2019 MRQA机器阅读理解评测](https://mrqa.github.io/shared)的部分比赛数据。下面我们利用该数据尝试完成一个基于BERT的机器阅读理解任务MRQA的单任务学习。
+
+用户可通过运行如下脚本一键开始本节任务的训练
+
+```shell
+bash run_demo1.sh
+```
+
+下面以该任务为例，讲解如何基于paddlepalm框架轻松实现该任务。
+
+**1. 配置任务实例**
+
+首先，我们编写该任务实例的配置文件`mrqa.yaml`，若该任务实例参与训练或预测，则框架将自动解析该配置文件并创建相应的任务实例。配置文件需符合yaml格式的要求。一个任务实例的配置文件最少应包含`train_file`，`reader`和`paradigm`这三个字段，分别代表训练集的文件路径`train_file`、使用的数据集载入与处理工具`reader`、任务范式`paradigm`。
+
+```yaml
+train_file: data/mrqa/train.json
+reader: mrc
+paradigm: mrc
+```
+
+*注：框架内置的其他数据集载入与处理工具见[这里](#附录a内置数据集载入与处理工具reader)，任务范式列表见[这里](#附录c内置任务范式paradigm)*
+
+此外，我们还需要配置reader的预处理规则，各个预置reader支持的预处理配置和规则请参考[这里](#附录a内置数据集载入与处理工具reader)。预处理规则同样直接写入`mrqa.yaml`中。
+
+```yaml
+max_seq_len: 512
+max_query_len: 64
+doc_stride: 128 # 在MRQA数据集中，存在较长的文档，因此我们这里使用滑动窗口处理样本，滑动步长设置为128
+do_lower_case: True
+vocab_path: "pretrain_model/bert/vocab.txt"
+```
+
+更详细的任务实例配置方法（为任务实例选择合适的reader、paradigm和backbone）可参考[这里](#readerbackbone与paradigm的选择)
+
+**2.配置backbone和训练规则**
+
+然后我们编写全局配置文件`config_demo1.yaml`。在这里可以完成对主干网络(backbone)、多任务学习规则以及[广播到任务实例](#配置广播机制)的配置。同样使用yaml格式描述，例如在这里我们可以配置一下需要学习的任务`task_instance`、模型的保存路径`save_path`、基于的主干网络`backbone`、优化器`optimizer`等。
+
+```yaml
+task_instance: "mrqa"
+
+save_path: "output_model/firstrun"
+
+backbone: "bert"
+backbone_config_path: "pretrain_model/bert/bert_config.json"
+
+optimizer: "adam"
+learning_rate: 3e-5
+batch_size: 4
+
+num_epochs: 2                                                                                    
+warmup_proportion: 0.1 
+```
+
+这里的task_instance即填写我们刚刚编写的任务实例配置文件的文件名`mrqa`**（注意不要包括.yaml后缀！）**。框架启动多任务学习后会根据`task_instance`中指定的任务实例来寻找相关配置文件，并创建任务实例。
+
+此外，backbone的相关配置除了可以直接写入全局配置文件以外，还可以在额外的一个json文件中进行描述，并在全局配置文件中通过`backbone_config_path`进行该配置文件路径的指定。
+
+*注：框架支持的其他内置全局参数见[这里](#附录d可配置的全局参数列表)*
+
+**3.开始训练**
+
+下面我们开始尝试启动MRQA任务的训练（该代码位于`demo1.py`中）。如[框架原理](#框架原理)所述，框架的核心组件是`Controller`，负责多任务学习的启动。
+
+```python
+# Demo 1: single task training of MRQA 
+import paddlepalm as palm
+
+if __name__ == '__main__':
+    controller = palm.Controller('config_demo1.yaml', task_dir='demo1_tasks')
+    controller.load_pretrain('pretrain_model/bert/params')
+    controller.train()
+```
+
+训练日志如下，可以看到loss值随着训练收敛。在训练结束后，`Controller`自动为mrqa任务保存预测模型。
+
+```
+Global step: 10. Task: mrqa, step 10/135 (epoch 0), loss: 5.928, speed: 0.67 steps/s
+Global step: 20. Task: mrqa, step 20/135 (epoch 0), loss: 4.594, speed: 0.75 steps/s
+Global step: 30. Task: mrqa, step 30/135 (epoch 0), loss: 1.663, speed: 0.75 steps/s
+...
+Global step: 250. Task: mrqa, step 115/135 (epoch 1), loss: 1.391, speed: 0.75 steps/s
+Global step: 260. Task: mrqa, step 125/135 (epoch 1), loss: 1.871, speed: 0.75 steps/s
+Global step: 270. Task: mrqa, step 135/135 (epoch 1), loss: 1.544, speed: 0.75 steps/s
+mrqa: train finished!
+mrqa: inference model saved at output_model/firstrun/mrqa/infer_model
+```
+
+### DEMO2：多任务辅助训练与目标任务预测
+
+本节我们考虑更加复杂的学习目标，我们引入一个掩码语言模型（Mask Language Model，MLM）问答匹配（QA Match）任务来辅助上一节MRQA任务的训练，相关训练数据分别位于`data/mlm4mrqa`和`data/match4mrqa`。并且我们这里换用ERNIE模型作为主干网络，来获得更佳的效果。在多任务训练结束后，我们使用训练好的模型来对MRQA任务的测试集进行预测。
+
+用户可通过运行如下脚本直接开始本节任务的训练
+
+```shell
+bash run_demo2.sh
+```
+
+下面以该任务为例，讲解如何基于paddlepalm框架轻松实现这个复杂的多任务学习。
+
+**1. 配置任务实例**
+
+首先，我们像上一节一样为MLM任务和Matching任务分别创建任务实例的配置文件`mlm4mrqa.yaml`和`match4mrqa.yaml`：
+
+```yaml
+----- mlm4mrqa.yaml -----
+train_file: "data/mlm4mrqa/train.tsv"
+reader: mlm
+paradigm: mlm
+
+----- match4mrqa.yaml -----
+train_file: "data/match/train.tsv"
+reader: match
+paradigm: match
+```
+
+由于我们在训练结束后要对MRQA任务的测试集进行预测，因此我们要在之前写好的`mrqa.yaml`中追加预测相关的配置
+```yaml
+pred_file: data/mrqa/dev.json
+pred_output_path: 'mrqa_output'
+max_answer_len: 30
+n_best_size: 20
+```
+
+**2.配置全局参数**
+
+由于MRQA、MLM和Matching任务有相同的字典、大小写配置、截断长度等，因此我们可以将这些各个任务中相同的参数写入到全局配置文件`mtl_config.yaml`中，**框架会自动将该文件中的配置广播（broadcast）到各个任务实例。**
+
+```yaml
+task_instance: "mrqa, mlm4mrqa, match4mrqa"
+target_tag: 1,0,0 
+
+save_path: "output_model/secondrun"
+
+backbone: "ernie"
+backbone_config_path: "pretrain_model/ernie/ernie_config.json"
+
+vocab_path: "pretrain_model/ernie/vocab.txt"
+do_lower_case: True
+max_seq_len: 512 # 写入全局配置文件的参数会被自动广播到各个任务实例
+
+batch_size: 4
+num_epochs: 2
+optimizer: "adam"
+learning_rate: 3e-5
+warmup_proportion: 0.1 
+weight_decay: 0.1 
+```
+
+这里我们可以使用`target_tag`来标记目标任务和辅助任务，各个任务的tag使用逗号`,`隔开。target_tag与task_instance中的元素一一对应，当某任务的tag设置为1时，表示对应的任务被设置为目标任务；设置为0时，表示对应的任务被设置为辅助任务，默认情况下所以任务均被设置为目标任务（即默认`target_tag`为全1）。
+
+辅助任务不会保存预测模型，且不会影响训练的终止，仅仅起到“陪同训练”的作用以期提高模型的泛化能力。当所有的目标任务达到预期的训练步数后多任务学习终止，框架自动为每个目标任务保存预测模型（inference model）到设置的`save_path`位置。
+
+同时需要注意的是，这里`num_epochs`指代目标任务`mrqa`的训练epoch数量（训练集遍历次数）。
+
+在训练过程中，默认每个训练step会从各个任务等概率采样，来决定当前step训练哪个任务。但包括辅助任务在内，各个任务的采样概率是可以被控制的。若用户希望改变采样比率，可以通过`mix_ratio`字段来进行设置，例如
+
+```yaml
+mix_ratio: 1.0, 0.5, 0.5
+```
+
+若将如上设置加入到全局配置文件中，则辅助任务`mlm4mrqa`和`match4mrqa`的采样概率/预估的训练步数仅为`mrqa`任务的一半。关于采样概率的更多介绍请参考进阶篇。
+
+
+
+**3.开始多任务训练**
+
+```python
+import paddlepalm as palm
+
+if __name__ == '__main__':
+    controller = palm.Controller('config_demo2.yaml', task_dir='demo2_tasks')
+    controller.load_pretrain('pretrain_model/ernie/params')
+    controller.train()
+
+```
+
+训练日志如下，在训练过程中可以看到每个任务的loss下降
+```
+Global step: 10. Task: mrqa, step 4/135 (epoch 0), loss: 6.235, speed: 0.75 steps/s
+Global step: 20. Task: mrqa, step 8/135 (epoch 0), loss: 5.652, speed: 0.75 steps/s
+Global step: 30. Task: mrqa, step 13/135 (epoch 0), loss: 6.031, speed: 0.75 steps/s
+Global step: 40. Task: match4mrqa, step 13/25 (epoch 0), loss: 0.758, speed: 2.52 steps/s
+Global step: 50. Task: mlm4mrqa, step 14/30 (epoch 0), loss: 7.322, speed: 3.24 steps/s
+...
+Global step: 547. Task: match4mrqa, step 13/25 (epoch 5), loss: 0.400, speed: 2.23 steps/s
+Global step: 548. Task: match4mrqa, step 14/25 (epoch 5), loss: 0.121, speed: 3.03 steps/s
+Global step: 549. Task: mrqa, step 134/135 (epoch 1), loss: 0.824, speed: 0.75 steps/s
+Global step: 550. Task: mlm4mrqa, step 22/30 (epoch 4), loss: 6.903, speed: 3.59 steps/s
+Global step: 551. Task: mrqa, step 135/135 (epoch 1), loss: 3.408, speed: 0.75 steps/s
+
+mrqa: train finished!
+mrqa: inference model saved at output_model/secondrun/mrqa/infer_model
+```
+
+**4.预测**
+
+在得到目标任务的预测模型（inference_model）后，我们可以加载预测模型对该任务的测试集进行预测。在多任务训练阶段，在全局配置文件的`save_path`指定的路径下会为每个目标任务创建同名子目录，子目录中都有预测模型文件夹`infermodel`。我们可以将该路径传给框架的`controller`来完成对该目标任务的预测。
+
+例如，我们在上一节得到了mrqa任务的预测模型。首先创建一个新的*Controller*，**并且创建时要将`for_train`标志位置为*False***。而后调用*pred*接口，将要预测的任务实例名字和预测模型的路径传入，即可完成相关预测。预测的结果默认保存在任务实例配置文件的`pred_output_path`指定的路径中。代码段如下：
+
+```python
+    controller = palm.Controller(config='config_demo2.yaml', task_dir='demo2_tasks', for_train=False)
+    controller.pred('mrqa', inference_model_dir='output_model/secondrun/mrqa/infermodel') 
+```
+
+我们可以在刚刚yaml文件中设置的`mrqa_output/`文件夹下的`predictions.json`文件中看到类似如下的预测结果
+
+```json
+{                                                                                           
+    "3f02f171c82e49828580007a71eefc31": "Ethan Allen", 
+    "98d0b8ce19d1434abdb42aa01e83db61": "McDonald's", 
+    "f0bc45a4dd7a4d8abf91a5e4fb25fe57": "Jesse James", 
+    ...
+}
+```
+
+其中的每一行是测试集中的一个question对应的预测答案（其中的key为question的id，详情见mrc reader的说明文档）。
+
+### DEMO3：多目标任务联合训练与任务层参数复用
+
+本节我们考虑一个更加复杂的大规模多任务学习场景。假如手头有若干任务，其中每个任务都可能将来被用于预测（即均为目标任务），且鉴于这若干个任务之间存在一些相关性，我们希望将其中一部分任务的任务层参数也进行复用。分类数据集位于`data/cls4mrqa`内。
+
+具体来说，例如我们有6个分类任务（CLS1 ~ CLS6），均为目标任务（每个任务的模型都希望未来拿来做预测和部署），且我们希望任务1，2，5的任务输出层共享同一份参数，任务3、4共享同一份参数，任务6自己一份参数，即希望对6个任务实现如图所示的参数复用关系。
+
+![image2](https://tva1.sinaimg.cn/large/006y8mN6ly1g8issdoli5j31ow08ogxv.jpg)
+
+如图，在同一个方框内的任务共享相同的任务层参数。
+
+用户可通过运行如下脚本一键开始学习本节任务目标：
+
+```shell
+bash run_demo3.sh
+```
+
+**1. 配置任务实例**
+
+为了演示方便，我们使用同一份数据集来创建6个分类的任务实例，分别命名为`cls1.yaml`, `cls2.yaml`, `cls3.yaml`, `cls4.yaml`, `cls5.yaml`, `cls6.yaml`。每个实例的配置文件中填入如下必要字段
+
+```yaml
+train_file: "data/cls4mrqa/train.tsv"
+reader: cls
+paradigm: cls
+
+n_classes: 4
+```
+
+**2.配置全局参数**
+
+在paddlepalm中可以轻松完成上述的复杂复用关系的定义，我们使用`task_reuse_tag`来描述任务层的参数复用关系，与`target_tag`一样，`task_reuse_tag`中的元素与`task_instance`一一对应，元素取值相同的任务会自动共享任务层参数，取值不同的任务不复用任务层参数。因此可以在全局配置文件中如下描述
+
+```yaml
+task_instance: "cls1, cls2, cls3, cls4, cls5, cls6"
+task_reuse_tag: 0, 0, 1, 1, 0, 2
+```
+
+同时，这6个任务均为目标任务，因此我们不需要手动设置`target_tag`了（任务默认即为目标任务）。不过，**设置多个目标的情况下，依然可以添加辅助任务陪同这些目标任务进行训练**，这时候就需要引入`target_tag`来区分目标任务和辅助任务了。而后，我们在全局配置文件中写入其他必要的参数（backbone、优化器等）。
+
+```yaml
+save_path: "output_model/secondrun"
+
+backbone: "ernie"
+backbone_config_path: "pretrain_model/ernie/ernie_config.json"
+
+vocab_path: "pretrain_model/ernie/vocab.txt"
+do_lower_case: True
+max_seq_len: 512 # 写入全局配置文件的参数会被自动广播到各个任务实例
+
+batch_size: 4
+num_epochs: 2
+optimizer: "adam"
+learning_rate: 3e-5
+warmup_proportion: 0.1 
+weight_decay: 0.1 
+```
+**3.开始多目标任务训练**
+
+最后，我们像DEMO1和DEMO2一样创建`Controller`，实例化各个任务实例、载入预训练模型并启动多任务训练：
+
+```yaml
+import paddlepalm as palm
+
+if __name__ == '__main__':
+    controller = palm.Controller('config_demo3.yaml', task_dir='demo3_tasks')
+    controller.load_pretrain('pretrain_model/ernie/params')
+    controller.train()
+ ```
+
+可以看到如下日志输出。
+
+```
+Global step: 1. Task: cls4, step 1/15 (epoch 0), loss: 1.344, speed: 0.50 steps/s
+Global step: 10. Task: cls4, step 5/15 (epoch 0), loss: 1.398, speed: 2.19 steps/s
+Global step: 20. Task: cls2, step 5/15 (epoch 0), loss: 1.260, speed: 2.64 steps/s
+cls4: train finished!
+cls4: inference model saved at output_model/thirdrun/infer_model
+cls5: train finished!
+cls5: inference model saved at output_model/thirdrun/infer_model
+Global step: 30. Task: cls2, step 7/15 (epoch 0), loss: 0.961, speed: 0.04 steps/s
+cls2: train finished!
+cls2: inference model saved at output_model/thirdrun/infer_model
+Global step: 40. Task: cls6, step 4/15 (epoch 0), loss: 1.412, speed: 2.74 steps/s
+Global step: 50. Task: cls2, step 12/15 (epoch 0), loss: 1.011, speed: 2.19 steps/s
+cls6: train finished!
+cls6: inference model saved at output_model/thirdrun/infer_model
+cls1: train finished!
+cls1: inference model saved at output_model/thirdrun/infer_model
+Global step: 60. Task: cls3, step 7/15 (epoch 0), loss: 1.363, speed: 2.72 steps/s
+cls3: train finished!
+cls3: inference model saved at output_model/thirdrun/infer_model
+```
+
+对本DEMO更深入的理解可以参考[多目标任务下的训练终止条件与预期训练步数](#多目标任务下的训练终止条件与预期训练步数)。
+
+## 进阶篇
+本章节更深入的对paddlepalm的使用方法展开介绍，并提供一些提高使用效率的小技巧。
+
+### 配置广播机制
+
+![PALM原理图](https://tva1.sinaimg.cn/large/006y8mN6ly1g8j1isf3fcj31ne0tyqbd.jpg)
+
+要完成多任务学习，我们需要对主干网络、各个任务以及训练方式进行必要的配置，为此，框架实现了一套高效的配置广播机制。如上图，通过yaml语言可以描述主干网络和各个任务实例的相关配置，并存储于文件中。由于任务实例可能有多个，且部分超参数会同时被主干网络和任务实例用到，因此对于这些需要“重复配置”却取值相同的超参数，可以写入全局配置文件中，框架在解析全局配置文件时会自动将其“广播”给主干网络和各个任务实例。
+
+此外，全局配置文件的优先级要高于主干网络和任务实例的配置文件，因此当某个超参数在全局配置文件的取值与其在其余位置的取值冲突时，框架以全局配置文件中的取值为准。
+
+同时，为了方便进行大规模实验和超参数调优，凡是在**全局配置文件**中出现的超参数，均可以通过命令行进行控制，例如，对于如下全局配置文件
+
+```yaml
+...
+learning_rate: 1e-3
+batch_size: 32
+...
+```
+
+我们可能希望通过命令行临时调整学习率`learning_rate`和批大小`batch_size`，因此我们在运行训练脚本时可以通过如下方式对其进行改变。
+
+```shell
+python demo3.py --learning_rate 1e-4 --batch_size 64
+```
+
+因此，各种配置方式的优先级如下
+
+**命令行 > 全局配置文件 > 任务实例配置文件&主干网络配置文件**
+
+### reader、backbone与paradigm的选择
+
+reader、backbone和paradigm是实现各类任务的三大基础组件，其中reader为数据集载入与处理工具，将一定格式的输入数据集自动转换成确定的输出元素字典（如单词id序列，位置id序列等）；backbone为主干网络，将来自reader的一部分输出转换为高阶抽象的输出元素字典（如词向量、句向量、编码器输出的上下文相关词向量等）；paradigm为任务范式，将来自reader的一部分输出和backbone输出的对原始输入的高阶抽象转换为训练所需要的loss以及预测所需要的输出等。
+
+框架对这三部分组件的实现基于一种解耦合的设计，每个组件都会包括对输入对象的描述inputs_attr(s)和对输出对象的描述outputs_attr，每个输入或输出对象都会包含名字（描述含义）、形状（tensor shape）和数值类型（data type）。例如，主干网络BERT的输入输出对象的声明如下
+
+```python
+    @property
+    def inputs_attr(self):
+        return {"token_ids": [[None, None], 'int64'],
+                "position_ids": [[None, None], 'int64'],
+                "segment_ids": [[None, None], 'int64'],
+                "input_mask": [[None, None], 'float32']}
+
+    @property
+    def outputs_attr(self):
+        return {"word_embedding": [[None, None, self._emb_size], 'float32'],
+                "embedding_table": [[None, self._voc_size, self._emb_size], 'float32'],
+                "encoder_outputs": [[None, None, self._emb_size], 'float32'],
+                "sentence_embedding": [[None, self._emb_size], 'float32'],
+                "sentence_pair_embedding": [[None, self._emb_size], 'float32']}
+```
+
+其中`inputs_attr`描述了BERT的输入对象，包含`token_ids`, `position_ids`, `segment_ids`和`input_mask`，并且附带了它们的形状（None表示Tensor在该维度的大小可变）和数据类型。`outputs_attr`则描述了BERT模块能提供的输出对象，包含`word_embedding`, `embedding_table`, `encoder_outputs`等。
+
+当用户创建任务实例时，只需要保证每个组件的输入对象是包含在上游组件的输出内的，那么这些组件就可以搭配在一起使用。其中，backbone的上游组件是reader，paradigm的上游组件同时包含reader和backbone。
+
+
+### 多目标任务下的训练终止条件与预期训练步数
+
+#### 多个目标任务
+框架支持设定多个目标任务，当全局配置文件的`task_instance`字段指定超过一个任务实例时，**这多个任务实例默认均为目标任务（即`target_tag`字段被自动填充为全1）**。对于被设置成目标任务的任务实例，框架会为其计算预期的训练步数并在达到预期训练步数后为其保存预测模型。
+
+当框架存在多个目标任务时，全局配置文件中的`num_epochs`（训练集遍历次数）仅会作用于第一个出现的目标任务，称为主任务（main task）。框架会根据主任务的训练步数来推理其他目标任务的预期训练步数（可通过`mix_ratio`控制，详情见下一节）。**注意，除了用来标记`num_epochs`的作用对象外，主任务与其他目标任务没有任何不同。**
+
+*注意：在多目标任务训练时，依然可以使用辅助任务来提升所有目标任务的测试集表现，但是要注意使用target_tag为引入的辅助任务打上辅助标记「0」*
+
+
+#### 训练终止条件
+在训练开始前，`Controller`会为所有每个目标任务计算出预期的训练步数。当某个目标任务的完成预期的训练步数后，`Controller`保存该任务的预测模型，而后继续按照设定的各任务的采样概率进行多任务训练。当所有目标任务均达到预期的训练步数后，多任务学习终止。需要注意的是，`Controller`不会为辅助任务计算预期训练步数，也不会为其保存预测模型，其仅仅起到“陪同目标任务训练”的作用，不会影响到多任务学习的终止与否。
+
+#### 任务采样概率与预期训练步数
+此外，在默认情况下，每个训练step的各个任务被采样到的概率均等，若用户希望更改其中某些任务的采样概率（比如某些任务的训练集较小，希望减少对其采样的次数；或某些任务较难，希望被更多的训练），可以在全局配置文件中通过`mix_ratio`字段控制各个任务的采样概率。例如，我们有三个任务，其中mrqa任务为目标任务，其余为辅助任务，我们对其`mix_ratio`进行如下设定：
+
+```yaml
+task_instance: mrqa, match4mrqa, mlm4mrqa
+mix_ratio: 1.0, 0.5, 0.5
+```
+
+上述设置表示`match4mrqa`和`mlm4mrqa`任务的期望被采样次数均为`mrqa`任务的一半。此时，在mrqa任务被设置为目标任务的情况下，若mrqa任务训练一个epoch要经历5000 steps，且全局配置文件中设置了num_epochs为2，则根据上述`mix_ratio`的设置，mrqa任务将被训练5000\*2\*1.0=10000个steps，而`match4mrqa`任务和`mlm4mrqa`任务都会被训练5000个steps**左右**。
+
+> 注意：若match4mrqa, mlm4mrqa被设置为辅助任务，则实际训练步数可能略多或略少于5000个steps。对于目标任务，则是精确的5000 steps。
+
+#### 多个目标任务时预期训练步数的计算
+
+当存在多个目标任务时，`num_epochs`仅作用于**第一个设定的目标任务（称为“主任务(main task)”）**，而后根据`mix_ratio`的设定为其余目标任务和辅助任务计算出预期的训练步数。
+
+### 模型保存与预测机制
+
+`Controller`可以在训练过程中保存两类模型，一类称为检查点模型（checkpoint），一类为预测模型（inference model）。
+
+检查点模型会描述当前训练时刻的网络全局状态，包括backbone、所有任务以及优化器的全局参数，局部参数，长期变量等，即完整的多任务学习计算图。检查点模型用于训练意外终止时的断点恢复，或分阶段的对相同的模型进行连续训练。对于检查点模型，`Controller`默认不进行保存，但是用户可以通过在全局配置文件中添加`save_every_n_steps`来控制检查点模型的保存频率，例如设置为5000，则表示每5000个全局训练steps就会保存一次检查点模型。检查点模型放置在全局配置文件中设置的`save_path`指定的路径下。
+
+预测模型则描述的是某个任务的完整预测模型，该模型内不会包含其他任务的参数，也不会保存优化器、dropout层等推理阶段不需要的节点。在保存预测模型时，`Controller`会同时保存预测相关的必要配置，如预测模型的输入输出列表，在进行预测时，可以调用实例化后的`Controller`的预测接口`pred`直接对相关任务进行预测。关于预测的用法示例可以参加DEMO2。
+
+### 分布式训练
+
+框架将单机单卡训练与单机多卡训练进行了无缝集成。当环境内有多张可用的GPU显卡时，框架会自动将模型复制到多张卡上，并且对于每个step，每张卡都会计算`batch_size`个训练样本，框架会自动对多卡的梯度进行合并。例如，环境中存在8张显卡，且`batch_size`设置为32时，这时每个step的实际batch size为32\*8=256。
+
+当用户在多卡环境下希望仅用一张卡进行训练时，可以通过改变环境变量[CUDA_VISIBLE_DEVICES](https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/)来进行控制。
+
+
+
+## 附录A：内置数据集载入与处理工具（reader）
+
+所有的内置reader均同时支持中英文输入数据，**默认读取的数据为英文数据**，希望读入中文数据时，需在配置文件中设置
+
+```yaml
+for_cn: True
+```
+
+所有的内置reader，均支持以下字段
+
+```yaml
+vocab_path（REQUIRED）: str类型。字典文件路径。
+max_seq_len（REQUIRED）: int类型。切词后的序列最大长度（即token ids的最大长度）。注意经过分词后，token ids的数量往往多于原始的单词数（e.g., 使用wordpiece tokenizer时）。
+batch_size（REQUIRED）: int类型。训练或预测时的批大小（每个step喂入神经网络的样本数）。
+train_file（REQUIRED）: str类型。训练集文件所在路径。仅进行预测时，该字段可不设置。
+pred_file（REQUIRED）: str类型。测试集文件所在路径。仅进行训练时，该字段可不设置。
+
+do_lower_case（OPTIONAL）: bool类型，默认为False。是否将大写英文字母转换成小写。
+shuffle（OPTIONAL）: bool类型，默认为True。训练阶段打乱数据集样本的标志位，当置为True时，对数据集的样本进行全局打乱。注意，该标志位的设置不会影响预测阶段（预测阶段不会shuffle数据集）。
+seed（OPTIONAL）: int类型，默认为。
+pred_batch_size（OPTIONAL）: int类型。预测阶段的批大小，当该参数未设置时，预测阶段的批大小取决于`batch_size`字段的值。
+print_first_n（OPTIONAL）: int类型。打印数据集的前n条样本和对应的reader输出，默认为0。
+```
+
+#### 文本分类数据集reader工具：cls
+
+该reader完成文本分类数据集的载入与处理，reader接受[tsv格式](https://en.wikipedia.org/wiki/Tab-separated_values)的数据集输入，数据集应该包含两列，一列为样本标签`label`，一列为原始文本`text_a`。数据集范例可参考`data/cls4mrqa`中的数据集文件，格式形如
+
+```
+label   text_a                                                                                   
+1   when was the last time the san antonio spurs missed the playoffshave only missed the playoffs four times since entering the NBA
+0   the creation of the federal reserve system was an attempt toReserve System ( also known as the Federal Reserve or simply the Fed ) is the central banking system of the United States of America . 
+2   group f / 64 was a major backlash against the earlier photographic movement off / 64 was formed , Edward Weston went to a meeting of the John Reed Club , which was founded to support Marxist artists and writers . 
+0   Bessarabia eventually became under the control of which country?
+```
+***注意：数据集的第一列必须为header，即标注每一列的列名***
+
+该reader额外包含以下配置字段
+
+```yaml
+n_classes（REQUIRED）: int类型。分类任务的类别数。
+```
+
+reader的输出（生成器每次yield出的数据）包含以下字段
+
+```yaml
+token_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的单词id。
+position_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的位置id。
+segment_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持BERT、ERNIE等模型的输入。
+input_mask: 一个shape为[batch_size, seq_len]的矩阵，其中的每个元素为0或1，表示该位置是否是padding词（为1时代表是真实词，为0时代表是填充词）。
+label_ids: 一个shape为[batch_size]的矩阵，其中的每个元素为该样本的类别标签。
+task_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持ERNIE模型的输入。
+```
+
+当处于预测阶段时，reader所yield出的数据不会包含`label_ids`字段。
+
+
+#### 文本匹配数据集reader工具：match
+
+该reader完成文本匹配数据集的载入与处理，reader接受[tsv格式](https://en.wikipedia.org/wiki/Tab-separated_values)的数据集输入，数据集应该包含三列，一列为样本标签`label`，其余两列分别为待匹配的文本`text_a`和文本`text_b`。数据集范例可参考`data/match4mrqa`中的数据集文件，格式形如
+
+```yaml
+label   text_a  text_b                                                                           
+1   From what work of Durkheim's was interaction ritual theory derived? **[TAB]** Subsequent to these developments, Randall Collins (2004) formulated his interaction ritual theory by drawing on Durkheim's work on totemic rituals that was extended by Goffman (1964/2013; 1967) into everyday focused encounters. Based on interaction ritual theory, we experience different levels
+0   where is port au prince located in haiti **[TAB]** Its population is difficult to ascertain due to the rapid growth of slums in the hillsides
+0   What is the world’s first-ever pilsner type blond lager, the company also awarded the Master Homebrewer Competition held in San Francisco to an award-winning brewer who won the prestigious American Homebrewers Associations' Homebrewer of the Year award in 2013? **[TAB]** of the Year award in 2013, becoming the first woman in thirty years, and the first African American person ever to ever win the award.
+1   What has Pakistan told phone companies? **[TAB]** Islamabad, Pakistan (CNN) -- Under heavy criticism for a telling cell phone carriers to ban certain words in text messages, the Pakistan Telecommunication Authority went into damage control mode Wednesday.
+```
+
+***注意：数据集的第一列必须为header，即标注每一列的列名***
+
+reader的输出（生成器每次yield出的数据）包含以下字段：
+
+```yaml
+token_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本（文本对），其中的每个元素为文本对中的每个token对应的单词id，文本对使用`[SEP]`所对应的id隔开。
+position_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的位置id。
+segment_ids: 一个shape为[batch_size, seq_len]的矩阵，在文本1的token位置，元素取值为0；在文本2的token位置，元素取值为1。用于支持BERT、ERNIE等模型的输入。
+input_mask: 一个shape为[batch_size, seq_len]的矩阵，其中的每个元素为0或1，表示该位置是否是padding词（为1时代表是真实词，为0时代表是填充词）。
+label_ids: 一个shape为[batch_size]的矩阵，其中的每个元素为该样本的类别标签，为0时表示两段文本不匹配，为1时代表构成匹配。
+task_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持ERNIE模型的输入。
+```
+
+当处于预测阶段时，reader所yield出的数据不会包含`label_ids`字段。
+
+
+#### 机器阅读理解数据集reader工具：mrc
+
+该reader支持基于滑动窗口的机器阅读理解数据集载入，可以自动将较长的context按照步长切分成若干子文档，每个子文档与question分别计算答案片段，并在最终阶段合并。该reader接受[json格式]()的数据集。数据集范例可参考`data/mrqa`中的数据集文件，格式如下。
+
+```json
+{
+    "version": "1.0",
+    "data": [
+        {"title": "...",
+         "paragraphs": [
+            {"context": "...",
+             "qas": [
+                {"question": "..."
+                 "id": "..."
+                 "answers": [
+                    {"text": "...",
+                     "answer_start": ...}
+                    {...}
+                    ...
+                    ]
+                 }
+                 {...}
+                 ...
+             {...},
+             ...
+             ]
+         }
+         {...}
+         ...
+     ]
+ }
+ ```
+ 
+数据集的最外层数据结构为字典，包含数据集版本号`version`和数据集`data`。在`data`字段内为各个样本，每个样本包含文章标题`title`和若干段落`paragraphs`，在`paragraphs`中的每个元素为一个段落`context`，基于该段落的内容，可以包含若干个问题和对应的答案`qas`，答案均位于该段落内。对于`qas`中的每个元素，包含一个问题`question`和一个全局唯一的标识`id`，以及（若干）答案`answers`。答案中的每个元素包含答案本身`text`及其在`context`中的起始位置`answer_start`。注意起始位置为字符级。此外，在测试集中，`qas`可以不包含`answers`字段。
+
+该reader包含如下额外的可配置字段：
+
+```yaml
+doc_stride (REQUIRED): int类型。对context应用滑动窗口时的滑动步长。
+max_query_len (REQUIRED): int类型。query的最大长度。
+max_answer_len (REQUIRED): int类型。预测阶段answer的最大长度，不训练时该字段可为空。
+n_best_size (OPTIONAL): int类型。预测阶段合并滑动窗口的样本时，每个样本所取的n_best列表大小。
+```
+
+
+reader的输出（生成器每次yield出的数据）包含以下字段：
+
+```yaml
+token_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本（文本对），文本1为context，文本2为question，其中的每个元素为文本对中的每个token对应的单词id，文本对使用`[SEP]`所对应的id隔开。
+position_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的位置id。
+segment_ids: 一个shape为[batch_size, seq_len]的矩阵，在文本1的token位置，元素取值为0；在文本2的token位置，元素取值为1。用于支持BERT、ERNIE等模型的输入。
+input_mask: 一个shape为[batch_size, seq_len]的矩阵，其中的每个元素为0或1，表示该位置是否是padding词（为1时代表是真实词，为0时代表是填充词）。
+task_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持ERNIE模型的输入。
+start_positions: 一个shape为[batch_size]的向量，每个元素代表当前样本的答案片段的起始位置。
+end_positions: 一个shape为[batch_size]的向量，每个元素代表当前样本的答案片段的结束位置。
+```
+
+当处于预测阶段时，reader所yield出的数据不会包含`label_ids`字段，但会额外的包含`unique_ids`字段：
+
+```yaml
+unique_ids: 一个shape为[batch_size, seq_len]的矩阵，代表每个样本的全局唯一的id，用于预测后对滑动窗口的结果进行合并。
+```
+
+
+#### 掩码语言模型数据集reader工具：mlm
+该reader完成掩码语言模型数据集的载入与处理，reader接受[tsv格式](https://en.wikipedia.org/wiki/Tab-separated_values)的数据集输入，MLM任务为自监督任务，数据集仅包含一列`text_a`，reader会自动为每个样本生成随机的训练标签。格式如下
+
+```
+text_a                                                                                           
+Subsequent to these developments, Randall Collins (2004) formulated his interaction ritual theory by drawing on Durkheim's work on totemic rituals that was extended by Goffman (1964/2013; 1967) into everyday focused encounters. 
+Presidential spokesman Abigail Valte earlier Saturday urged residents of low-lying and mountainous areas that could be hit hard by the storm to evacuate, the state news agency said, citing an interview conducted on a government radio station. World Vision, the Christian humanitarian organization, said Saturday that it had to postpone some of its relief efforts due to Nalgae, with two of three emergency teams set to deploy once the storm passes. Another team is in Bulcan province, most of which is "still submerged" because of Nesat. The group is focusing its post-Nesat efforts on two communities in Manila and three in the northern Isabela and Zambales provinces. 
+of the Year award in 2013, becoming the first woman in thirty years, and the first African American person ever to ever win the award. After an extensive career with the California State Legislature she began working for PicoBrew, a product development company in Seattle, WA that specializes in automated brewing equipment. 
+the gakkel ridge is a boundary between which two tectonic plates Mid-Atlantic Ridge ( MAR ) is a mid-ocean ridge , a divergent tectonic plate or constructive plate boundary located along the floor of the Atlantic Ocean , and part of the longest mountain range in the world . The ridge extends from a junction with the Gakkel Ridge ( Mid-Arctic Ridge ) northeast of Greenland southward to the Bouvet Triple Junction in the South Atlantic .
+```
+
+***注意：数据集的第一列必须为header，即标注每一列的列名***
+
+reader的输出（生成器每次yield出的数据）包含以下对象：
+
+```yaml
+token_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的单词id。
+position_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的位置id。
+segment_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持BERT、ERNIE等模型的输入。
+input_mask: 一个shape为[batch_size, seq_len]的矩阵，其中的每个元素为0或1，表示该位置是否是padding词（为1时代表是真实词，为0时代表是填充词）。
+mask_label: 一个shape为[None]的向量，其中的每个元素为被mask掉的单词的真实单词id。
+mask_pos: 一个shape为[None]的向量，长度与`mask_pos`一致且元素一一对应。每个元素表示被mask掉的单词的位置。
+task_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持ERNIE模型的输入。
+```
+
+## 附录B：内置主干网络（backbone）
+
+框架中内置了BERT和ERNIE作为主干网络，未来框架会引入更多的骨干网络如XLNet等。
+
+#### BERT
+
+BERT包含了如下输入对象
+
+```yaml
+token_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的单词id。
+position_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的位置id。
+segment_ids: 一个shape为[batch_size, seq_len]的0/1矩阵，用于支持BERT、ERNIE等模型的输入，当元素为0时，代表当前token属于分类任务或匹配任务的text1，为1时代表当前token属于匹配任务的text2.
+input_mask: 一个shape为[batch_size, seq_len]的矩阵，其中的每个元素为0或1，表示该位置是否是padding词（为1时代表是真实词，为0时代表是填充词）。
+```
+
+提供了如下输出对象供下游组件使用。
+
+```yaml
+word_embedding: 一个shape为[batch_size, seq_len, emb_size]的张量（Tensor），float32类型。表示当前batch中各个样本的（上下文无关）词向量序列。
+embedding_table: 一个shape为[vocab_size, emb_size]的矩阵，float32类型。表示BERT当前维护的词向量查找表矩阵。
+encoder_outputs: 一个shape为[batch_size, seq_len, hidden_size]的Tensor, float32类型。表示BERT encoder对当前batch中各个样本的encoding结果。
+sentence_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+sentence_pair_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+```
+
+#### ERNIE
+
+ERNIE包含了如下输入对象
+
+```yaml
+token_ids: 。一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的单词id。
+position_ids: 一个shape为[batch_size, seq_len]的矩阵，每行是一条样本，其中的每个元素为文本中的每个token对应的位置id。
+segment_ids: 一个shape为[batch_size, seq_len]的0/1矩阵，用于支持BERT、ERNIE等模型的输入，当元素为0时，代表当前token属于分类任务或匹配任务的text1，为1时代表当前token属于匹配任务的text2.
+input_mask: 一个shape为[batch_size, seq_len]的矩阵，其中的每个元素为0或1，表示该位置是否是padding词（为1时代表是真实词，为0时代表是填充词）。
+segment_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持BERT、ERNIE等模型的输入。
+task_ids: 一个shape为[batch_size, seq_len]的全0矩阵，用于支持ERNIE finetuning。
+```
+
+提供了如下输出对象供下游组件使用。
+
+```yaml
+word_embedding: 一个shape为[batch_size, seq_len, emb_size]的张量（Tensor），float32类型。表示当前batch中各个样本的（上下文无关）词向量序列。
+embedding_table: 一个shape为[vocab_size, emb_size]的矩阵，float32类型。表示BERT当前维护的词向量查找表矩阵。
+encoder_outputs: 一个shape为[batch_size, seq_len, hidden_size]的Tensor, float32类型。表示BERT encoder对当前batch中各个样本的encoding结果。
+sentence_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+sentence_pair_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+```
+
+
+## 附录C：内置任务范式（paradigm）
+
+#### 分类范式：cls
+
+分类范式额外包含以下配置字段：
+
+```yaml
+n_classes（REQUIRED）: int类型。分类任务的类别数。
+pred_output_path (OPTIONAL) : str类型。预测输出结果的保存路径，当该参数未空时，保存至全局配置文件中的`save_path`字段指定路径下的任务目录。
+```
+
+分类范式包含如下的输入对象：
+
+训练阶段：
+```yaml
+sentence_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+label_ids: 一个shape为[batch_size]的矩阵，其中的每个元素为该样本的类别标签。
+```
+
+预测阶段：
+```yaml
+sentence_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+```
+
+在训练阶段，输出loss；预测阶段输出各个类别的预测概率。
+
+#### 匹配范式：match
+
+
+匹配范式额外包含以下配置字段：
+
+```yaml
+pred_output_path (OPTIONAL) : str类型。预测输出结果的保存路径，当该参数未空时，保存至全局配置文件中的`save_path`字段指定路径下的任务目录。
+```
+
+匹配范式包含如下的输入对象：
+
+训练阶段：
+```yaml
+sentence_pair_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+label_ids: 一个shape为[batch_size]的矩阵，其中的每个元素为该样本的类别标签，为0时表示两段文本不匹配，为1时代表构成匹配
+```
+
+预测阶段：
+```yaml
+sentence_pair_embedding: 一个shape为[batch_size, hidden_size]的matrix, float32类型。每一行代表BERT encoder对当前batch中相应样本的句子向量（sentence embedding）
+```
+
+在训练阶段，输出loss；预测阶段输出匹配与否的概率分布。
+
+#### 机器阅读理解范式：mrc
+
+
+分类范式额外包含以下配置字段：
+
+```yaml
+max_answer_len（REQUIRED）: int类型。预测的最大答案长度
+n_best_size (OPTIONAL) : int类型，默认为20。预测时保存的nbest回答文件中每条样本的n_best数量
+pred_output_path (OPTIONAL) : str类型。预测输出结果的保存路径，当该参数未空时，保存至全局配置文件中的`save_path`字段指定路径下的任务目录
+```
+
+机器阅读理解范式包含如下的输入对象：
+
+训练阶段：
+```yaml
+encoder_outputs: 一个shape为[batch_size, seq_len, hidden_size]的Tensor, float32类型。表示BERT encoder对当前batch中各个样本的encoding结果。
+start_positions: 一个shape为[batch_size]的向量，每个元素代表当前样本的答案片段的起始位置。
+end_positions: 一个shape为[batch_size]的向量，每个元素代表当前样本的答案片段的结束位置。
+```
+
+预测阶段：
+```yaml
+encoder_outputs: 一个shape为[batch_size, seq_len, hidden_size]的Tensor, float32类型。表示BERT encoder对当前batch中各个样本的encoding结果。
+unique_ids: 一个shape为[batch_size, seq_len]的矩阵，代表每个样本的全局唯一的id，用于预测后对滑动窗口的结果进行合并。
+```
+
+
+#### 掩码语言模型范式：mlm
+
+该任务范式为无监督任务范式，不支持预测，仅用于（辅助）训练。包含如下的输入对象：
+
+```yaml
+mask_label: 一个shape为[None]的向量，其中的每个元素为被mask掉的单词的真实单词id。
+mask_pos": 一个shape为[None]的向量，长度与`mask_pos`一致且元素一一对应。每个元素表示被mask掉的单词的位置。
+embedding_table: 一个shape为[vocab_size, emb_size]的矩阵，float32类型。表示BERT当前维护的词向量查找表矩阵。
+encoder_outputs: 一个shape为[batch_size, seq_len, hidden_size]的Tensor, float32类型。表示BERT encoder对当前batch中各个样本的encoding结果。
+```
+
+## 附录D：可配置的全局参数列表
+
+```yaml
+
+task_instance（REQUIRED）: str类型。需要进行训练或预测的任务实例名。在多任务模式下，多个任务之间使用逗号`,`隔开。名称选取自任务实例配置文件的文件名（不包含后缀.yaml）。
+mix_ratio (OPTIONAL): str类型。每个任务的训练阶段的采样概率，各个值通过逗号`,`隔开，且与task_instance中的元素一一对应。默认每个任务的采样概率均为1.0，即所有任务等概率采样（代表与主任务采样次数的期望相同）。详情见 《进阶篇-训练终止条件与预期训练步数》。
+target_tag (OPTIONAL): str类型。目标/辅助任务标志位，各个值通过逗号`,`隔开，且与task_instance中的元素一一对应。标记为1的任务代表目标任务，标记为0的任务代表辅助任务。默认每个值均为1（即默认每个任务为目标任务）。相关使用示例见DEMO2。
+task_reuse_tag (OPTIONAL): str类型。任务层复用标志位，各个值通过逗号`,`隔开，且与task_instance中的元素一一对应。元素取值相同的任务会自动共享任务层参数，取值不同的任务不复用任务层参数。相关使用示例见DEMO3。
+
+backbone（REQUIRED）: str类型。主干网络名。
+backbone_config_path (OPTIONAL): str类型。主干网络配置文件路径。
+
+save_path（REQUIRED）: str类型。checkpoint文件和各个目标任务的预测模型保存路径。
+vocab_path（REQUIRED）: str类型。字典文件，纯文本格式存储，其中每行为一个单词，供reader、backbone和各个任务使用。
+do_lower_case (OPTIONAL): bool类型。大小写标志位。默认为False，即区分大小写。
+for_cn: bool类型。中文模式标志位。默认为False，即默认输入为英文，设置为True后，分词器、后处理等按照中文语言进行处理。
+
+print_every_n_steps (OPTIONAL): int类型。默认为5。训练阶段打印日志的频率（step为单位）。
+save_every_n_steps (OPTIONAL): int类型。默认为-1。训练过程中保存checkpoint模型的频率，默认不保存。
+
+optimizer（REQUIRED）: str类型。优化器名称，目前框架只支持adam，未来会支持更多优化器。
+learning_rate（REQUIRED）: str类型。训练阶段的学习率。
+batch_size（REQUIRED）: int类型。批大小，即每个训练或推理step所使用样本数。
+epoch（REQUIRED）: int类型。主任务的训练epoch数。
+
+use_gpu (OPTIONAL): bool类型。默认为True。框架默认使用GPU进行单机单卡或分布式训练，若希望使用cpu训练或推理，可将该标志位置为False。
+
+warmup_proportion (OPTIONAL): float类型。默认为0。对预训练模型finetuning时的warmup的训练step占预估的全部训练步数的比例。
+use_ema (OPTIONAL): bool类型。默认为False。是否开启[ema](https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average) 进行训练和推理。
+ema_decay (OPTIONAL): float类型。默认为0。开启ema时的权重衰减指数。
+random_seed (OPTIONAL): int类型。随机种子，默认1。
+```
+
+## License
+
+This tutorial is contributed by [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) and licensed under the [Apache-2.0 license](https://github.com/PaddlePaddle/models/blob/develop/LICENSE).
+
+## 许可证书
+
+此向导由[PaddlePaddle](https://github.com/PaddlePaddle/Paddle)贡献，受[Apache-2.0 license](https://github.com/PaddlePaddle/models/blob/develop/LICENSE)许可认证。
+

backbone/bert.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""v1.1 
-BERT model."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from paddle import fluid
-from paddle.fluid import layers
-
-from paddlepalm.backbone.utils.transformer import pre_process_layer, encoder
-from paddlepalm.interface import backbone
-
-    
-class Model(backbone):
-    
-    def __init__(self, config, phase):
-
-        # self._is_training = phase == 'train' # backbone一般不用关心运行阶段，因为outputs在任何阶段基本不会变
-        self._emb_size = config["hidden_size"]
-        self._n_layer = config["num_hidden_layers"]
-        self._n_head = config["num_attention_heads"]
-        self._voc_size = config["vocab_size"]
-        self._max_position_seq_len = config["max_position_embeddings"]
-        self._sent_types = config["type_vocab_size"]
-        self._hidden_act = config["hidden_act"]
-        self._prepostprocess_dropout = config["hidden_dropout_prob"]
-        self._attention_dropout = config["attention_probs_dropout_prob"]
-
-        self._word_emb_name = "word_embedding"
-        self._pos_emb_name = "pos_embedding"
-        self._sent_emb_name = "sent_embedding"
-
-        # Initialize all weigths by truncated normal initializer, and all biases 
-        # will be initialized by constant zero by default.
-        self._param_initializer = fluid.initializer.TruncatedNormal(
-            scale=config["initializer_range"])
-
-    @property
-    def inputs_attr(self):
-        return {"token_ids": [[-1, -1, 1], 'int64'],
-                "position_ids": [[-1, -1, 1], 'int64'],
-                "segment_ids": [[-1, -1, 1], 'int64'],
-                "input_mask": [[-1, -1, 1], 'float32']}
-
-    @property
-    def outputs_attr(self):
-        return {"word_embedding": [[-1, -1, self._emb_size], 'float32'],
-                "embedding_table": [[-1, self._voc_size, self._emb_size], 'float32'],
-                "encoder_outputs": [[-1, -1, self._emb_size], 'float32'],
-                "sentence_embedding": [[-1, self._emb_size], 'float32'],
-                "sentence_pair_embedding": [[-1, self._emb_size], 'float32']}
-
-    def build(self, inputs, scope_name=""):
-        src_ids = inputs['token_ids']
-        pos_ids = inputs['position_ids']
-        sent_ids = inputs['segment_ids']
-        input_mask = inputs['input_mask']
-
-        self._emb_dtype = 'float32'
-        # padding id in vocabulary must be set to 0
-        emb_out = fluid.layers.embedding(
-            input=src_ids,
-            size=[self._voc_size, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._word_emb_name, initializer=self._param_initializer),
-            is_sparse=False)
-
-        # fluid.global_scope().find_var('backbone-word_embedding').get_tensor()
-        embedding_table = fluid.default_main_program().global_block().var(scope_name+self._word_emb_name)
-        
-        position_emb_out = fluid.layers.embedding(
-            input=pos_ids,
-            size=[self._max_position_seq_len, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._pos_emb_name, initializer=self._param_initializer))
-
-        sent_emb_out = fluid.layers.embedding(
-            sent_ids,
-            size=[self._sent_types, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._sent_emb_name, initializer=self._param_initializer))
-
-        emb_out = emb_out + position_emb_out
-        emb_out = emb_out + sent_emb_out
-
-        emb_out = pre_process_layer(
-            emb_out, 'nd', self._prepostprocess_dropout, name=scope_name+'pre_encoder')
-
-        self_attn_mask = fluid.layers.matmul(
-            x=input_mask, y=input_mask, transpose_y=True)
-
-        self_attn_mask = fluid.layers.scale(
-            x=self_attn_mask, scale=10000.0, bias=-1.0, bias_after_scale=False)
-        n_head_self_attn_mask = fluid.layers.stack(
-            x=[self_attn_mask] * self._n_head, axis=1)
-        n_head_self_attn_mask.stop_gradient = True
-
-        enc_out = encoder(
-            enc_input=emb_out,
-            attn_bias=n_head_self_attn_mask,
-            n_layer=self._n_layer,
-            n_head=self._n_head,
-            d_key=self._emb_size // self._n_head,
-            d_value=self._emb_size // self._n_head,
-            d_model=self._emb_size,
-            d_inner_hid=self._emb_size * 4,
-            prepostprocess_dropout=self._prepostprocess_dropout,
-            attention_dropout=self._attention_dropout,
-            relu_dropout=0,
-            hidden_act=self._hidden_act,
-            preprocess_cmd="",
-            postprocess_cmd="dan",
-            param_initializer=self._param_initializer,
-            name=scope_name+'encoder')
-
-        
-        next_sent_feat = fluid.layers.slice(
-            input=enc_out, axes=[1], starts=[0], ends=[1])
-        next_sent_feat = fluid.layers.reshape(next_sent_feat, [-1, next_sent_feat.shape[-1]])
-        next_sent_feat = fluid.layers.fc(
-            input=next_sent_feat,
-            size=self._emb_size,
-            act="tanh",
-            param_attr=fluid.ParamAttr(
-                name=scope_name+"pooled_fc.w_0", initializer=self._param_initializer),
-            bias_attr=scope_name+"pooled_fc.b_0")
-
-        return {'embedding_table': embedding_table,
-                'word_embedding': emb_out,
-                'encoder_outputs': enc_out,
-                'sentence_embedding': next_sent_feat,
-                'sentence_pair_embedding': next_sent_feat}
-
-    def postprocess(self, rt_outputs):
-        pass
-
-

backbone/ernie.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Ernie model."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from __future__ import absolute_import
-
-from paddle import fluid
-from paddle.fluid import layers
-
-from paddlepalm.backbone.utils.transformer import pre_process_layer, encoder
-from paddlepalm.interface import backbone
-
-
-class Model(backbone):
-
-    def __init__(self,
-                 config,
-                 phase):
-
-        # self._is_training = phase == 'train' # backbone一般不用关心运行阶段，因为outputs在任何阶段基本不会变
-
-        self._emb_size = config['hidden_size']
-        self._n_layer = config['num_hidden_layers']
-        self._n_head = config['num_attention_heads']
-        self._voc_size = config['vocab_size']
-        self._max_position_seq_len = config['max_position_embeddings']
-        if config['sent_type_vocab_size']:
-            self._sent_types = config['sent_type_vocab_size']
-        else:
-            self._sent_types = config['type_vocab_size']
-
-        self._task_types = config['task_type_vocab_size']
-
-        self._hidden_act = config['hidden_act']
-        self._prepostprocess_dropout = config['hidden_dropout_prob']
-        self._attention_dropout = config['attention_probs_dropout_prob']
-
-        self._word_emb_name = "word_embedding"
-        self._pos_emb_name = "pos_embedding"
-        self._sent_emb_name = "sent_embedding"
-        self._task_emb_name = "task_embedding"
-        self._emb_dtype = "float32"
-
-        self._param_initializer = fluid.initializer.TruncatedNormal(
-            scale=config['initializer_range'])
-
-    @property
-    def inputs_attr(self):
-        return {"token_ids": [[-1, -1, 1], 'int64'],
-                "position_ids": [[-1, -1, 1], 'int64'],
-                "segment_ids": [[-1, -1, 1], 'int64'],
-                "input_mask": [[-1, -1, 1], 'float32'],
-                "task_ids": [[-1,-1, 1], 'int64']}
-
-    @property
-    def outputs_attr(self):
-        return {"word_embedding": [[-1, -1, self._emb_size], 'float32'],
-                "embedding_table": [[-1, self._voc_size, self._emb_size], 'float32'],
-                "encoder_outputs": [[-1, -1, self._emb_size], 'float32'],
-                "sentence_embedding": [[-1, self._emb_size], 'float32'],
-                "sentence_pair_embedding": [[-1, self._emb_size], 'float32']}
-
-    def build(self, inputs, scope_name=""):
-
-        src_ids = inputs['token_ids']
-        pos_ids = inputs['position_ids']
-        sent_ids = inputs['segment_ids']
-        input_mask = inputs['input_mask']
-        task_ids = inputs['task_ids']
-
-        # padding id in vocabulary must be set to 0
-        emb_out = fluid.layers.embedding(
-            input=src_ids,
-            size=[self._voc_size, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._word_emb_name, initializer=self._param_initializer),
-            is_sparse=False)
-
-        # fluid.global_scope().find_var('backbone-word_embedding').get_tensor()
-        embedding_table = fluid.default_main_program().global_block().var(scope_name+self._word_emb_name)
-        
-        position_emb_out = fluid.layers.embedding(
-            input=pos_ids,
-            size=[self._max_position_seq_len, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._pos_emb_name, initializer=self._param_initializer))
-
-        sent_emb_out = fluid.layers.embedding(
-            sent_ids,
-            size=[self._sent_types, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._sent_emb_name, initializer=self._param_initializer))
-
-        emb_out = emb_out + position_emb_out
-        emb_out = emb_out + sent_emb_out
-
-        task_emb_out = fluid.layers.embedding(
-            task_ids,
-            size=[self._task_types, self._emb_size],
-            dtype=self._emb_dtype,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+self._task_emb_name,
-                initializer=self._param_initializer))
-
-        emb_out = emb_out + task_emb_out
-
-        emb_out = pre_process_layer(
-            emb_out, 'nd', self._prepostprocess_dropout, name=scope_name+'pre_encoder')
-
-        self_attn_mask = fluid.layers.matmul(
-            x=input_mask, y=input_mask, transpose_y=True)
-
-        self_attn_mask = fluid.layers.scale(
-            x=self_attn_mask, scale=10000.0, bias=-1.0, bias_after_scale=False)
-        n_head_self_attn_mask = fluid.layers.stack(
-            x=[self_attn_mask] * self._n_head, axis=1)
-        n_head_self_attn_mask.stop_gradient = True
-
-        enc_out = encoder(
-            enc_input=emb_out,
-            attn_bias=n_head_self_attn_mask,
-            n_layer=self._n_layer,
-            n_head=self._n_head,
-            d_key=self._emb_size // self._n_head,
-            d_value=self._emb_size // self._n_head,
-            d_model=self._emb_size,
-            d_inner_hid=self._emb_size * 4,
-            prepostprocess_dropout=self._prepostprocess_dropout,
-            attention_dropout=self._attention_dropout,
-            relu_dropout=0,
-            hidden_act=self._hidden_act,
-            preprocess_cmd="",
-            postprocess_cmd="dan",
-            param_initializer=self._param_initializer,
-            name=scope_name+'encoder')
-
-        
-        next_sent_feat = fluid.layers.slice(
-            input=enc_out, axes=[1], starts=[0], ends=[1])
-        next_sent_feat = fluid.layers.reshape(next_sent_feat, [-1, next_sent_feat.shape[-1]])
-        next_sent_feat = fluid.layers.fc(
-            input=next_sent_feat,
-            size=self._emb_size,
-            act="tanh",
-            param_attr=fluid.ParamAttr(
-                name=scope_name+"pooled_fc.w_0", initializer=self._param_initializer),
-            bias_attr=scope_name+"pooled_fc.b_0")
-
-        return {'embedding_table': embedding_table,
-                'word_embedding': emb_out,
-                'encoder_outputs': enc_out,
-                'sentence_embedding': next_sent_feat,
-                'sentence_pair_embedding': next_sent_feat}
-
-    def postprocess(self, rt_outputs):
-        pass

backbone/utils/transformer.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Transformer encoder."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from functools import partial
-
-import paddle.fluid as fluid
-import paddle.fluid.layers as layers
-
-from paddle.fluid.layer_helper import LayerHelper as LayerHelper
-from functools import reduce # py3
-def layer_norm(x, begin_norm_axis=1, epsilon=1e-6, param_attr=None, bias_attr=None):
-    helper = LayerHelper('layer_norm', **locals())
-    mean = layers.reduce_mean(x, dim=begin_norm_axis, keep_dim=True)
-    shift_x = layers.elementwise_sub(x=x, y=mean, axis=0)
-    variance = layers.reduce_mean(layers.square(shift_x), dim=begin_norm_axis, keep_dim=True)
-    r_stdev = layers.rsqrt(variance + epsilon)
-    norm_x = layers.elementwise_mul(x=shift_x, y=r_stdev, axis=0)
-
-    param_shape = [reduce(lambda x, y: x * y, norm_x.shape[begin_norm_axis:])]
-    param_dtype = norm_x.dtype
-    scale = helper.create_parameter(
-        attr=param_attr,
-        shape=param_shape,
-        dtype=param_dtype,
-        default_initializer=fluid.initializer.Constant(1.))
-    bias = helper.create_parameter(
-        attr=bias_attr,
-        shape=param_shape,
-        dtype=param_dtype,
-        is_bias=True,
-        default_initializer=fluid.initializer.Constant(0.))
-
-    out = layers.elementwise_mul(x=norm_x, y=scale, axis=-1)
-    out = layers.elementwise_add(x=out, y=bias, axis=-1)
-
-    return out
-
-
-def multi_head_attention(queries,
-                         keys,
-                         values,
-                         attn_bias,
-                         d_key,
-                         d_value,
-                         d_model,
-                         n_head=1,
-                         dropout_rate=0.,
-                         cache=None,
-                         param_initializer=None,
-                         name='multi_head_att'):
-    """
-    Multi-Head Attention. Note that attn_bias is added to the logit before
-    computing softmax activiation to mask certain selected positions so that
-    they will not considered in attention weights.
-    """
-    keys = queries if keys is None else keys
-    values = keys if values is None else values
-
-    if not (len(queries.shape) == len(keys.shape) == len(values.shape) == 3):
-        raise ValueError(
-            "Inputs: quries, keys and values should all be 3-D tensors.")
-
-    def __compute_qkv(queries, keys, values, n_head, d_key, d_value):
-        """
-        Add linear projection to queries, keys, and values.
-        """
-        q = layers.fc(input=queries,
-                      size=d_key * n_head,
-                      num_flatten_dims=2,
-                      param_attr=fluid.ParamAttr(
-                          name=name + '_query_fc.w_0',
-                          initializer=param_initializer),
-                      bias_attr=name + '_query_fc.b_0')
-        k = layers.fc(input=keys,
-                      size=d_key * n_head,
-                      num_flatten_dims=2,
-                      param_attr=fluid.ParamAttr(
-                          name=name + '_key_fc.w_0',
-                          initializer=param_initializer),
-                      bias_attr=name + '_key_fc.b_0')
-        v = layers.fc(input=values,
-                      size=d_value * n_head,
-                      num_flatten_dims=2,
-                      param_attr=fluid.ParamAttr(
-                          name=name + '_value_fc.w_0',
-                          initializer=param_initializer),
-                      bias_attr=name + '_value_fc.b_0')
-        return q, k, v
-
-    def __split_heads(x, n_head):
-        """
-        Reshape the last dimension of inpunt tensor x so that it becomes two
-        dimensions and then transpose. Specifically, input a tensor with shape
-        [bs, max_sequence_length, n_head * hidden_dim] then output a tensor
-        with shape [bs, n_head, max_sequence_length, hidden_dim].
-        """
-        hidden_size = x.shape[-1]
-        # The value 0 in shape attr means copying the corresponding dimension
-        # size of the input as the output dimension size.
-        reshaped = layers.reshape(
-            x=x, shape=[0, 0, n_head, hidden_size // n_head], inplace=True)
-
-        # permuate the dimensions into:
-        # [batch_size, n_head, max_sequence_len, hidden_size_per_head]
-        return layers.transpose(x=reshaped, perm=[0, 2, 1, 3])
-
-    def __combine_heads(x):
-        """
-        Transpose and then reshape the last two dimensions of inpunt tensor x
-        so that it becomes one dimension, which is reverse to __split_heads.
-        """
-        if len(x.shape) == 3: return x
-        if len(x.shape) != 4:
-            raise ValueError("Input(x) should be a 4-D Tensor.")
-
-        trans_x = layers.transpose(x, perm=[0, 2, 1, 3])
-        # The value 0 in shape attr means copying the corresponding dimension
-        # size of the input as the output dimension size.
-        return layers.reshape(
-            x=trans_x,
-            shape=[0, 0, trans_x.shape[2] * trans_x.shape[3]],
-            inplace=True)
-
-    def scaled_dot_product_attention(q, k, v, attn_bias, d_key, dropout_rate):
-        """
-        Scaled Dot-Product Attention
-        """
-        scaled_q = layers.scale(x=q, scale=d_key**-0.5)
-        product = layers.matmul(x=scaled_q, y=k, transpose_y=True)
-        if attn_bias:
-            product += attn_bias
-        weights = layers.softmax(product)
-        if dropout_rate:
-            weights = layers.dropout(
-                weights,
-                dropout_prob=dropout_rate,
-                dropout_implementation="upscale_in_train",
-                is_test=False)
-        out = layers.matmul(weights, v)
-        return out
-
-    q, k, v = __compute_qkv(queries, keys, values, n_head, d_key, d_value)
-
-    if cache is not None:  # use cache and concat time steps
-        # Since the inplace reshape in __split_heads changes the shape of k and
-        # v, which is the cache input for next time step, reshape the cache
-        # input from the previous time step first.
-        k = cache["k"] = layers.concat(
-            [layers.reshape(
-                cache["k"], shape=[0, 0, d_model]), k], axis=1)
-        v = cache["v"] = layers.concat(
-            [layers.reshape(
-                cache["v"], shape=[0, 0, d_model]), v], axis=1)
-
-    q = __split_heads(q, n_head)
-    k = __split_heads(k, n_head)
-    v = __split_heads(v, n_head)
-
-    ctx_multiheads = scaled_dot_product_attention(q, k, v, attn_bias, d_key,
-                                                  dropout_rate)
-
-    out = __combine_heads(ctx_multiheads)
-
-    # Project back to the model size.
-    proj_out = layers.fc(input=out,
-                         size=d_model,
-                         num_flatten_dims=2,
-                         param_attr=fluid.ParamAttr(
-                             name=name + '_output_fc.w_0',
-                             initializer=param_initializer),
-                         bias_attr=name + '_output_fc.b_0')
-    return proj_out
-
-
-def positionwise_feed_forward(x,
-                              d_inner_hid,
-                              d_hid,
-                              dropout_rate,
-                              hidden_act,
-                              param_initializer=None,
-                              name='ffn'):
-    """
-    Position-wise Feed-Forward Networks.
-    This module consists of two linear transformations with a ReLU activation
-    in between, which is applied to each position separately and identically.
-    """
-    hidden = layers.fc(input=x,
-                       size=d_inner_hid,
-                       num_flatten_dims=2,
-                       act=hidden_act,
-                       param_attr=fluid.ParamAttr(
-                           name=name + '_fc_0.w_0',
-                           initializer=param_initializer),
-                       bias_attr=name + '_fc_0.b_0')
-    if dropout_rate:
-        hidden = layers.dropout(
-            hidden,
-            dropout_prob=dropout_rate,
-            dropout_implementation="upscale_in_train",
-            is_test=False)
-    out = layers.fc(input=hidden,
-                    size=d_hid,
-                    num_flatten_dims=2,
-                    param_attr=fluid.ParamAttr(
-                        name=name + '_fc_1.w_0', initializer=param_initializer),
-                    bias_attr=name + '_fc_1.b_0')
-    return out
-
-
-def pre_post_process_layer(prev_out, out, process_cmd, dropout_rate=0.,
-                           name=''):
-    """
-    Add residual connection, layer normalization and droput to the out tensor
-    optionally according to the value of process_cmd.
-    This will be used before or after multi-head attention and position-wise
-    feed-forward networks.
-    """
-    for cmd in process_cmd:
-        if cmd == "a":  # add residual connection
-            out = out + prev_out if prev_out else out
-        elif cmd == "n":  # add layer normalization
-            out_dtype = out.dtype
-            if out_dtype == fluid.core.VarDesc.VarType.FP16:
-                out = layers.cast(x=out, dtype="float32")
-            out = layer_norm(
-                out,
-                begin_norm_axis=len(out.shape) - 1,
-                param_attr=fluid.ParamAttr(
-                    name=name + '_layer_norm_scale',
-                    initializer=fluid.initializer.Constant(1.)),
-                bias_attr=fluid.ParamAttr(
-                    name=name + '_layer_norm_bias',
-                    initializer=fluid.initializer.Constant(0.)))
-            if out_dtype == fluid.core.VarDesc.VarType.FP16:
-                out = layers.cast(x=out, dtype="float16")
-        elif cmd == "d":  # add dropout
-            if dropout_rate:
-                out = layers.dropout(
-                    out,
-                    dropout_prob=dropout_rate,
-                    dropout_implementation="upscale_in_train",
-                    is_test=False)
-    return out
-
-
-pre_process_layer = partial(pre_post_process_layer, None)
-post_process_layer = pre_post_process_layer
-
-
-def encoder_layer(enc_input,
-                  attn_bias,
-                  n_head,
-                  d_key,
-                  d_value,
-                  d_model,
-                  d_inner_hid,
-                  prepostprocess_dropout,
-                  attention_dropout,
-                  relu_dropout,
-                  hidden_act,
-                  preprocess_cmd="n",
-                  postprocess_cmd="da",
-                  param_initializer=None,
-                  name=''):
-    """The encoder layers that can be stacked to form a deep encoder.
-    This module consits of a multi-head (self) attention followed by
-    position-wise feed-forward networks and both the two components companied
-    with the post_process_layer to add residual connection, layer normalization
-    and droput.
-    """
-    attn_output = multi_head_attention(
-        pre_process_layer(
-            enc_input,
-            preprocess_cmd,
-            prepostprocess_dropout,
-            name=name + '_pre_att'),
-        None,
-        None,
-        attn_bias,
-        d_key,
-        d_value,
-        d_model,
-        n_head,
-        attention_dropout,
-        param_initializer=param_initializer,
-        name=name + '_multi_head_att')
-    attn_output = post_process_layer(
-        enc_input,
-        attn_output,
-        postprocess_cmd,
-        prepostprocess_dropout,
-        name=name + '_post_att')
-    ffd_output = positionwise_feed_forward(
-        pre_process_layer(
-            attn_output,
-            preprocess_cmd,
-            prepostprocess_dropout,
-            name=name + '_pre_ffn'),
-        d_inner_hid,
-        d_model,
-        relu_dropout,
-        hidden_act,
-        param_initializer=param_initializer,
-        name=name + '_ffn')
-    return post_process_layer(
-        attn_output,
-        ffd_output,
-        postprocess_cmd,
-        prepostprocess_dropout,
-        name=name + '_post_ffn')
-
-
-def encoder(enc_input,
-            attn_bias,
-            n_layer,
-            n_head,
-            d_key,
-            d_value,
-            d_model,
-            d_inner_hid,
-            prepostprocess_dropout,
-            attention_dropout,
-            relu_dropout,
-            hidden_act,
-            preprocess_cmd="n",
-            postprocess_cmd="da",
-            param_initializer=None,
-            name=''):
-    """
-    The encoder is composed of a stack of identical layers returned by calling
-    encoder_layer.
-    """
-    for i in range(n_layer):
-        enc_output = encoder_layer(
-            enc_input,
-            attn_bias,
-            n_head,
-            d_key,
-            d_value,
-            d_model,
-            d_inner_hid,
-            prepostprocess_dropout,
-            attention_dropout,
-            relu_dropout,
-            hidden_act,
-            preprocess_cmd,
-            postprocess_cmd,
-            param_initializer=param_initializer,
-            name=name + '_layer_' + str(i))
-        enc_input = enc_output
-    enc_output = pre_process_layer(
-        enc_output, preprocess_cmd, prepostprocess_dropout, name="post_encoder")
-
-    return enc_output

demo/demo2/\

+
+from paddle import fluid
+from paddle.fluid import layers
+from paddlepalm.distribute import gpu_dev_count, cpu_dev_count
+from paddlepalm import Trainer
+from paddlepalm.utils import reader_helper
+import time
+
+dev_count = 1 if gpu_dev_count <= 1 else gpu_dev_count
+VERBOSE=False
+
+
+class MultiHeadTrainer(Trainer):
+    
+    def __init__(self, trainers, reuse_flags=None):
+        if reuse_flags is not None:
+            assert len(reuse_flags) == len(trainers)
+
+        self._trainers = trainers
+
+        self._train_init = False
+        self._predict_init = False
+        self._feeded_var_names = None
+        self._cur_train_step = 0
+        self._target_vars = None
+
+        self._inputname_to_varname = {}
+        self._pred_input_name_list = []
+        self._pred_input_varname_list = []
+        self._pred_fetch_name_list = []
+        self._pred_fetch_var_list = []
+
+        self._exe = None
+
+        self._save_protocol = {
+            'input_names': 'self._pred_input_name_list',
+            'input_varnames': 'self._pred_input_varname_list',
+            'fetch_list': 'self._pred_fetch_name_list'}
+
+        self._check_save = lambda: False
+        for t in self._trainers:
+            t._set_multitask()
+
+    def build_forward(self, backbone, heads):
+
+        if isinstance(heads, list):
+            head_dict = {k.name: v for k,v in zip(self._trainers, heads)}
+        elif isinstance(heads, dict):
+            head_dict = heads
+        else:
+            raise ValueError()
+
+        num_heads = len(self._trainers)
+        assert len(head_dict) == num_heads
+
+        for t in self._trainers:
+            assert t.name in head_dict, "expected: {}, exists: {}".format(t.name, head_dict.keys())
+        
+        train_prog = fluid.Program()
+        train_init_prog = fluid.Program()
+        self._train_prog = train_prog
+        self._train_init_prog = train_init_prog
+
+        def get_loss(i):
+            head = head_dict[self._trainers[i].name]
+            # loss_var = self._trainers[i].build_forward(backbone, head, train_prog, train_init_prog)
+            loss_var = self._trainers[i].build_forward(backbone, head)
+            return loss_var
+      
+        # task_fns = {}
+        # for i in range(num_heads):
+
+        #     def task_loss():
+        #         task_id = i
+        #         return lambda: get_loss(task_id)
+
+        #     task_fns[i] = task_loss()
+
+
+        # task_fns = {i: lambda: get_loss(i) for i in range(num_heads)}
+        task_fns = {i: lambda i=i: get_loss(i) for i in range(num_heads)}
+
+        with fluid.program_guard(train_prog, train_init_prog):
+            task_id_var = fluid.data(name="__task_id",shape=[1],dtype='int64')
+            task_id_var += 0
+            # task_id_var = fluid.layers.fill_constant(shape=[1],dtype='int64', value=1)
+            # print(task_id_var.name)
+
+            loss_var = layers.switch_case(
+                branch_index=task_id_var,
+                branch_fns=task_fns
+            )
+        self._task_id_var = task_id_var
+        self._loss_var = loss_var
+        self._fetch_list = [loss_var.name]
+        for b in train_prog.blocks:
+            for var in b.vars:
+                pass
+                # if 'task_id' in var:
+                #     print(var)
+                #     exit()
+                # print(var)
+        return loss_var
+
+    def fit_readers(self, reader_dict):
+        raise NotImplementedError()
+
+    def fit_readers_with_mixratio(self, readers, sampling_reference, num_epochs, phase='train'):
+
+        if isinstance(readers, list):
+            reader_dict = {k.name: v for k,v in zip(self._trainers, readers)}
+        elif isinstance(readers, dict):
+            reader_dict = readers
+        else:
+            raise ValueError()
+        
+        num_heads = len(self._trainers)
+        assert len(reader_dict) == num_heads
+
+        trainer_dict = {t.name: t for t in self._trainers}
+        assert sampling_reference in trainer_dict
+
+        trainer_dict[sampling_reference].fit_reader(reader_dict[sampling_reference])
+        base_steps_pur_epoch = trainer_dict[sampling_reference]._steps_pur_epoch
+
+        input_names = []
+        name_to_pos = []
+        joint_shape_and_dtypes = []
+        iterators = []
+        prefixes = []
+        mrs = []
+        net_inputs = []
+        global_steps = 0
+        for t in self._trainers:
+            assert t.name in reader_dict
+            assert reader_dict[t.name].num_epochs is None, "{}: num_epochs is not None. \
+                To run with multi-head mode, num_epochs of each Trainer should be set as None.".format(t.name)
+            # print(num_epochs, t.mix_ratio, base_steps_pur_epoch)
+            max_train_steps = int(num_epochs * t.mix_ratio * base_steps_pur_epoch)
+            if not t._as_auxilary:
+                print('{}: expected train steps {}.'.format(t.name, max_train_steps))
+            global_steps += max_train_steps
+            if t.name != sampling_reference:
+                t.fit_reader(reader_dict[t.name])
+            net_inputs.append(t._net_inputs)
+            prefixes.append(t.name)
+            mrs.append(t.mix_ratio)
+            iterators.append(t._raw_iterator_fn())
+            input_names.append(t._input_names)
+            name_to_pos.append(t._name_to_position)
+            joint_shape_and_dtypes.append(t._shape_and_dtypes)
+
+        print('Estimated overall train steps {}.'.format(global_steps))
+        self._overall_train_steps = global_steps
+
+        iterator_fn = reader_helper.create_multihead_iterator_fn(iterators, prefixes, joint_shape_and_dtypes, \
+            mrs, input_names, name_to_pos, dev_count=dev_count)
+        feed_batch_process_fn = reader_helper.create_feed_batch_process_fn(net_inputs)
+
+        if gpu_dev_count > 1:
+            distribute_feeder_fn = data_feeder(iterator_fn, feed_batch_process_fn)
+        else:
+            distribute_feeder_fn = iterator_fn
+
+        if phase == 'train':
+            self._train_reader = distribute_feeder_fn()
+            self._feed_batch_process_fn = feed_batch_process_fn
+        elif phase == 'predict':
+            self._predict_reader = distribute_feeder_fn()
+            self._pred_feed_batch_process_fn = feed_batch_process_fn
+        
+    def train(self, save_path=None, save_steps=None, save_type='ckpt', print_steps=5):
+        iterator = self._train_reader
+        self._distribute_train_prog = fluid.CompiledProgram(self._train_prog).with_data_parallel(loss_name=self._loss_var.name)
+
+        save_type = save_type.split(',')
+        if 'predict' in save_type:
+            assert self._pred_head is not None, "Predict head not found! You should build_predict_head first if you want to save predict model."
+            assert save_path is not None and save_steps is not None, 'save_path and save_steps is required to save model.'
+            save_predict = True
+            if not os.path.exists(save_path):
+                os.makedirs(save_path)
+        else:
+            save_predict = False
+
+        if 'ckpt' in save_type:
+            if save_path is not None and save_steps is not None:
+                save_ckpt = True
+                if not os.path.exists(save_path):
+                    os.makedirs(save_path)
+            else:
+                "WARNING: save_path or save_steps is not set, model will not be saved during training."
+                save_ckpt = False
+        else:
+            save_ckpt = False
+
+        time_begin = time.time()
+        for feed in iterator:
+            # batch, task_id = feed
+            rt_outputs, task_id = self.train_one_step(feed)
+
+            task_rt_outputs = {k[len(self._trainers[task_id].name+'.'):]: v for k,v in rt_outputs.items() if k.startswith(self._trainers[task_id].name+'.')}
+            self._task_head.batch_postprocess(task_rt_outputs)
+
+            if print_steps > 0 and self._cur_train_step % print_steps == 0:
+                loss = rt_outputs[self._trainers[task_id].name+'.loss']
+                loss = np.mean(np.squeeze(loss)).tolist()
+
+                time_end = time.time()
+                time_cost = time_end - time_begin
+
+                print("global step: {}, step {}/{} (epoch {}), loss: {:.3f}, speed: {:.2f} steps/s".format(
+                       (self._cur_train_step, self._trainers[task_id]._cur_train_step-1) % self._trainers[task_id]._steps_pur_epoch + 1, self._trainers[task_id]._steps_pur_epoch, self._trainers[task_id]._cur_train_epoch,
+                       loss, print_steps / time_cost))
+                time_begin = time.time()
+
+            self._check_save()
+
+            # if cur_task.train_finish and cur_task.cur_train_step + cur_task.cur_train_epoch * cur_task.steps_pur_epoch == cur_task.expected_train_steps:
+            #     print(cur_task.name+': train finished!')
+            #     cur_task.save()
+
+            # if (save_predict or save_ckpt) and self._cur_train_step % save_steps == 0:
+            #     if save_predict:
+            #         self.save(save_path, suffix='pred.step'+str(self._cur_train_step))
+            #     if save_ckpt:
+            #         fluid.io.save_persistables(self._exe, os.path.join(save_path, 'ckpt.step'+str(self._cur_train_step)), self._train_prog)
+            #         print('checkpoint has been saved at '+os.path.join(save_path, 'ckpt.step'+str(self._cur_train_step)))
+
+            if self._num_epochs is None and self._cur_train_step == self._steps_pur_epoch:
+                break
+
+
+    def train_one_step(self, batch):
+
+        if dev_count > 1:
+            assert isinstance(batch, list)
+            # for f in batch:
+            #     f['branch'] = np.array([task_id], dtype='int64')
+            task_id = batch[0]['__task_id'][0]
+        else:
+            assert isinstance(batch, dict)
+            task_id = batch['__task_id'][0]
+            # batch['branch'] = np.array([task_id], dtype='int64')
+            
+        # feed = self._trainers[task_id].get_one_batch()
+        print(batch)
+        print(self._distribute_train_prog)
+        rt_outputs = self._trainers[task_id].train_one_step(batch, self._exe, self._distribute_train_prog, self._fetch_list)
+
+        self._cur_train_steps += 1
+        return rt_outputs, task_id
+        
+        # if dev_count > 1:
+        #     # feed, mask, task_id = batch
+        #     for f in feed:
+        #         f['branch'] = np.array([task_id], dtype='int64')
+        #     rt_outputs = self.exe.run(self._distribute_train_prog, feed=feed, fetch_list=self._trainers[task_id]._fetch_list)
+        #     num_fakes = decode_fake(len(rt_outputs[0]), mask, self._trainers[task_id]._batch_size)
+        #     for _ in range(num_fakes):
+        #         for item in rt_outputs:
+        #             item.pop()
+        # else:
+        #     feed, task_id = batch
+        #     feed['branch'] = np.array([task_id], dtype='int64')
+        #     rt_outputs = self._exe.run(self._distribute_train_prog, feed=feed, fetch_list=self._trainers[task_id]._fetch_list)
+
+    def predict_one_batch(self, batch):
+        raise NotImplementedError()
+
+    def predict(self, output_dir=None, print_steps=1000):
+        raise NotImplementedError()
+
+    @property
+    def overall_train_steps(self):
+        return self._overall_train_steps

demo/demo2/data/cls4mrqa/dev.tsv

+label	text_a
+1	when was the last time the san antonio spurs missed the playoffshave only missed the playoffs four times since entering the NBA ; they have not missed the playoffs in the 20 seasons since Tim Duncan was drafted by the Spurs in 1997 . With their 50th win in the 2016 -- 17 season , the Spurs extended their record for most consecutive 50 - win seasons to 18 ( the Spurs did not
+0	the creation of the federal reserve system was an attempt toReserve System ( also known as the Federal Reserve or simply the Fed ) is the central banking system of the United States of America . Over the years , events such as the Great Depression in the 1930s and the Great Recession during the 2000s have led to the expansion of the
+2	group f / 64 was a major backlash against the earlier photographic movement off / 64 was formed , Edward Weston went to a meeting of the John Reed Club , which was founded to support Marxist artists and writers . These circumstances not only helped set up the situation in which a group
+0	Bessarabia eventually became under the control of which country?city of Vilnius – its historical capital, which was under Polish control during the inter-war
+0	Iran's inflation led to what in 1975-1976?the economy of Iran was flooded with foreign currency, which caused inflation. By 1974, the economy of Iran was experiencing double digit inflation, and despite many large projects to modernize the country, corruption was rampant and caused large
+1	How many steam warships did Japan have in 1867?Yokosuka and Nagasaki. By the end of the Tokugawa shogunate in 1867, the Japanese navy of the shogun already possessed eight western-style steam warships around the flagship Kaiyō Maru, which were used against pro-imperial forces during the Boshin war, under the command
+0	How many people were inside?f former NFL head coach Dan Reeves, suffered a broken back. DeCamillis was seen on a stretcher wearing a neck brace. A line of heavy thunderstorms was moving through the Dallas area at the time, he said, but no other damage to buildings was reported, said Mike Adams, a dispatcher for the Irving, Texas, fire department. Watch the roof collapse on players, coaches » Arnold Payne, a photographer for WFAA, was shooting the Cowboys' practice session when rain began falling "tremendously hard." "I noticed the walls started to waver ... and then I noticed that the lights that were hanging from the ceiling started to sway, and it wouldn't stop," Payne told CNN. Shortly after that, he said, "It was as if someone took a stick pin and hit a balloon." Watch Payne describe being inside when structure collpased » Payne said
+0	Ishita Dutta is the sister of an actress who is typically cast in what genre of movies?he suspense thriller film "Drishyam" (2015) and the Hindi soap opera "Ek Ghar Banaunga", that aired on Star Plus. She is the younger sister of actress Tanushree Dutta. Dutta is the recipient of Femina Miss India Universe title in 2004. During the same year
+3	when did the the civil war start and end/Th> </Tr> <Tr> <Td> <P> 110,000 + killed in action / died of wounds 230,000 + accident / disease deaths 25,000 -- 30,000 died in Confederate prisons </P> <P> 365,000 + total dead
+1	What has Pakistan told phone companies?Islamabad, Pakistan (CNN) -- Under heavy criticism for a telling cell phone carriers to ban certain words in text messages, the Pakistan Telecommunication Authority went into damage control mode Wednesday. PTA spokesman Mohammed Younis Wednesday denied the existence of the plan, which has met with derision from mobile phone users in the country. "If at all we finally decide to
+0	What did Bush say the proposal was to a proposal he vetoed before?(CNN) -- President Bush vetoed an expansion of the federally funded, state-run health insurance program for poor children for a second time Wednesday, telling Congress the bill "moves our country's health care system in the wrong direction." In his veto message, President Bush calls on Congress to extend funding for the current program. "Because the Congress has chosen to send me an essentially identical bill that has the same problems as the flawed bill I previously vetoed, I must veto this legislation, too," he said in a statement released by the White House. The bill would
+0	Where did the football team that Bob Simmons coached from 1995 to 2000 play their home games?Cowboys football team [SEP] The 1998 Oklahoma State Cowboys football team represented the Oklahoma State University during the 1998 NCAA Division I-A football season. They participated as members of the Big 12 Conference in the South Division. They were coached by head coach Bob Simmons. [PAR] [TLE] Bob Simmons (American football coach) [SEP] Bob
+2	What anniversary was recently celebrated in Iran?us to move our policy in a new direction," Obama said. "So there are going to be a set of objectives that we have in these conversations, but I think that there's the possibility at least of a relationship of mutual respect and progress." The United States and Iran have not had diplomatic relations since 1979. During that year, the Shah of Iran was forced to flee the country and the Ayatollah Khomeini took power. Later that year, Iranian students took over and seized hostages at the U.S. Embassy. Relations have been cut since then. U.S. President George W. Bush labeled Iran as a member of the "axis of evil" after the Sept. 11, 2001 attacks. Iran celebrated the 30th anniversary of the revolution Tuesday with crowds chanting "Death to America." Watch the parade in Iran » Tensions have rippled over issues such as Iran's nuclear program, Israel, and Iraq, and have been aggravated since the outspoken Ahmadinejad came to power in 2005. Western
+1	Which Italian composer did George Balanchine add in 1976?[PAR] [TLE] Arcangelo Corelli [SEP] Arcangelo Corelli ( ; 17 February 1653 – 8 January 1713) was an Italian violinist and composer of the Baroque era. His music
+0	Will the playstation 4 be announced?a new system sometime in the next five years, of course. Sony continued to sell the PlayStation 2 system and games years after the PlayStation 3 debuted in stores. For Sony's next console, the company will not deploy a streaming delivery system like OnLive, or fully cut out disc retailers like Best Buy and GameStop, Hirai said. While Sony has increased the number of games and other media available for download or streaming through its networks, most people cannot be expected to frequently download several gigabytes worth of data, which can be a time-consuming process, he said. Sony Computer Entertainment president Andrew House said earlier that Sony is not planning to discuss a new console, the website ComputerAndVideogames.com reported on Monday.
+1	How many children were the Americans trying to kidnap out of Haiti?Port-au-Prince, Haiti (CNN) -- A Haitian attorney representing 10 Americans charged with kidnapping for trying to take 33 children out of Haiti told CNN Sunday he has resigned. Edwin Coq said he had quit as a lawyer for the Americans. It wasn't immediately clear who would replace him. "I know that they have been looking at other lawyers," said Phyllis Allison, mother of one of those detained, Jim Allen. "They don't know what to do." The 10 missionaries, including group leader Laura Silsby, were charged Thursday with kidnapping children and criminal association. Coq had said that court hearings would be held Monday
+0	who kills tree gelbman in happy death dayTree convinces Carter of her predicament by showing that she holds foreknowledge of the day 's events . Tree admits to Carter she does n't like who
+0	What will no person be denied the enjoyment of in Georgia based on their religious principles?amended as follows: "Article IV. Section 10. No person within this state shall, upon any pretense, be deprived of the inestimable privilege of worshipping God in any
+0	who came up with the idea of footballpass . The popularity of college football grew as it became the dominant version of the sport in the United States for the first half of the 20th century . Bowl games , a college football tradition , attracted a national audience for college
+0	what is the name of the female smurfbefore the smurflings created Sassette , Smurfette was the only female smurf in the Smurf Village .
+3	Who contributed to the American studies programs at Yale and University of Wyoming?struggle. Norman Holmes Pearson, who worked for the Office of Strategic Studies in London during World War II, returned to Yale and headed the new American studies program, in which scholarship quickly became an instrument of promoting
+0	What is the group's former name that now has an office with the Chief Actuary besides the Social Security Administration?Office of the Chief Actuary [SEP] The Office of the Chief Actuary is a government agency that has responsibility for actuarial estimates regarding social welfare programs. In Canada, the Office of the Chief Actuary works with the Canada Pension Plan and the Old Age Security Program. In the United States, both the Social Security Administration and the Centers for Medicare and Medicaid Services have an Office of the Chief Actuary that deals with Social Security and Medicare, respectively. A similar agency in the United Kingdom is called the Government Actuary's Department
+0	The actor that playes Han Solo in the "Star Wars" film series stars with Blake Lively and Michiel Huisman in a film directed by who?about a woman who stops aging after an accident at the age of 29. Mills Goodloe and Salvador Paskowitz. The film stars Blake Lively, Michiel Huisman, Kathy Baker, Amanda Crew, Harrison Ford, and Ellen Burstyn. The film was theatrically released on April 24, 2015 by Lionsgate. [PAR] [TLE] Harrison Ford [SEP] Harrison Ford
+0	What historically black university's men's basketball coach was formerly head coach at Virginia Tech?well as an 1890 Historically Black Land-Grant University. The University is a member-school of the Thurgood Marshall College Fund. He was also the head coach at Virginia Tech, Tennessee
+0	what year did syracuse win the ncaa tournament. Their combined record is 67 -- 39 .
+1	where do i get chips at a casino<P> Money is exchanged for tokens in a casino at the casino cage , at the gaming tables , or at a cashier station . The tokens are
+0	when was the winter fuel payment first introducedheating over the winter months .
+0	Trophy hunting can include areas which would likely be unsuitable for what other types of ecotourism?study states that less than 3% of a trophy hunters' expenditures reach the local level, meaning that the economic incentive and benefit is "minimal, particularly when we consider the vast areas of
+1	In simple language, what are the interconnections in an embedding matrix?Since it was quite easy to stack interconnections (wires) inside the embedding matrix, the approach allowed designers to forget completely about the routing of wires (usually a time-consuming operation of PCB design): Anywhere the designer needs a connection, the machine will draw a wire in straight line from one location/pin
+2	rho has been to the most all star games in baseballn4 </Li> <Li> Stan Musial 24 </Li>
+0	In 1169, Ireland was invaded by which people?High King to ensure the terms of the Treaty of Windsor led Henry II, as King of England, to rule as effective monarch under the title of Lord of Ireland. This title was granted to his younger son but when Henry's heir unexpectedly died the title of King of England and Lord of Ireland became entwined in one
+1	What year did a biracial Populist fusion gain the Governors office?to the legislature and governor's office, but the Populists attracted voters displeased with them. In 1896 a biracial, Populist-Republican Fusionist coalition gained the governor's office. The Democrats regained control of the legislature
+1	nearest metro station to majnu ka tilla delhiRing Road of Delhi . It is at a walkable distance from ISBT Kashmere Gate . It is approachable through the Kashmeri Gate station of the Delhi Metro , lies on both the Red ( Dilshad Garden - Rithala ) and Yellow Lines ( Samaypur
+3	where is california located in the region of the united states<P> California is a U.S. state in the Pacific Region of the United States . With 39.5 million residents , California is the most populous state in the United States and the third largest by area . The
+1	when did the baptist church start in americacoworker for religious freedom , are variously credited as founding the earliest Baptist church in North America . In 1639 , Williams established a Baptist church in Providence , Rhode Island , and Clarke began a Baptist church in
+0	where was the first capital of the united states locatedpassed to pave the way for a permanent capital . The decision to locate the capital was contentious , but Alexander Hamilton helped broker a compromise in which the federal government would take on war debt incurred during the American Revolutionary War , in exchange for support from northern states for locating the capital along the Potomac
+0	What will new regulations will reduce?products off of the consumer market," said Michael Fry, director of conservation advocacy for the American Bird Conservancy. "By putting these restrictions in place, they are allowing a compromise to be made between themselves and organizations who have been working on this problem for a long time." The EPA's new measures, which were handed down Thursday, require that rat poisons be kept in bait stations above ground and in containers that meet agency standards. Loose bait, such as pellets, and the four most hazardous types of pesticides, known as "second-generation anticoagulants," will no longer be sold for personal use. Under the new restrictions, only farmers, livestock owners and certified rodent control employees will be allowed to purchase rat poison in bulk. Bags larger than 8 pounds will no longer be sold at hardware and home-improvement stores. Children who come into contact
+0	who played lois lane in the man of steelmixture of toughness and vulnerability , but Peter Bradshaw thought that the character was `` sketchily conceived '' and criticized her lack of chemistry with Cavill . Even so , the film earned over $660 million to become one of her biggest box
+0	What year did the writer of the 1968 novel "The Iron Man" become Poet Laurete?Giant is a 1999 American animated science-fiction comedy-drama action film using both traditional animation and computer animation, produced by and directed by Brad Bird in his directorial debut. It is based on the 1968 novel "The Iron Man" by Ted Hughes (which was published in the United States as "The Iron Giant") and was scripted by Tim McCanlies from a story treatment by Bird. The film stars the voices of Eli Marienthal,
+2	The conquest of Nice was an effort by Suleiman and what French king?allies. A month prior to the siege of Nice, France supported the Ottomans with an artillery unit during the 1543 Ottoman conquest of Esztergom in northern Hungary. After further advances by the Turks, the Habsburg ruler Ferdinand officially recognized Ottoman ascendancy in Hungary in
+0	when was the vaccine receivedfor swine flu, also known as 2009 H1N1, using reverse genetics, he said. "Suitable viruses will hopefully be sent to manufacturers by end of next week," Skinner wrote. Once that happens, vaccine makers will tweak the virus and have "pilot lots" of vaccine ready to be tested by mid- to late June. Several thousand cases have been reported
+1	What is the nationality of the actor who costarred with Matt LeBlanc in "All the Queen's Men"?n approximate -99.92% return. [PAR] [TLE] Eddie Izzard [SEP] Edward John "Eddie" Izzard ( ; born 7 February 1962) is an English stand-up comedian, actor, writer and political activist. His comedic style takes the form of rambling, whimsical monologue, and self-referential pantomime. He
+0	What sickened thousands of children?executives detained, a local official said, according to Xinhua, Initial tests showed more than 1,300 children in the Hunan province town of Wenping have excessive lead in their blood from the Wugang Manganese Smelting Plant. A second round of testing has been ordered to confirm the results. The plant opened in May 2008 without gaining the approval of the local environment protection bureau, said Huang Wenbin, a deputy environment chief in Wugang City, Xinhua reported. The plant was within 500 meters (about a quarter mile) of three schools. The
+0	What percentage of the population are the Kpelle?are descendants of African American and West Indian, mostly Barbadian settlers, make up 2.5%. Congo people, descendants of repatriated Congo and Afro-Caribbean
+1	Amount of people left homeless?86 dead, the state news agency said. About 30 people are missing, the official news agency Agencia Brasil said, citing civil defense officials. Earlier reports had indicated as many as 100 people were dead. In addition, more than 54,000 residents have been left homeless, and another 1.5 million have been affected by the heavy rains, the state news agency reported. Brazilian President Luiz Inacio Lula da Silva announced he will release nearly 700 million reais ($350 million)
+2	What other countries were in disagreement with the United Nations decision on Burma ?that strongly called upon the government of Myanmar to end its systematic violations of human rights. In January 2007, Russia and China vetoed a
+0	Besides Barcelona and Real Madrid, what other team has remained in the Primera Division?first football club to win six out of six competitions in a single year, completing the sextuple in also winning the Spanish Super Cup, UEFA Super Cup and FIFA Club World Cup. In 2011, the club became
+0	William Frederick Truax, is a former professional American football tight end in the National Football League (NFL) from 1964 to 1973 for the Los Angeles Rams and the Dallas Cowboys, following the 1970 NFL season, Truax was traded by the Rams to the Cowboys for wide receiver Lance Rentzel, a former American football flanker, in which organization?in New Orleans and college football at Louisiana State University and was drafted in the second round of the 1964 NFL draft. Following the 1970 NFL season, Truax was traded by the Rams to the Cowboys for wide receiver Lance Rentzel. He was part of the Cowboys' Super Bowl VI championship team in 1971. He played
+3	What year did Chopin learn that the uprising in Warsaw was crushed?enlist. Chopin, now alone in Vienna, was nostalgic for his homeland, and wrote to a friend, "I curse the moment of my departure." When in September 1831 he learned, while travelling from Vienna to Paris, that the uprising had been crushed, he expressed his anguish in the pages of his private journal: "Oh
+1	where do they make money in washington dc; all coinage is produced by the United States Mint . With production facilities in Washington , DC , and Fort Worth , Texas , the Bureau of Engraving and Printing is the largest producer of government security documents in the United States . </P>
+0	What did a researcher compare this process to?which makes it one of the highest rates of maternal mortality in the Americas. In wealthy developed nations, only nine women die for every 100,000 births. The five main causes of pregnancy-related deaths in Peru are hemorrhage, pre-eclampsia, infection, complications following abortion and obstructed birth, according to Peru's Ministry of Health figures. Amnesty's Peru researcher Nuria Garcia said, in a written statement: "The rates of maternal mortality in Peru are scandalous. The fact that so many women are dying from preventable causes is a human rights violation. "The Peruvian state is simply ignoring
+0	How many containers can Longtan Containers Port Area handle?Port of Nanjing is the largest inland port in China, with annual cargo tonnage reached 191,970,000 t in 2012. The port area is 98 kilometres (61 mi) in length and has 64 berths
+0	The 2011 New York City Marathon was sponsored by which Dutch multinational banking corporation?are retail banking, direct banking, commercial banking, investment banking, asset management, and insurance services. ING is an abbreviation for "Internationale Nederlanden Groep " (English: International Netherlands Group). [PAR] [TLE] 2011 New York City Marathon [SEP] The 42nd New York City Marathon took
+0	What is human flourishing?it does not involve believing that human nature is purely good or that all people can live up to the Humanist ideals without help. If anything, there is recognition that living up to one's potential is hard
+0	What was the result of Dida appealto play in next month's Champions League match at Shakhtar Donetsk after partially winning his appeal to UEFA against a two-match ban. Dida has had one game of his two-match ban suspended for a year following an appeal to UEFA. Brazilian Dida was also fined 60,000 Swiss francs by European football's ruling body following an incident involving a supporter during the Champions clash against Celtic in Scotland on October 3. The 34-year-old Brazilian was initially banned for two games for his theatrics following a Celtic fan's encroachment onto the pitch during the 2-1 defeat at Celtic
+1	What is more plentiful in capital projects?generates economic distortion in the public sector by diverting public investment into capital projects where bribes and kickbacks are more plentiful. Officials may increase the technical complexity of public sector projects to conceal or
+0	where were band greeted with cheers?the United States for a show in Stamford, Connecticut, on Tuesday, after they have "a few days off to recuperate," Robinson said. The trio was the opening act for Nelson until they were loudly booed in Toronto, a day after the actor-musician's bizarre interview with a CBC radio host. Ironically, the comments that offended Canadians included Thornton's assessment that they were "very reserved" and "it doesn't matter what you say to them." "It's mashed potatoes with no gravy," Thornton told CBC host Jian Ghomeshi. "We tend to play places where people throw things at each other and here they just sort of sit there," he said. Watch Thornton's interview » The audience at Thursday night's show in Toronto loudly booed the Boxmasters, with some shouts of "Here comes the gravy!" The Toronto Star newspaper reported. Thornton's remarks about
+0	What do Mexicans call Mexico City?the Federal District in Spanish: D.F., which is read "De-Efe"). They are formally called capitalinos (in reference to the city being the capital of the country), but "[p]erhaps because capitalino is the
+0	where does lock stock and barrel come fromindividual components one at a time . One craftsman made the `` lock '' which would have been a `` match lock '' , `` wheel lock '' , `` flint lock '' etc .
+1	who has the power to establish a prison system<P> The Federal Bureau of Prisons ( BOP ) is a United States federal law enforcement agency . A subdivision of
+0	what are south americas only 2 landlocked countriessuch countries , including five partially recognised states .

demo/demo2/data/cls4mrqa/train.tsv

+label	text_a
+1	when was the last time the san antonio spurs missed the playoffshave only missed the playoffs four times since entering the NBA ; they have not missed the playoffs in the 20 seasons since Tim Duncan was drafted by the Spurs in 1997 . With their 50th win in the 2016 -- 17 season , the Spurs extended their record for most consecutive 50 - win seasons to 18 ( the Spurs did not
+0	the creation of the federal reserve system was an attempt toReserve System ( also known as the Federal Reserve or simply the Fed ) is the central banking system of the United States of America . Over the years , events such as the Great Depression in the 1930s and the Great Recession during the 2000s have led to the expansion of the
+2	group f / 64 was a major backlash against the earlier photographic movement off / 64 was formed , Edward Weston went to a meeting of the John Reed Club , which was founded to support Marxist artists and writers . These circumstances not only helped set up the situation in which a group
+0	Bessarabia eventually became under the control of which country?city of Vilnius – its historical capital, which was under Polish control during the inter-war
+0	Iran's inflation led to what in 1975-1976?the economy of Iran was flooded with foreign currency, which caused inflation. By 1974, the economy of Iran was experiencing double digit inflation, and despite many large projects to modernize the country, corruption was rampant and caused large
+1	How many steam warships did Japan have in 1867?Yokosuka and Nagasaki. By the end of the Tokugawa shogunate in 1867, the Japanese navy of the shogun already possessed eight western-style steam warships around the flagship Kaiyō Maru, which were used against pro-imperial forces during the Boshin war, under the command
+0	How many people were inside?f former NFL head coach Dan Reeves, suffered a broken back. DeCamillis was seen on a stretcher wearing a neck brace. A line of heavy thunderstorms was moving through the Dallas area at the time, he said, but no other damage to buildings was reported, said Mike Adams, a dispatcher for the Irving, Texas, fire department. Watch the roof collapse on players, coaches » Arnold Payne, a photographer for WFAA, was shooting the Cowboys' practice session when rain began falling "tremendously hard." "I noticed the walls started to waver ... and then I noticed that the lights that were hanging from the ceiling started to sway, and it wouldn't stop," Payne told CNN. Shortly after that, he said, "It was as if someone took a stick pin and hit a balloon." Watch Payne describe being inside when structure collpased » Payne said
+0	Ishita Dutta is the sister of an actress who is typically cast in what genre of movies?he suspense thriller film "Drishyam" (2015) and the Hindi soap opera "Ek Ghar Banaunga", that aired on Star Plus. She is the younger sister of actress Tanushree Dutta. Dutta is the recipient of Femina Miss India Universe title in 2004. During the same year
+3	when did the the civil war start and end/Th> </Tr> <Tr> <Td> <P> 110,000 + killed in action / died of wounds 230,000 + accident / disease deaths 25,000 -- 30,000 died in Confederate prisons </P> <P> 365,000 + total dead
+1	What has Pakistan told phone companies?Islamabad, Pakistan (CNN) -- Under heavy criticism for a telling cell phone carriers to ban certain words in text messages, the Pakistan Telecommunication Authority went into damage control mode Wednesday. PTA spokesman Mohammed Younis Wednesday denied the existence of the plan, which has met with derision from mobile phone users in the country. "If at all we finally decide to
+0	What did Bush say the proposal was to a proposal he vetoed before?(CNN) -- President Bush vetoed an expansion of the federally funded, state-run health insurance program for poor children for a second time Wednesday, telling Congress the bill "moves our country's health care system in the wrong direction." In his veto message, President Bush calls on Congress to extend funding for the current program. "Because the Congress has chosen to send me an essentially identical bill that has the same problems as the flawed bill I previously vetoed, I must veto this legislation, too," he said in a statement released by the White House. The bill would
+0	Where did the football team that Bob Simmons coached from 1995 to 2000 play their home games?Cowboys football team [SEP] The 1998 Oklahoma State Cowboys football team represented the Oklahoma State University during the 1998 NCAA Division I-A football season. They participated as members of the Big 12 Conference in the South Division. They were coached by head coach Bob Simmons. [PAR] [TLE] Bob Simmons (American football coach) [SEP] Bob
+2	What anniversary was recently celebrated in Iran?us to move our policy in a new direction," Obama said. "So there are going to be a set of objectives that we have in these conversations, but I think that there's the possibility at least of a relationship of mutual respect and progress." The United States and Iran have not had diplomatic relations since 1979. During that year, the Shah of Iran was forced to flee the country and the Ayatollah Khomeini took power. Later that year, Iranian students took over and seized hostages at the U.S. Embassy. Relations have been cut since then. U.S. President George W. Bush labeled Iran as a member of the "axis of evil" after the Sept. 11, 2001 attacks. Iran celebrated the 30th anniversary of the revolution Tuesday with crowds chanting "Death to America." Watch the parade in Iran » Tensions have rippled over issues such as Iran's nuclear program, Israel, and Iraq, and have been aggravated since the outspoken Ahmadinejad came to power in 2005. Western
+1	Which Italian composer did George Balanchine add in 1976?[PAR] [TLE] Arcangelo Corelli [SEP] Arcangelo Corelli ( ; 17 February 1653 – 8 January 1713) was an Italian violinist and composer of the Baroque era. His music
+0	Will the playstation 4 be announced?a new system sometime in the next five years, of course. Sony continued to sell the PlayStation 2 system and games years after the PlayStation 3 debuted in stores. For Sony's next console, the company will not deploy a streaming delivery system like OnLive, or fully cut out disc retailers like Best Buy and GameStop, Hirai said. While Sony has increased the number of games and other media available for download or streaming through its networks, most people cannot be expected to frequently download several gigabytes worth of data, which can be a time-consuming process, he said. Sony Computer Entertainment president Andrew House said earlier that Sony is not planning to discuss a new console, the website ComputerAndVideogames.com reported on Monday.
+1	How many children were the Americans trying to kidnap out of Haiti?Port-au-Prince, Haiti (CNN) -- A Haitian attorney representing 10 Americans charged with kidnapping for trying to take 33 children out of Haiti told CNN Sunday he has resigned. Edwin Coq said he had quit as a lawyer for the Americans. It wasn't immediately clear who would replace him. "I know that they have been looking at other lawyers," said Phyllis Allison, mother of one of those detained, Jim Allen. "They don't know what to do." The 10 missionaries, including group leader Laura Silsby, were charged Thursday with kidnapping children and criminal association. Coq had said that court hearings would be held Monday
+0	who kills tree gelbman in happy death dayTree convinces Carter of her predicament by showing that she holds foreknowledge of the day 's events . Tree admits to Carter she does n't like who
+0	What will no person be denied the enjoyment of in Georgia based on their religious principles?amended as follows: "Article IV. Section 10. No person within this state shall, upon any pretense, be deprived of the inestimable privilege of worshipping God in any
+0	who came up with the idea of footballpass . The popularity of college football grew as it became the dominant version of the sport in the United States for the first half of the 20th century . Bowl games , a college football tradition , attracted a national audience for college
+0	what is the name of the female smurfbefore the smurflings created Sassette , Smurfette was the only female smurf in the Smurf Village .
+3	Who contributed to the American studies programs at Yale and University of Wyoming?struggle. Norman Holmes Pearson, who worked for the Office of Strategic Studies in London during World War II, returned to Yale and headed the new American studies program, in which scholarship quickly became an instrument of promoting
+0	What is the group's former name that now has an office with the Chief Actuary besides the Social Security Administration?Office of the Chief Actuary [SEP] The Office of the Chief Actuary is a government agency that has responsibility for actuarial estimates regarding social welfare programs. In Canada, the Office of the Chief Actuary works with the Canada Pension Plan and the Old Age Security Program. In the United States, both the Social Security Administration and the Centers for Medicare and Medicaid Services have an Office of the Chief Actuary that deals with Social Security and Medicare, respectively. A similar agency in the United Kingdom is called the Government Actuary's Department
+0	The actor that playes Han Solo in the "Star Wars" film series stars with Blake Lively and Michiel Huisman in a film directed by who?about a woman who stops aging after an accident at the age of 29. Mills Goodloe and Salvador Paskowitz. The film stars Blake Lively, Michiel Huisman, Kathy Baker, Amanda Crew, Harrison Ford, and Ellen Burstyn. The film was theatrically released on April 24, 2015 by Lionsgate. [PAR] [TLE] Harrison Ford [SEP] Harrison Ford
+0	What historically black university's men's basketball coach was formerly head coach at Virginia Tech?well as an 1890 Historically Black Land-Grant University. The University is a member-school of the Thurgood Marshall College Fund. He was also the head coach at Virginia Tech, Tennessee
+0	what year did syracuse win the ncaa tournament. Their combined record is 67 -- 39 .
+1	where do i get chips at a casino<P> Money is exchanged for tokens in a casino at the casino cage , at the gaming tables , or at a cashier station . The tokens are
+0	when was the winter fuel payment first introducedheating over the winter months .
+0	Trophy hunting can include areas which would likely be unsuitable for what other types of ecotourism?study states that less than 3% of a trophy hunters' expenditures reach the local level, meaning that the economic incentive and benefit is "minimal, particularly when we consider the vast areas of
+1	In simple language, what are the interconnections in an embedding matrix?Since it was quite easy to stack interconnections (wires) inside the embedding matrix, the approach allowed designers to forget completely about the routing of wires (usually a time-consuming operation of PCB design): Anywhere the designer needs a connection, the machine will draw a wire in straight line from one location/pin
+2	rho has been to the most all star games in baseballn4 </Li> <Li> Stan Musial 24 </Li>
+0	In 1169, Ireland was invaded by which people?High King to ensure the terms of the Treaty of Windsor led Henry II, as King of England, to rule as effective monarch under the title of Lord of Ireland. This title was granted to his younger son but when Henry's heir unexpectedly died the title of King of England and Lord of Ireland became entwined in one
+1	What year did a biracial Populist fusion gain the Governors office?to the legislature and governor's office, but the Populists attracted voters displeased with them. In 1896 a biracial, Populist-Republican Fusionist coalition gained the governor's office. The Democrats regained control of the legislature
+1	nearest metro station to majnu ka tilla delhiRing Road of Delhi . It is at a walkable distance from ISBT Kashmere Gate . It is approachable through the Kashmeri Gate station of the Delhi Metro , lies on both the Red ( Dilshad Garden - Rithala ) and Yellow Lines ( Samaypur
+3	where is california located in the region of the united states<P> California is a U.S. state in the Pacific Region of the United States . With 39.5 million residents , California is the most populous state in the United States and the third largest by area . The
+1	when did the baptist church start in americacoworker for religious freedom , are variously credited as founding the earliest Baptist church in North America . In 1639 , Williams established a Baptist church in Providence , Rhode Island , and Clarke began a Baptist church in
+0	where was the first capital of the united states locatedpassed to pave the way for a permanent capital . The decision to locate the capital was contentious , but Alexander Hamilton helped broker a compromise in which the federal government would take on war debt incurred during the American Revolutionary War , in exchange for support from northern states for locating the capital along the Potomac
+0	What will new regulations will reduce?products off of the consumer market," said Michael Fry, director of conservation advocacy for the American Bird Conservancy. "By putting these restrictions in place, they are allowing a compromise to be made between themselves and organizations who have been working on this problem for a long time." The EPA's new measures, which were handed down Thursday, require that rat poisons be kept in bait stations above ground and in containers that meet agency standards. Loose bait, such as pellets, and the four most hazardous types of pesticides, known as "second-generation anticoagulants," will no longer be sold for personal use. Under the new restrictions, only farmers, livestock owners and certified rodent control employees will be allowed to purchase rat poison in bulk. Bags larger than 8 pounds will no longer be sold at hardware and home-improvement stores. Children who come into contact
+0	who played lois lane in the man of steelmixture of toughness and vulnerability , but Peter Bradshaw thought that the character was `` sketchily conceived '' and criticized her lack of chemistry with Cavill . Even so , the film earned over $660 million to become one of her biggest box
+0	What year did the writer of the 1968 novel "The Iron Man" become Poet Laurete?Giant is a 1999 American animated science-fiction comedy-drama action film using both traditional animation and computer animation, produced by and directed by Brad Bird in his directorial debut. It is based on the 1968 novel "The Iron Man" by Ted Hughes (which was published in the United States as "The Iron Giant") and was scripted by Tim McCanlies from a story treatment by Bird. The film stars the voices of Eli Marienthal,
+2	The conquest of Nice was an effort by Suleiman and what French king?allies. A month prior to the siege of Nice, France supported the Ottomans with an artillery unit during the 1543 Ottoman conquest of Esztergom in northern Hungary. After further advances by the Turks, the Habsburg ruler Ferdinand officially recognized Ottoman ascendancy in Hungary in
+0	when was the vaccine receivedfor swine flu, also known as 2009 H1N1, using reverse genetics, he said. "Suitable viruses will hopefully be sent to manufacturers by end of next week," Skinner wrote. Once that happens, vaccine makers will tweak the virus and have "pilot lots" of vaccine ready to be tested by mid- to late June. Several thousand cases have been reported
+1	What is the nationality of the actor who costarred with Matt LeBlanc in "All the Queen's Men"?n approximate -99.92% return. [PAR] [TLE] Eddie Izzard [SEP] Edward John "Eddie" Izzard ( ; born 7 February 1962) is an English stand-up comedian, actor, writer and political activist. His comedic style takes the form of rambling, whimsical monologue, and self-referential pantomime. He
+0	What sickened thousands of children?executives detained, a local official said, according to Xinhua, Initial tests showed more than 1,300 children in the Hunan province town of Wenping have excessive lead in their blood from the Wugang Manganese Smelting Plant. A second round of testing has been ordered to confirm the results. The plant opened in May 2008 without gaining the approval of the local environment protection bureau, said Huang Wenbin, a deputy environment chief in Wugang City, Xinhua reported. The plant was within 500 meters (about a quarter mile) of three schools. The
+0	What percentage of the population are the Kpelle?are descendants of African American and West Indian, mostly Barbadian settlers, make up 2.5%. Congo people, descendants of repatriated Congo and Afro-Caribbean
+1	Amount of people left homeless?86 dead, the state news agency said. About 30 people are missing, the official news agency Agencia Brasil said, citing civil defense officials. Earlier reports had indicated as many as 100 people were dead. In addition, more than 54,000 residents have been left homeless, and another 1.5 million have been affected by the heavy rains, the state news agency reported. Brazilian President Luiz Inacio Lula da Silva announced he will release nearly 700 million reais ($350 million)
+2	What other countries were in disagreement with the United Nations decision on Burma ?that strongly called upon the government of Myanmar to end its systematic violations of human rights. In January 2007, Russia and China vetoed a
+0	Besides Barcelona and Real Madrid, what other team has remained in the Primera Division?first football club to win six out of six competitions in a single year, completing the sextuple in also winning the Spanish Super Cup, UEFA Super Cup and FIFA Club World Cup. In 2011, the club became
+0	William Frederick Truax, is a former professional American football tight end in the National Football League (NFL) from 1964 to 1973 for the Los Angeles Rams and the Dallas Cowboys, following the 1970 NFL season, Truax was traded by the Rams to the Cowboys for wide receiver Lance Rentzel, a former American football flanker, in which organization?in New Orleans and college football at Louisiana State University and was drafted in the second round of the 1964 NFL draft. Following the 1970 NFL season, Truax was traded by the Rams to the Cowboys for wide receiver Lance Rentzel. He was part of the Cowboys' Super Bowl VI championship team in 1971. He played
+3	What year did Chopin learn that the uprising in Warsaw was crushed?enlist. Chopin, now alone in Vienna, was nostalgic for his homeland, and wrote to a friend, "I curse the moment of my departure." When in September 1831 he learned, while travelling from Vienna to Paris, that the uprising had been crushed, he expressed his anguish in the pages of his private journal: "Oh
+1	where do they make money in washington dc; all coinage is produced by the United States Mint . With production facilities in Washington , DC , and Fort Worth , Texas , the Bureau of Engraving and Printing is the largest producer of government security documents in the United States . </P>
+0	What did a researcher compare this process to?which makes it one of the highest rates of maternal mortality in the Americas. In wealthy developed nations, only nine women die for every 100,000 births. The five main causes of pregnancy-related deaths in Peru are hemorrhage, pre-eclampsia, infection, complications following abortion and obstructed birth, according to Peru's Ministry of Health figures. Amnesty's Peru researcher Nuria Garcia said, in a written statement: "The rates of maternal mortality in Peru are scandalous. The fact that so many women are dying from preventable causes is a human rights violation. "The Peruvian state is simply ignoring
+0	How many containers can Longtan Containers Port Area handle?Port of Nanjing is the largest inland port in China, with annual cargo tonnage reached 191,970,000 t in 2012. The port area is 98 kilometres (61 mi) in length and has 64 berths
+0	The 2011 New York City Marathon was sponsored by which Dutch multinational banking corporation?are retail banking, direct banking, commercial banking, investment banking, asset management, and insurance services. ING is an abbreviation for "Internationale Nederlanden Groep " (English: International Netherlands Group). [PAR] [TLE] 2011 New York City Marathon [SEP] The 42nd New York City Marathon took
+0	What is human flourishing?it does not involve believing that human nature is purely good or that all people can live up to the Humanist ideals without help. If anything, there is recognition that living up to one's potential is hard
+0	What was the result of Dida appealto play in next month's Champions League match at Shakhtar Donetsk after partially winning his appeal to UEFA against a two-match ban. Dida has had one game of his two-match ban suspended for a year following an appeal to UEFA. Brazilian Dida was also fined 60,000 Swiss francs by European football's ruling body following an incident involving a supporter during the Champions clash against Celtic in Scotland on October 3. The 34-year-old Brazilian was initially banned for two games for his theatrics following a Celtic fan's encroachment onto the pitch during the 2-1 defeat at Celtic
+1	What is more plentiful in capital projects?generates economic distortion in the public sector by diverting public investment into capital projects where bribes and kickbacks are more plentiful. Officials may increase the technical complexity of public sector projects to conceal or
+0	where were band greeted with cheers?the United States for a show in Stamford, Connecticut, on Tuesday, after they have "a few days off to recuperate," Robinson said. The trio was the opening act for Nelson until they were loudly booed in Toronto, a day after the actor-musician's bizarre interview with a CBC radio host. Ironically, the comments that offended Canadians included Thornton's assessment that they were "very reserved" and "it doesn't matter what you say to them." "It's mashed potatoes with no gravy," Thornton told CBC host Jian Ghomeshi. "We tend to play places where people throw things at each other and here they just sort of sit there," he said. Watch Thornton's interview » The audience at Thursday night's show in Toronto loudly booed the Boxmasters, with some shouts of "Here comes the gravy!" The Toronto Star newspaper reported. Thornton's remarks about
+0	What do Mexicans call Mexico City?the Federal District in Spanish: D.F., which is read "De-Efe"). They are formally called capitalinos (in reference to the city being the capital of the country), but "[p]erhaps because capitalino is the
+0	where does lock stock and barrel come fromindividual components one at a time . One craftsman made the `` lock '' which would have been a `` match lock '' , `` wheel lock '' , `` flint lock '' etc .
+1	who has the power to establish a prison system<P> The Federal Bureau of Prisons ( BOP ) is a United States federal law enforcement agency . A subdivision of
+0	what are south americas only 2 landlocked countriessuch countries , including five partially recognised states .

demo/demo2/log.txt

+{'token_ids': [[-1, -1], 'int64'], 'label_ids': [[-1], 'int64']}
+{'token_ids': [[-1, -1], 'int64']}
+<paddlepalm.backbone.ernie.ERNIE object at 0x7f6645f5d410>
+{'token_ids': [[-1, -1], 'int64'], 'label_ids': [[-1], 'int64'], u'input_mask': [[-1, -1, 1], 'float32'], u'position_ids': [[-1, -1], 'int64'], u'task_ids': [[-1, -1], 'int64'], u'segment_ids': [[-1, -1], 'int64']}
+{'token_ids': [[-1, -1], 'int64']}
+preparing data...
+0
+61
+done!
+name: "tmp_0"
+type {
+  type: LOD_TENSOR
+  lod_tensor {
+    tensor {
+      data_type: INT64
+      dims: 1
+    }
+    lod_level: 0
+  }
+}
+
+name: "reduce_sum_0.tmp_0"
+type {
+  type: LOD_TENSOR
+  lod_tensor {
+    tensor {
+      data_type: FP32
+      dims: 1
+    }
+  }
+}
+persistable: false
+
+name: "reduce_sum_1.tmp_0"
+type {
+  type: LOD_TENSOR
+  lod_tensor {
+    tensor {
+      data_type: FP32
+      dims: 1
+    }
+  }
+}
+persistable: false
+
+random init params...
+Loading pretraining parameters from pretrain/ernie/params...
+Warning: cls.cls_out_w not found in pretrain/ernie/params.
+Warning: cls.cls_out_b not found in pretrain/ernie/params.
+Warning: senti_cls.cls_out_w not found in pretrain/ernie/params.
+Warning: senti_cls.cls_out_b not found in pretrain/ernie/params.
+
+ok!
+cls: expected train steps 30.
+senti_cls: expected train steps 30.
+ok!
+Estimated overall train steps 60.
+{'__task_id': array([0]), u'token_ids': array([[  101,  2073,  2515,  5843,  4518,  1998,  8460,  2272,  2013,
+        22254, 12848,  3593,  8787,  6177,  2028,  2012,  1037,  2051,
+         1012,   100, 26286,  2081,  1996,  1036,  1036,  5843,  1005,
+         1005,  2029,  2052,  2031,  2042,  1037,  1036,  1036,  2674,
+         5843,  1005,  1005,  1010,  1036,  1036,  5217,  5843,  1005,
+         1005,  1010,  1036,  1036, 13493,  5843,  1005,  1005,  4385,
+         1012,   102,     0,     0,     0,     0,     0,     0,     0,
+            0,     0,     0,     0,     0,     0,     0,     0,     0,
+            0,     0,     0,     0,     0,     0,     0,     0,     0,
+            0,     0],
+       [  101,   100, 12904,  2683,  1010,   100,  2001, 10836,  2011,
+         2029,  2111,  1029,   100,   100,  2000,  5676,  1996,  3408,
+         1997,  1996,   100,  1997,   100,  2419,   100,   100,  1010,
+         2004,   100,  1997,   100,  1010,  2000,  3627,  2004,  4621,
+        11590,  2104,  1996,  2516,  1997,   100,  1997,   100,  1012,
+          100,  2516,  2001,  4379,  2000,  2010,  3920,  2365,  2021,
+         2043,   100,  1005,  1055,  8215, 14153,  2351,  1996,  2516,
+         1997,   100,  1997,   100,  1998,   100,  1997,   100,  2150,
+         4372, 21077,  1999,  2028,   102,     0,     0,     0,     0,
+            0,     0],
+       [  101,   100,   100,  2003,  1996,  2905,  1997,  2019,  3883,
+         2040,  2003,  4050,  3459,  1999,  2054,  6907,  1997,  5691,
+         1029,  2002, 23873, 10874,  2143,  1000,   100,  1000,  1006,
+         2325,  1007,  1998,  1996,   100,  7815,  3850,  1000,   100,
+          100,   100,  1000,  1010,  2008,  4836,  2006,   100,   100,
+         1012,   100,  2003,  1996,  3920,  2905,  1997,  3883,   100,
+          100,  1012,   100,  2003,  1996,  7799,  1997,   100,   100,
+          100,   100,  2516,  1999,  2432,  1012,   100,  1996,  2168,
+         2095,   102,     0,     0,     0,     0,     0,     0,     0,
+            0,     0],
+       [  101,   100,  2095,  2106,   100,  4553,  2008,  1996, 10138,
+         1999,   100,  2001, 10560,  1029, 28845,  1012,   100,  1010,
+         2085,  2894,  1999,   100,  1010,  2001, 16839,  9080, 12863,
+         2005,  2010, 10759,  1010,  1998,  2626,  2000,  1037,  2767,
+         1010,  1000,   100,  8364,  1996,  2617,  1997,  2026,  6712,
+         1012,  1000,   100,  1999,   100, 10937,  2002,  4342,  1010,
+         2096,  8932,  2013,   100,  2000,   100,  1010,  2008,  1996,
+        10138,  2018,  2042, 10560,  1010,  2002,  5228,  2010, 21782,
+         1999,  1996,  5530,  1997,  2010,  2797,  3485,  1024,  1000,
+          100,   102]]), u'input_mask': array([[[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.]],
+
+       [[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.]],
+
+       [[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.]],
+
+       [[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.]]], dtype=float32), u'position_ids': array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55,  0,  0,  0,  0,  0,  0,  0,  0,
+         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
+         0,  0,  0],
+       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
+        64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76,  0,  0,  0,
+         0,  0,  0],
+       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
+        64, 65, 66, 67, 68, 69, 70, 71, 72, 73,  0,  0,  0,  0,  0,  0,
+         0,  0,  0],
+       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
+        64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
+        80, 81, 82]]), u'task_ids': array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'cls.label_ids': array([0, 0, 0, 3]), u'segment_ids': array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}
+<paddle.fluid.compiler.CompiledProgram object at 0x7f6602584a10>
+<paddle.fluid.compiler.CompiledProgram object at 0x7f6602584a10>
+hahahahahahhahah
+{u'token_ids': array([[  101,  2073,  2515,  5843,  4518,  1998,  8460,  2272,  2013,
+        22254, 12848,  3593,  8787,  6177,  2028,  2012,  1037,  2051,
+         1012,   100, 26286,  2081,  1996,  1036,  1036,  5843,  1005,
+         1005,  2029,  2052,  2031,  2042,  1037,  1036,  1036,  2674,
+         5843,  1005,  1005,  1010,  1036,  1036,  5217,  5843,  1005,
+         1005,  1010,  1036,  1036, 13493,  5843,  1005,  1005,  4385,
+         1012,   102,     0,     0,     0,     0,     0,     0,     0,
+            0,     0,     0,     0,     0,     0,     0,     0,     0,
+            0,     0,     0,     0,     0,     0,     0,     0,     0,
+            0,     0],
+       [  101,   100, 12904,  2683,  1010,   100,  2001, 10836,  2011,
+         2029,  2111,  1029,   100,   100,  2000,  5676,  1996,  3408,
+         1997,  1996,   100,  1997,   100,  2419,   100,   100,  1010,
+         2004,   100,  1997,   100,  1010,  2000,  3627,  2004,  4621,
+        11590,  2104,  1996,  2516,  1997,   100,  1997,   100,  1012,
+          100,  2516,  2001,  4379,  2000,  2010,  3920,  2365,  2021,
+         2043,   100,  1005,  1055,  8215, 14153,  2351,  1996,  2516,
+         1997,   100,  1997,   100,  1998,   100,  1997,   100,  2150,
+         4372, 21077,  1999,  2028,   102,     0,     0,     0,     0,
+            0,     0],
+       [  101,   100,   100,  2003,  1996,  2905,  1997,  2019,  3883,
+         2040,  2003,  4050,  3459,  1999,  2054,  6907,  1997,  5691,
+         1029,  2002, 23873, 10874,  2143,  1000,   100,  1000,  1006,
+         2325,  1007,  1998,  1996,   100,  7815,  3850,  1000,   100,
+          100,   100,  1000,  1010,  2008,  4836,  2006,   100,   100,
+         1012,   100,  2003,  1996,  3920,  2905,  1997,  3883,   100,
+          100,  1012,   100,  2003,  1996,  7799,  1997,   100,   100,
+          100,   100,  2516,  1999,  2432,  1012,   100,  1996,  2168,
+         2095,   102,     0,     0,     0,     0,     0,     0,     0,
+            0,     0],
+       [  101,   100,  2095,  2106,   100,  4553,  2008,  1996, 10138,
+         1999,   100,  2001, 10560,  1029, 28845,  1012,   100,  1010,
+         2085,  2894,  1999,   100,  1010,  2001, 16839,  9080, 12863,
+         2005,  2010, 10759,  1010,  1998,  2626,  2000,  1037,  2767,
+         1010,  1000,   100,  8364,  1996,  2617,  1997,  2026,  6712,
+         1012,  1000,   100,  1999,   100, 10937,  2002,  4342,  1010,
+         2096,  8932,  2013,   100,  2000,   100,  1010,  2008,  1996,
+        10138,  2018,  2042, 10560,  1010,  2002,  5228,  2010, 21782,
+         1999,  1996,  5530,  1997,  2010,  2797,  3485,  1024,  1000,
+          100,   102]]), u'input_mask': array([[[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.]],
+
+       [[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.]],
+
+       [[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.],
+        [0.]],
+
+       [[1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.],
+        [1.]]], dtype=float32), u'position_ids': array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55,  0,  0,  0,  0,  0,  0,  0,  0,
+         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
+         0,  0,  0],
+       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
+        64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76,  0,  0,  0,
+         0,  0,  0],
+       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
+        64, 65, 66, 67, 68, 69, 70, 71, 72, 73,  0,  0,  0,  0,  0,  0,
+         0,  0,  0],
+       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
+        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
+        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
+        64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
+        80, 81, 82]]), u'task_ids': array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), u'cls.label_ids': array([0, 0, 0, 3]), u'segment_ids': array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}

demo/demo2/run.py

+# coding=utf-8
 import paddlepalm as palm
+import json
 
 if __name__ == '__main__':
 
     max_seqlen = 512
-    batch_size = 32
-
-    match_reader = palm.reader.match(train_file, vocab, \
-        max_seqlen, file_format='csv', tokenizer='wordpiece', \
-        lang='en', shuffle_train=True)
-    mrc_reader = palm.reader.mrc(train_file, phase='train')
-    mlm_reader = palm.reader.mlm(train_file, phase='train')
-    palm.reader.
-
-    match = palm.tasktype.cls(num_classes=4)
-    mrc = palm.tasktype.match(learning_strategy='pairwise')
-    mlm = palm.tasktype.mlm()
-    mlm.print()
-
-    
-    bb_flags = palm.load_json('./pretrain/ernie/ernie_config.json')
-    bb = palm.backbone.ernie(bb_flags['xx'], xxx)
-    bb.print()
-
-    match4mrqa = palm.Task('match4mrqa', match_reader, match_tt)
-    mrc4mrqa = palm.Task('match4mrqa', match_reader, match_tt)
-
-    # match4mrqa.reuse_with(mrc4mrqa)
-
-
-    controller = palm.Controller([mrqa, match4mrqa, mlm4mrqa])
-
-    loss = controller.build_forward(bb, mask_task=[])
-
-    n_steps = controller.estimate_train_steps(basetask=mrqa, num_epochs=2, batch_size=8, dev_count=4)
-    adam = palm.optimizer.Adam(loss)
-    sched = palm.schedualer.LinearWarmup(learning_rate, max_train_steps=n_steps, warmup_steps=0.1*n_steps)
-    
-    controller.build_backward(optimizer=adam, schedualer=sched, weight_decay=0.001, use_ema=True, ema_decay=0.999)
-
-    controller.random_init_params()
-    controller.load_pretrain('../../pretrain_model/ernie/params')
-    controller.train()
-
-
-
-
-
-    # controller = palm.Controller(config='config.yaml', task_dir='tasks', for_train=False)
-    # controller.pred('mrqa', inference_model_dir='output_model/secondrun/mrqa/infer_model')
-
+    batch_size = 4
+    num_epochs = 2
+    lr = 1e-3
+    vocab_path = './pretrain/ernie/vocab.txt'
+
+    train_file = './data/cls4mrqa/train.tsv'
+    predict_file = './data/cls4mrqa/dev.tsv'
+
+    config = json.load(open('./pretrain/ernie/ernie_config.json'))
+    # ernie = palm.backbone.ERNIE(...)
+    ernie = palm.backbone.ERNIE.from_config(config)
+
+    # cls_reader2 = palm.reader.cls(train_file_topic, vocab_path, batch_size, max_seqlen)
+    # cls_reader3 = palm.reader.cls(train_file_subj, vocab_path, batch_size, max_seqlen)
+    # topic_trainer = palm.Trainer('topic_cls', cls_reader2, cls)
+    # subj_trainer = palm.Trainer('subj_cls', cls_reader3, cls)
+
+    # 创建该分类任务的reader，由诸多参数控制数据集读入格式、文件数量、预处理规则等
+    cls_reader = palm.reader.ClassifyReader(vocab_path, max_seqlen)
+    cls_reader2 = palm.reader.ClassifyReader(vocab_path, max_seqlen)
+    predict_cls_reader = palm.reader.ClassifyReader(vocab_path, max_seqlen, phase='predict')
+    print(cls_reader.outputs_attr)
+    print(predict_cls_reader.outputs_attr)
+    # 不同的backbone会对任务reader有不同的特征要求，例如对于分类任务，基本的输入feature为token_ids和label_ids，但是对于BERT，还要求从输入中额外提取position、segment、input_mask等特征，因此经过register后，reader会自动补充backbone所要求的字段
+    cls_reader.register_with(ernie)
+    cls_reader2.register_with(ernie)
+    print(cls_reader.outputs_attr)
+    print(predict_cls_reader.outputs_attr)
+
+    print("preparing data...")
+    print(cls_reader.num_examples)
+    cls_reader.load_data(train_file, batch_size)
+    cls_reader2.load_data(train_file, batch_size)
+    print(cls_reader.num_examples)
+    print('done!')
+
+    # 创建任务头（task head），如分类、匹配、机器阅读理解等。每个任务头有跟该任务相关的必选/可选参数。注意，任务头与reader是解耦合的，只要任务头依赖的数据集侧的字段能被reader提供，那么就是合法的
+    cls_head = palm.head.Classify(4, 1024, 0.1)
+    cls_head2 = palm.head.Classify(4, 1024, 0.1)
+
+    # 根据reader和任务头来创建一个训练器trainer，trainer代表了一个训练任务，内部维护着训练进程、和任务的关键信息，并完成合法性校验，该任务的模型保存、载入等相关规则控制
+    trainer = palm.Trainer('cls')
+    trainer2 = palm.Trainer('senti_cls')
+    mh_trainer = palm.MultiHeadTrainer([trainer, trainer2])
+
+    # match4mrqa.reuse_head_with(mrc4mrqa)
+
+    # data_vars = cls_reader.build()
+    # output_vars = ernie.build(data_vars)
+    # cls_head.build({'backbone': output_vars, 'reader': data_vars})
+
+    loss_var = mh_trainer.build_forward(ernie, [cls_head, cls_head2])
+
+    n_steps = cls_reader.num_examples * num_epochs // batch_size
+    warmup_steps = int(0.1 * n_steps)
+    print(warmup_steps)
+    sched = palm.lr_sched.TriangularSchedualer(warmup_steps, n_steps)
+
+    adam = palm.optimizer.Adam(loss_var, lr, sched)
+
+    mh_trainer.build_backward(optimizer=adam, weight_decay=0.001)
+
+    mh_trainer.random_init_params()
+    mh_trainer.load_pretrain('pretrain/ernie/params')
+
+    # trainer.train(iterator_fn, print_steps=1, save_steps=5, save_path='outputs', save_type='ckpt,predict')
+    mh_trainer.fit_readers_with_mixratio([cls_reader, cls_reader2], 'cls', 2)
+    mh_trainer.train(print_steps=1)
+    # trainer.save()
 

demo/demo2/run.sh

-export CUDA_VISIBLE_DEVICES=0
+export CUDA_VISIBLE_DEVICES=3
 python run.py 
 

demo/demo3/data/cls4mrqa/dev.tsv

+label	text_a
+1	when was the last time the san antonio spurs missed the playoffshave only missed the playoffs four times since entering the NBA ; they have not missed the playoffs in the 20 seasons since Tim Duncan was drafted by the Spurs in 1997 . With their 50th win in the 2016 -- 17 season , the Spurs extended their record for most consecutive 50 - win seasons to 18 ( the Spurs did not
+0	the creation of the federal reserve system was an attempt toReserve System ( also known as the Federal Reserve or simply the Fed ) is the central banking system of the United States of America . Over the years , events such as the Great Depression in the 1930s and the Great Recession during the 2000s have led to the expansion of the
+2	group f / 64 was a major backlash against the earlier photographic movement off / 64 was formed , Edward Weston went to a meeting of the John Reed Club , which was founded to support Marxist artists and writers . These circumstances not only helped set up the situation in which a group
+0	Bessarabia eventually became under the control of which country?city of Vilnius – its historical capital, which was under Polish control during the inter-war
+0	Iran's inflation led to what in 1975-1976?the economy of Iran was flooded with foreign currency, which caused inflation. By 1974, the economy of Iran was experiencing double digit inflation, and despite many large projects to modernize the country, corruption was rampant and caused large
+1	How many steam warships did Japan have in 1867?Yokosuka and Nagasaki. By the end of the Tokugawa shogunate in 1867, the Japanese navy of the shogun already possessed eight western-style steam warships around the flagship Kaiyō Maru, which were used against pro-imperial forces during the Boshin war, under the command
+0	How many people were inside?f former NFL head coach Dan Reeves, suffered a broken back. DeCamillis was seen on a stretcher wearing a neck brace. A line of heavy thunderstorms was moving through the Dallas area at the time, he said, but no other damage to buildings was reported, said Mike Adams, a dispatcher for the Irving, Texas, fire department. Watch the roof collapse on players, coaches » Arnold Payne, a photographer for WFAA, was shooting the Cowboys' practice session when rain began falling "tremendously hard." "I noticed the walls started to waver ... and then I noticed that the lights that were hanging from the ceiling started to sway, and it wouldn't stop," Payne told CNN. Shortly after that, he said, "It was as if someone took a stick pin and hit a balloon." Watch Payne describe being inside when structure collpased » Payne said
+0	Ishita Dutta is the sister of an actress who is typically cast in what genre of movies?he suspense thriller film "Drishyam" (2015) and the Hindi soap opera "Ek Ghar Banaunga", that aired on Star Plus. She is the younger sister of actress Tanushree Dutta. Dutta is the recipient of Femina Miss India Universe title in 2004. During the same year
+3	when did the the civil war start and end/Th> </Tr> <Tr> <Td> <P> 110,000 + killed in action / died of wounds 230,000 + accident / disease deaths 25,000 -- 30,000 died in Confederate prisons </P> <P> 365,000 + total dead
+1	What has Pakistan told phone companies?Islamabad, Pakistan (CNN) -- Under heavy criticism for a telling cell phone carriers to ban certain words in text messages, the Pakistan Telecommunication Authority went into damage control mode Wednesday. PTA spokesman Mohammed Younis Wednesday denied the existence of the plan, which has met with derision from mobile phone users in the country. "If at all we finally decide to
+0	What did Bush say the proposal was to a proposal he vetoed before?(CNN) -- President Bush vetoed an expansion of the federally funded, state-run health insurance program for poor children for a second time Wednesday, telling Congress the bill "moves our country's health care system in the wrong direction." In his veto message, President Bush calls on Congress to extend funding for the current program. "Because the Congress has chosen to send me an essentially identical bill that has the same problems as the flawed bill I previously vetoed, I must veto this legislation, too," he said in a statement released by the White House. The bill would
+0	Where did the football team that Bob Simmons coached from 1995 to 2000 play their home games?Cowboys football team [SEP] The 1998 Oklahoma State Cowboys football team represented the Oklahoma State University during the 1998 NCAA Division I-A football season. They participated as members of the Big 12 Conference in the South Division. They were coached by head coach Bob Simmons. [PAR] [TLE] Bob Simmons (American football coach) [SEP] Bob
+2	What anniversary was recently celebrated in Iran?us to move our policy in a new direction," Obama said. "So there are going to be a set of objectives that we have in these conversations, but I think that there's the possibility at least of a relationship of mutual respect and progress." The United States and Iran have not had diplomatic relations since 1979. During that year, the Shah of Iran was forced to flee the country and the Ayatollah Khomeini took power. Later that year, Iranian students took over and seized hostages at the U.S. Embassy. Relations have been cut since then. U.S. President George W. Bush labeled Iran as a member of the "axis of evil" after the Sept. 11, 2001 attacks. Iran celebrated the 30th anniversary of the revolution Tuesday with crowds chanting "Death to America." Watch the parade in Iran » Tensions have rippled over issues such as Iran's nuclear program, Israel, and Iraq, and have been aggravated since the outspoken Ahmadinejad came to power in 2005. Western
+1	Which Italian composer did George Balanchine add in 1976?[PAR] [TLE] Arcangelo Corelli [SEP] Arcangelo Corelli ( ; 17 February 1653 – 8 January 1713) was an Italian violinist and composer of the Baroque era. His music
+0	Will the playstation 4 be announced?a new system sometime in the next five years, of course. Sony continued to sell the PlayStation 2 system and games years after the PlayStation 3 debuted in stores. For Sony's next console, the company will not deploy a streaming delivery system like OnLive, or fully cut out disc retailers like Best Buy and GameStop, Hirai said. While Sony has increased the number of games and other media available for download or streaming through its networks, most people cannot be expected to frequently download several gigabytes worth of data, which can be a time-consuming process, he said. Sony Computer Entertainment president Andrew House said earlier that Sony is not planning to discuss a new console, the website ComputerAndVideogames.com reported on Monday.
+1	How many children were the Americans trying to kidnap out of Haiti?Port-au-Prince, Haiti (CNN) -- A Haitian attorney representing 10 Americans charged with kidnapping for trying to take 33 children out of Haiti told CNN Sunday he has resigned. Edwin Coq said he had quit as a lawyer for the Americans. It wasn't immediately clear who would replace him. "I know that they have been looking at other lawyers," said Phyllis Allison, mother of one of those detained, Jim Allen. "They don't know what to do." The 10 missionaries, including group leader Laura Silsby, were charged Thursday with kidnapping children and criminal association. Coq had said that court hearings would be held Monday
+0	who kills tree gelbman in happy death dayTree convinces Carter of her predicament by showing that she holds foreknowledge of the day 's events . Tree admits to Carter she does n't like who
+0	What will no person be denied the enjoyment of in Georgia based on their religious principles?amended as follows: "Article IV. Section 10. No person within this state shall, upon any pretense, be deprived of the inestimable privilege of worshipping God in any
+0	who came up with the idea of footballpass . The popularity of college football grew as it became the dominant version of the sport in the United States for the first half of the 20th century . Bowl games , a college football tradition , attracted a national audience for college
+0	what is the name of the female smurfbefore the smurflings created Sassette , Smurfette was the only female smurf in the Smurf Village .
+3	Who contributed to the American studies programs at Yale and University of Wyoming?struggle. Norman Holmes Pearson, who worked for the Office of Strategic Studies in London during World War II, returned to Yale and headed the new American studies program, in which scholarship quickly became an instrument of promoting
+0	What is the group's former name that now has an office with the Chief Actuary besides the Social Security Administration?Office of the Chief Actuary [SEP] The Office of the Chief Actuary is a government agency that has responsibility for actuarial estimates regarding social welfare programs. In Canada, the Office of the Chief Actuary works with the Canada Pension Plan and the Old Age Security Program. In the United States, both the Social Security Administration and the Centers for Medicare and Medicaid Services have an Office of the Chief Actuary that deals with Social Security and Medicare, respectively. A similar agency in the United Kingdom is called the Government Actuary's Department
+0	The actor that playes Han Solo in the "Star Wars" film series stars with Blake Lively and Michiel Huisman in a film directed by who?about a woman who stops aging after an accident at the age of 29. Mills Goodloe and Salvador Paskowitz. The film stars Blake Lively, Michiel Huisman, Kathy Baker, Amanda Crew, Harrison Ford, and Ellen Burstyn. The film was theatrically released on April 24, 2015 by Lionsgate. [PAR] [TLE] Harrison Ford [SEP] Harrison Ford
+0	What historically black university's men's basketball coach was formerly head coach at Virginia Tech?well as an 1890 Historically Black Land-Grant University. The University is a member-school of the Thurgood Marshall College Fund. He was also the head coach at Virginia Tech, Tennessee
+0	what year did syracuse win the ncaa tournament. Their combined record is 67 -- 39 .
+1	where do i get chips at a casino<P> Money is exchanged for tokens in a casino at the casino cage , at the gaming tables , or at a cashier station . The tokens are
+0	when was the winter fuel payment first introducedheating over the winter months .
+0	Trophy hunting can include areas which would likely be unsuitable for what other types of ecotourism?study states that less than 3% of a trophy hunters' expenditures reach the local level, meaning that the economic incentive and benefit is "minimal, particularly when we consider the vast areas of
+1	In simple language, what are the interconnections in an embedding matrix?Since it was quite easy to stack interconnections (wires) inside the embedding matrix, the approach allowed designers to forget completely about the routing of wires (usually a time-consuming operation of PCB design): Anywhere the designer needs a connection, the machine will draw a wire in straight line from one location/pin
+2	rho has been to the most all star games in baseballn4 </Li> <Li> Stan Musial 24 </Li>
+0	In 1169, Ireland was invaded by which people?High King to ensure the terms of the Treaty of Windsor led Henry II, as King of England, to rule as effective monarch under the title of Lord of Ireland. This title was granted to his younger son but when Henry's heir unexpectedly died the title of King of England and Lord of Ireland became entwined in one
+1	What year did a biracial Populist fusion gain the Governors office?to the legislature and governor's office, but the Populists attracted voters displeased with them. In 1896 a biracial, Populist-Republican Fusionist coalition gained the governor's office. The Democrats regained control of the legislature
+1	nearest metro station to majnu ka tilla delhiRing Road of Delhi . It is at a walkable distance from ISBT Kashmere Gate . It is approachable through the Kashmeri Gate station of the Delhi Metro , lies on both the Red ( Dilshad Garden - Rithala ) and Yellow Lines ( Samaypur
+3	where is california located in the region of the united states<P> California is a U.S. state in the Pacific Region of the United States . With 39.5 million residents , California is the most populous state in the United States and the third largest by area . The
+1	when did the baptist church start in americacoworker for religious freedom , are variously credited as founding the earliest Baptist church in North America . In 1639 , Williams established a Baptist church in Providence , Rhode Island , and Clarke began a Baptist church in
+0	where was the first capital of the united states locatedpassed to pave the way for a permanent capital . The decision to locate the capital was contentious , but Alexander Hamilton helped broker a compromise in which the federal government would take on war debt incurred during the American Revolutionary War , in exchange for support from northern states for locating the capital along the Potomac
+0	What will new regulations will reduce?products off of the consumer market," said Michael Fry, director of conservation advocacy for the American Bird Conservancy. "By putting these restrictions in place, they are allowing a compromise to be made between themselves and organizations who have been working on this problem for a long time." The EPA's new measures, which were handed down Thursday, require that rat poisons be kept in bait stations above ground and in containers that meet agency standards. Loose bait, such as pellets, and the four most hazardous types of pesticides, known as "second-generation anticoagulants," will no longer be sold for personal use. Under the new restrictions, only farmers, livestock owners and certified rodent control employees will be allowed to purchase rat poison in bulk. Bags larger than 8 pounds will no longer be sold at hardware and home-improvement stores. Children who come into contact
+0	who played lois lane in the man of steelmixture of toughness and vulnerability , but Peter Bradshaw thought that the character was `` sketchily conceived '' and criticized her lack of chemistry with Cavill . Even so , the film earned over $660 million to become one of her biggest box
+0	What year did the writer of the 1968 novel "The Iron Man" become Poet Laurete?Giant is a 1999 American animated science-fiction comedy-drama action film using both traditional animation and computer animation, produced by and directed by Brad Bird in his directorial debut. It is based on the 1968 novel "The Iron Man" by Ted Hughes (which was published in the United States as "The Iron Giant") and was scripted by Tim McCanlies from a story treatment by Bird. The film stars the voices of Eli Marienthal,
+2	The conquest of Nice was an effort by Suleiman and what French king?allies. A month prior to the siege of Nice, France supported the Ottomans with an artillery unit during the 1543 Ottoman conquest of Esztergom in northern Hungary. After further advances by the Turks, the Habsburg ruler Ferdinand officially recognized Ottoman ascendancy in Hungary in
+0	when was the vaccine receivedfor swine flu, also known as 2009 H1N1, using reverse genetics, he said. "Suitable viruses will hopefully be sent to manufacturers by end of next week," Skinner wrote. Once that happens, vaccine makers will tweak the virus and have "pilot lots" of vaccine ready to be tested by mid- to late June. Several thousand cases have been reported
+1	What is the nationality of the actor who costarred with Matt LeBlanc in "All the Queen's Men"?n approximate -99.92% return. [PAR] [TLE] Eddie Izzard [SEP] Edward John "Eddie" Izzard ( ; born 7 February 1962) is an English stand-up comedian, actor, writer and political activist. His comedic style takes the form of rambling, whimsical monologue, and self-referential pantomime. He
+0	What sickened thousands of children?executives detained, a local official said, according to Xinhua, Initial tests showed more than 1,300 children in the Hunan province town of Wenping have excessive lead in their blood from the Wugang Manganese Smelting Plant. A second round of testing has been ordered to confirm the results. The plant opened in May 2008 without gaining the approval of the local environment protection bureau, said Huang Wenbin, a deputy environment chief in Wugang City, Xinhua reported. The plant was within 500 meters (about a quarter mile) of three schools. The
+0	What percentage of the population are the Kpelle?are descendants of African American and West Indian, mostly Barbadian settlers, make up 2.5%. Congo people, descendants of repatriated Congo and Afro-Caribbean
+1	Amount of people left homeless?86 dead, the state news agency said. About 30 people are missing, the official news agency Agencia Brasil said, citing civil defense officials. Earlier reports had indicated as many as 100 people were dead. In addition, more than 54,000 residents have been left homeless, and another 1.5 million have been affected by the heavy rains, the state news agency reported. Brazilian President Luiz Inacio Lula da Silva announced he will release nearly 700 million reais ($350 million)
+2	What other countries were in disagreement with the United Nations decision on Burma ?that strongly called upon the government of Myanmar to end its systematic violations of human rights. In January 2007, Russia and China vetoed a
+0	Besides Barcelona and Real Madrid, what other team has remained in the Primera Division?first football club to win six out of six competitions in a single year, completing the sextuple in also winning the Spanish Super Cup, UEFA Super Cup and FIFA Club World Cup. In 2011, the club became
+0	William Frederick Truax, is a former professional American football tight end in the National Football League (NFL) from 1964 to 1973 for the Los Angeles Rams and the Dallas Cowboys, following the 1970 NFL season, Truax was traded by the Rams to the Cowboys for wide receiver Lance Rentzel, a former American football flanker, in which organization?in New Orleans and college football at Louisiana State University and was drafted in the second round of the 1964 NFL draft. Following the 1970 NFL season, Truax was traded by the Rams to the Cowboys for wide receiver Lance Rentzel. He was part of the Cowboys' Super Bowl VI championship team in 1971. He played
+3	What year did Chopin learn that the uprising in Warsaw was crushed?enlist. Chopin, now alone in Vienna, was nostalgic for his homeland, and wrote to a friend, "I curse the moment of my departure." When in September 1831 he learned, while travelling from Vienna to Paris, that the uprising had been crushed, he expressed his anguish in the pages of his private journal: "Oh
+1	where do they make money in washington dc; all coinage is produced by the United States Mint . With production facilities in Washington , DC , and Fort Worth , Texas , the Bureau of Engraving and Printing is the largest producer of government security documents in the United States . </P>
+0	What did a researcher compare this process to?which makes it one of the highest rates of maternal mortality in the Americas. In wealthy developed nations, only nine women die for every 100,000 births. The five main causes of pregnancy-related deaths in Peru are hemorrhage, pre-eclampsia, infection, complications following abortion and obstructed birth, according to Peru's Ministry of Health figures. Amnesty's Peru researcher Nuria Garcia said, in a written statement: "The rates of maternal mortality in Peru are scandalous. The fact that so many women are dying from preventable causes is a human rights violation. "The Peruvian state is simply ignoring
+0	How many containers can Longtan Containers Port Area handle?Port of Nanjing is the largest inland port in China, with annual cargo tonnage reached 191,970,000 t in 2012. The port area is 98 kilometres (61 mi) in length and has 64 berths
+0	The 2011 New York City Marathon was sponsored by which Dutch multinational banking corporation?are retail banking, direct banking, commercial banking, investment banking, asset management, and insurance services. ING is an abbreviation for "Internationale Nederlanden Groep " (English: International Netherlands Group). [PAR] [TLE] 2011 New York City Marathon [SEP] The 42nd New York City Marathon took
+0	What is human flourishing?it does not involve believing that human nature is purely good or that all people can live up to the Humanist ideals without help. If anything, there is recognition that living up to one's potential is hard
+0	What was the result of Dida appealto play in next month's Champions League match at Shakhtar Donetsk after partially winning his appeal to UEFA against a two-match ban. Dida has had one game of his two-match ban suspended for a year following an appeal to UEFA. Brazilian Dida was also fined 60,000 Swiss francs by European football's ruling body following an incident involving a supporter during the Champions clash against Celtic in Scotland on October 3. The 34-year-old Brazilian was initially banned for two games for his theatrics following a Celtic fan's encroachment onto the pitch during the 2-1 defeat at Celtic
+1	What is more plentiful in capital projects?generates economic distortion in the public sector by diverting public investment into capital projects where bribes and kickbacks are more plentiful. Officials may increase the technical complexity of public sector projects to conceal or
+0	where were band greeted with cheers?the United States for a show in Stamford, Connecticut, on Tuesday, after they have "a few days off to recuperate," Robinson said. The trio was the opening act for Nelson until they were loudly booed in Toronto, a day after the actor-musician's bizarre interview with a CBC radio host. Ironically, the comments that offended Canadians included Thornton's assessment that they were "very reserved" and "it doesn't matter what you say to them." "It's mashed potatoes with no gravy," Thornton told CBC host Jian Ghomeshi. "We tend to play places where people throw things at each other and here they just sort of sit there," he said. Watch Thornton's interview » The audience at Thursday night's show in Toronto loudly booed the Boxmasters, with some shouts of "Here comes the gravy!" The Toronto Star newspaper reported. Thornton's remarks about
+0	What do Mexicans call Mexico City?the Federal District in Spanish: D.F., which is read "De-Efe"). They are formally called capitalinos (in reference to the city being the capital of the country), but "[p]erhaps because capitalino is the
+0	where does lock stock and barrel come fromindividual components one at a time . One craftsman made the `` lock '' which would have been a `` match lock '' , `` wheel lock '' , `` flint lock '' etc .
+1	who has the power to establish a prison system<P> The Federal Bureau of Prisons ( BOP ) is a United States federal law enforcement agency . A subdivision of
+0	what are south americas only 2 landlocked countriessuch countries , including five partially recognised states .

demo/demo3/log

+{'token_ids': [[-1, -1], 'int64'], 'label_ids': [[-1], 'int64']}
+<paddlepalm.backbone.ernie.ERNIE object at 0x7fcf583f53d0>
+{'token_ids': [[-1, -1], 'int64'], 'label_ids': [[-1], 'int64'], u'input_mask': [[-1, -1, 1], 'float32'], u'position_ids': [[-1, -1], 'int64'], u'task_ids': [[-1, -1], 'int64'], u'segment_ids': [[-1, -1], 'int64']}
+[debug] : 0, input_mask
+[debug] : 0, position_ids
+[debug] : 0, segment_ids
+[debug] : 0, task_ids
+[debug] : 0, token_ids
+[debug] : 0, senti_cls.label_ids
+[debug] : 0, print_token_ids_0.tmp_0
+[debug] : 0, word_embedding
+[debug] : 0, embedding_0.tmp_0
+[debug] : 0, pos_embedding
+[debug] : 0, embedding_1.tmp_0
+[debug] : 0, sent_embedding
+[debug] : 0, embedding_2.tmp_0
+[debug] : 0, tmp_0
+[debug] : 0, tmp_1
+[debug] : 0, task_embedding
+[debug] : 0, embedding_3.tmp_0
+[debug] : 0, tmp_2
+[debug] : 0, reduce_mean_0.tmp_0
+[debug] : 0, elementwise_sub_0
+[debug] : 0, square_0.tmp_0
+[debug] : 0, reduce_mean_1.tmp_0
+[debug] : 0, tmp_3
+[debug] : 0, rsqrt_0.tmp_0
+[debug] : 0, elementwise_mul_0
+[debug] : 0, pre_encoder_layer_norm_scale
+[debug] : 0, pre_encoder_layer_norm_bias
+[debug] : 0, elementwise_mul_1
+[debug] : 0, elementwise_add_0
+[debug] : 0, dropout_0.tmp_0
+[debug] : 0, dropout_0.tmp_1
+[debug] : 0, matmul_0.tmp_0
+[debug] : 0, scale_0.tmp_0
+[debug] : 0, stack_0.tmp_0
+[debug] : 0, encoder_layer_0_multi_head_att_query_fc.w_0
+[debug] : 0, fc_0.tmp_0
+[debug] : 0, encoder_layer_0_multi_head_att_query_fc.b_0
+[debug] : 0, fc_0.tmp_1
+[debug] : 0, encoder_layer_0_multi_head_att_key_fc.w_0
+[debug] : 0, fc_1.tmp_0
+[debug] : 0, encoder_layer_0_multi_head_att_key_fc.b_0
+[debug] : 0, fc_1.tmp_1
+[debug] : 0, encoder_layer_0_multi_head_att_value_fc.w_0
+[debug] : 0, fc_2.tmp_0
+[debug] : 0, encoder_layer_0_multi_head_att_value_fc.b_0
+[debug] : 0, fc_2.tmp_1
+[debug] : 0, reshape2_0.tmp_0
+[debug] : 0, transpose_0.tmp_0
+[debug] : 0, transpose_0.tmp_1
+[debug] : 0, reshape2_1.tmp_0
+[debug] : 0, transpose_1.tmp_0
+[debug] : 0, transpose_1.tmp_1
+[debug] : 0, reshape2_2.tmp_0
+[debug] : 0, transpose_2.tmp_0
+[debug] : 0, transpose_2.tmp_1
+[debug] : 0, scale_1.tmp_0
+[debug] : 0, matmul_1.tmp_0
+[debug] : 0, tmp_4
+[debug] : 0, softmax_0.tmp_0
+[debug] : 0, dropout_1.tmp_0
+[debug] : 0, dropout_1.tmp_1
+[debug] : 0, matmul_2.tmp_0
+[debug] : 0, transpose_3.tmp_0
+[debug] : 0, transpose_3.tmp_1
+[debug] : 0, reshape2_3.tmp_0
+[debug] : 0, encoder_layer_0_multi_head_att_output_fc.w_0
+[debug] : 0, fc_3.tmp_0
+[debug] : 0, encoder_layer_0_multi_head_att_output_fc.b_0
+[debug] : 0, fc_3.tmp_1
+[debug] : 0, dropout_2.tmp_0
+[debug] : 0, dropout_2.tmp_1
+[debug] : 0, tmp_5
+[debug] : 0, reduce_mean_2.tmp_0
+[debug] : 0, elementwise_sub_1
+[debug] : 0, square_1.tmp_0
+[debug] : 0, reduce_mean_3.tmp_0
+[debug] : 0, tmp_6
+[debug] : 0, rsqrt_1.tmp_0
+[debug] : 0, elementwise_mul_2
+[debug] : 0, encoder_layer_0_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_0_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_3
+[debug] : 0, elementwise_add_1
+[debug] : 0, encoder_layer_0_ffn_fc_0.w_0
+[debug] : 0, fc_4.tmp_0
+[debug] : 0, encoder_layer_0_ffn_fc_0.b_0
+[debug] : 0, fc_4.tmp_1
+[debug] : 0, fc_4.tmp_2
+[debug] : 0, encoder_layer_0_ffn_fc_1.w_0
+[debug] : 0, fc_5.tmp_0
+[debug] : 0, encoder_layer_0_ffn_fc_1.b_0
+[debug] : 0, fc_5.tmp_1
+[debug] : 0, dropout_3.tmp_0
+[debug] : 0, dropout_3.tmp_1
+[debug] : 0, tmp_7
+[debug] : 0, reduce_mean_4.tmp_0
+[debug] : 0, elementwise_sub_2
+[debug] : 0, square_2.tmp_0
+[debug] : 0, reduce_mean_5.tmp_0
+[debug] : 0, tmp_8
+[debug] : 0, rsqrt_2.tmp_0
+[debug] : 0, elementwise_mul_4
+[debug] : 0, encoder_layer_0_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_0_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_5
+[debug] : 0, elementwise_add_2
+[debug] : 0, encoder_layer_1_multi_head_att_query_fc.w_0
+[debug] : 0, fc_6.tmp_0
+[debug] : 0, encoder_layer_1_multi_head_att_query_fc.b_0
+[debug] : 0, fc_6.tmp_1
+[debug] : 0, encoder_layer_1_multi_head_att_key_fc.w_0
+[debug] : 0, fc_7.tmp_0
+[debug] : 0, encoder_layer_1_multi_head_att_key_fc.b_0
+[debug] : 0, fc_7.tmp_1
+[debug] : 0, encoder_layer_1_multi_head_att_value_fc.w_0
+[debug] : 0, fc_8.tmp_0
+[debug] : 0, encoder_layer_1_multi_head_att_value_fc.b_0
+[debug] : 0, fc_8.tmp_1
+[debug] : 0, reshape2_4.tmp_0
+[debug] : 0, transpose_4.tmp_0
+[debug] : 0, transpose_4.tmp_1
+[debug] : 0, reshape2_5.tmp_0
+[debug] : 0, transpose_5.tmp_0
+[debug] : 0, transpose_5.tmp_1
+[debug] : 0, reshape2_6.tmp_0
+[debug] : 0, transpose_6.tmp_0
+[debug] : 0, transpose_6.tmp_1
+[debug] : 0, scale_2.tmp_0
+[debug] : 0, matmul_3.tmp_0
+[debug] : 0, tmp_9
+[debug] : 0, softmax_1.tmp_0
+[debug] : 0, dropout_4.tmp_0
+[debug] : 0, dropout_4.tmp_1
+[debug] : 0, matmul_4.tmp_0
+[debug] : 0, transpose_7.tmp_0
+[debug] : 0, transpose_7.tmp_1
+[debug] : 0, reshape2_7.tmp_0
+[debug] : 0, encoder_layer_1_multi_head_att_output_fc.w_0
+[debug] : 0, fc_9.tmp_0
+[debug] : 0, encoder_layer_1_multi_head_att_output_fc.b_0
+[debug] : 0, fc_9.tmp_1
+[debug] : 0, dropout_5.tmp_0
+[debug] : 0, dropout_5.tmp_1
+[debug] : 0, tmp_10
+[debug] : 0, reduce_mean_6.tmp_0
+[debug] : 0, elementwise_sub_3
+[debug] : 0, square_3.tmp_0
+[debug] : 0, reduce_mean_7.tmp_0
+[debug] : 0, tmp_11
+[debug] : 0, rsqrt_3.tmp_0
+[debug] : 0, elementwise_mul_6
+[debug] : 0, encoder_layer_1_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_1_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_7
+[debug] : 0, elementwise_add_3
+[debug] : 0, encoder_layer_1_ffn_fc_0.w_0
+[debug] : 0, fc_10.tmp_0
+[debug] : 0, encoder_layer_1_ffn_fc_0.b_0
+[debug] : 0, fc_10.tmp_1
+[debug] : 0, fc_10.tmp_2
+[debug] : 0, encoder_layer_1_ffn_fc_1.w_0
+[debug] : 0, fc_11.tmp_0
+[debug] : 0, encoder_layer_1_ffn_fc_1.b_0
+[debug] : 0, fc_11.tmp_1
+[debug] : 0, dropout_6.tmp_0
+[debug] : 0, dropout_6.tmp_1
+[debug] : 0, tmp_12
+[debug] : 0, reduce_mean_8.tmp_0
+[debug] : 0, elementwise_sub_4
+[debug] : 0, square_4.tmp_0
+[debug] : 0, reduce_mean_9.tmp_0
+[debug] : 0, tmp_13
+[debug] : 0, rsqrt_4.tmp_0
+[debug] : 0, elementwise_mul_8
+[debug] : 0, encoder_layer_1_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_1_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_9
+[debug] : 0, elementwise_add_4
+[debug] : 0, encoder_layer_2_multi_head_att_query_fc.w_0
+[debug] : 0, fc_12.tmp_0
+[debug] : 0, encoder_layer_2_multi_head_att_query_fc.b_0
+[debug] : 0, fc_12.tmp_1
+[debug] : 0, encoder_layer_2_multi_head_att_key_fc.w_0
+[debug] : 0, fc_13.tmp_0
+[debug] : 0, encoder_layer_2_multi_head_att_key_fc.b_0
+[debug] : 0, fc_13.tmp_1
+[debug] : 0, encoder_layer_2_multi_head_att_value_fc.w_0
+[debug] : 0, fc_14.tmp_0
+[debug] : 0, encoder_layer_2_multi_head_att_value_fc.b_0
+[debug] : 0, fc_14.tmp_1
+[debug] : 0, reshape2_8.tmp_0
+[debug] : 0, transpose_8.tmp_0
+[debug] : 0, transpose_8.tmp_1
+[debug] : 0, reshape2_9.tmp_0
+[debug] : 0, transpose_9.tmp_0
+[debug] : 0, transpose_9.tmp_1
+[debug] : 0, reshape2_10.tmp_0
+[debug] : 0, transpose_10.tmp_0
+[debug] : 0, transpose_10.tmp_1
+[debug] : 0, scale_3.tmp_0
+[debug] : 0, matmul_5.tmp_0
+[debug] : 0, tmp_14
+[debug] : 0, softmax_2.tmp_0
+[debug] : 0, dropout_7.tmp_0
+[debug] : 0, dropout_7.tmp_1
+[debug] : 0, matmul_6.tmp_0
+[debug] : 0, transpose_11.tmp_0
+[debug] : 0, transpose_11.tmp_1
+[debug] : 0, reshape2_11.tmp_0
+[debug] : 0, encoder_layer_2_multi_head_att_output_fc.w_0
+[debug] : 0, fc_15.tmp_0
+[debug] : 0, encoder_layer_2_multi_head_att_output_fc.b_0
+[debug] : 0, fc_15.tmp_1
+[debug] : 0, dropout_8.tmp_0
+[debug] : 0, dropout_8.tmp_1
+[debug] : 0, tmp_15
+[debug] : 0, reduce_mean_10.tmp_0
+[debug] : 0, elementwise_sub_5
+[debug] : 0, square_5.tmp_0
+[debug] : 0, reduce_mean_11.tmp_0
+[debug] : 0, tmp_16
+[debug] : 0, rsqrt_5.tmp_0
+[debug] : 0, elementwise_mul_10
+[debug] : 0, encoder_layer_2_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_2_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_11
+[debug] : 0, elementwise_add_5
+[debug] : 0, encoder_layer_2_ffn_fc_0.w_0
+[debug] : 0, fc_16.tmp_0
+[debug] : 0, encoder_layer_2_ffn_fc_0.b_0
+[debug] : 0, fc_16.tmp_1
+[debug] : 0, fc_16.tmp_2
+[debug] : 0, encoder_layer_2_ffn_fc_1.w_0
+[debug] : 0, fc_17.tmp_0
+[debug] : 0, encoder_layer_2_ffn_fc_1.b_0
+[debug] : 0, fc_17.tmp_1
+[debug] : 0, dropout_9.tmp_0
+[debug] : 0, dropout_9.tmp_1
+[debug] : 0, tmp_17
+[debug] : 0, reduce_mean_12.tmp_0
+[debug] : 0, elementwise_sub_6
+[debug] : 0, square_6.tmp_0
+[debug] : 0, reduce_mean_13.tmp_0
+[debug] : 0, tmp_18
+[debug] : 0, rsqrt_6.tmp_0
+[debug] : 0, elementwise_mul_12
+[debug] : 0, encoder_layer_2_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_2_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_13
+[debug] : 0, elementwise_add_6
+[debug] : 0, encoder_layer_3_multi_head_att_query_fc.w_0
+[debug] : 0, fc_18.tmp_0
+[debug] : 0, encoder_layer_3_multi_head_att_query_fc.b_0
+[debug] : 0, fc_18.tmp_1
+[debug] : 0, encoder_layer_3_multi_head_att_key_fc.w_0
+[debug] : 0, fc_19.tmp_0
+[debug] : 0, encoder_layer_3_multi_head_att_key_fc.b_0
+[debug] : 0, fc_19.tmp_1
+[debug] : 0, encoder_layer_3_multi_head_att_value_fc.w_0
+[debug] : 0, fc_20.tmp_0
+[debug] : 0, encoder_layer_3_multi_head_att_value_fc.b_0
+[debug] : 0, fc_20.tmp_1
+[debug] : 0, reshape2_12.tmp_0
+[debug] : 0, transpose_12.tmp_0
+[debug] : 0, transpose_12.tmp_1
+[debug] : 0, reshape2_13.tmp_0
+[debug] : 0, transpose_13.tmp_0
+[debug] : 0, transpose_13.tmp_1
+[debug] : 0, reshape2_14.tmp_0
+[debug] : 0, transpose_14.tmp_0
+[debug] : 0, transpose_14.tmp_1
+[debug] : 0, scale_4.tmp_0
+[debug] : 0, matmul_7.tmp_0
+[debug] : 0, tmp_19
+[debug] : 0, softmax_3.tmp_0
+[debug] : 0, dropout_10.tmp_0
+[debug] : 0, dropout_10.tmp_1
+[debug] : 0, matmul_8.tmp_0
+[debug] : 0, transpose_15.tmp_0
+[debug] : 0, transpose_15.tmp_1
+[debug] : 0, reshape2_15.tmp_0
+[debug] : 0, encoder_layer_3_multi_head_att_output_fc.w_0
+[debug] : 0, fc_21.tmp_0
+[debug] : 0, encoder_layer_3_multi_head_att_output_fc.b_0
+[debug] : 0, fc_21.tmp_1
+[debug] : 0, dropout_11.tmp_0
+[debug] : 0, dropout_11.tmp_1
+[debug] : 0, tmp_20
+[debug] : 0, reduce_mean_14.tmp_0
+[debug] : 0, elementwise_sub_7
+[debug] : 0, square_7.tmp_0
+[debug] : 0, reduce_mean_15.tmp_0
+[debug] : 0, tmp_21
+[debug] : 0, rsqrt_7.tmp_0
+[debug] : 0, elementwise_mul_14
+[debug] : 0, encoder_layer_3_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_3_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_15
+[debug] : 0, elementwise_add_7
+[debug] : 0, encoder_layer_3_ffn_fc_0.w_0
+[debug] : 0, fc_22.tmp_0
+[debug] : 0, encoder_layer_3_ffn_fc_0.b_0
+[debug] : 0, fc_22.tmp_1
+[debug] : 0, fc_22.tmp_2
+[debug] : 0, encoder_layer_3_ffn_fc_1.w_0
+[debug] : 0, fc_23.tmp_0
+[debug] : 0, encoder_layer_3_ffn_fc_1.b_0
+[debug] : 0, fc_23.tmp_1
+[debug] : 0, dropout_12.tmp_0
+[debug] : 0, dropout_12.tmp_1
+[debug] : 0, tmp_22
+[debug] : 0, reduce_mean_16.tmp_0
+[debug] : 0, elementwise_sub_8
+[debug] : 0, square_8.tmp_0
+[debug] : 0, reduce_mean_17.tmp_0
+[debug] : 0, tmp_23
+[debug] : 0, rsqrt_8.tmp_0
+[debug] : 0, elementwise_mul_16
+[debug] : 0, encoder_layer_3_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_3_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_17
+[debug] : 0, elementwise_add_8
+[debug] : 0, encoder_layer_4_multi_head_att_query_fc.w_0
+[debug] : 0, fc_24.tmp_0
+[debug] : 0, encoder_layer_4_multi_head_att_query_fc.b_0
+[debug] : 0, fc_24.tmp_1
+[debug] : 0, encoder_layer_4_multi_head_att_key_fc.w_0
+[debug] : 0, fc_25.tmp_0
+[debug] : 0, encoder_layer_4_multi_head_att_key_fc.b_0
+[debug] : 0, fc_25.tmp_1
+[debug] : 0, encoder_layer_4_multi_head_att_value_fc.w_0
+[debug] : 0, fc_26.tmp_0
+[debug] : 0, encoder_layer_4_multi_head_att_value_fc.b_0
+[debug] : 0, fc_26.tmp_1
+[debug] : 0, reshape2_16.tmp_0
+[debug] : 0, transpose_16.tmp_0
+[debug] : 0, transpose_16.tmp_1
+[debug] : 0, reshape2_17.tmp_0
+[debug] : 0, transpose_17.tmp_0
+[debug] : 0, transpose_17.tmp_1
+[debug] : 0, reshape2_18.tmp_0
+[debug] : 0, transpose_18.tmp_0
+[debug] : 0, transpose_18.tmp_1
+[debug] : 0, scale_5.tmp_0
+[debug] : 0, matmul_9.tmp_0
+[debug] : 0, tmp_24
+[debug] : 0, softmax_4.tmp_0
+[debug] : 0, dropout_13.tmp_0
+[debug] : 0, dropout_13.tmp_1
+[debug] : 0, matmul_10.tmp_0
+[debug] : 0, transpose_19.tmp_0
+[debug] : 0, transpose_19.tmp_1
+[debug] : 0, reshape2_19.tmp_0
+[debug] : 0, encoder_layer_4_multi_head_att_output_fc.w_0
+[debug] : 0, fc_27.tmp_0
+[debug] : 0, encoder_layer_4_multi_head_att_output_fc.b_0
+[debug] : 0, fc_27.tmp_1
+[debug] : 0, dropout_14.tmp_0
+[debug] : 0, dropout_14.tmp_1
+[debug] : 0, tmp_25
+[debug] : 0, reduce_mean_18.tmp_0
+[debug] : 0, elementwise_sub_9
+[debug] : 0, square_9.tmp_0
+[debug] : 0, reduce_mean_19.tmp_0
+[debug] : 0, tmp_26
+[debug] : 0, rsqrt_9.tmp_0
+[debug] : 0, elementwise_mul_18
+[debug] : 0, encoder_layer_4_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_4_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_19
+[debug] : 0, elementwise_add_9
+[debug] : 0, encoder_layer_4_ffn_fc_0.w_0
+[debug] : 0, fc_28.tmp_0
+[debug] : 0, encoder_layer_4_ffn_fc_0.b_0
+[debug] : 0, fc_28.tmp_1
+[debug] : 0, fc_28.tmp_2
+[debug] : 0, encoder_layer_4_ffn_fc_1.w_0
+[debug] : 0, fc_29.tmp_0
+[debug] : 0, encoder_layer_4_ffn_fc_1.b_0
+[debug] : 0, fc_29.tmp_1
+[debug] : 0, dropout_15.tmp_0
+[debug] : 0, dropout_15.tmp_1
+[debug] : 0, tmp_27
+[debug] : 0, reduce_mean_20.tmp_0
+[debug] : 0, elementwise_sub_10
+[debug] : 0, square_10.tmp_0
+[debug] : 0, reduce_mean_21.tmp_0
+[debug] : 0, tmp_28
+[debug] : 0, rsqrt_10.tmp_0
+[debug] : 0, elementwise_mul_20
+[debug] : 0, encoder_layer_4_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_4_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_21
+[debug] : 0, elementwise_add_10
+[debug] : 0, encoder_layer_5_multi_head_att_query_fc.w_0
+[debug] : 0, fc_30.tmp_0
+[debug] : 0, encoder_layer_5_multi_head_att_query_fc.b_0
+[debug] : 0, fc_30.tmp_1
+[debug] : 0, encoder_layer_5_multi_head_att_key_fc.w_0
+[debug] : 0, fc_31.tmp_0
+[debug] : 0, encoder_layer_5_multi_head_att_key_fc.b_0
+[debug] : 0, fc_31.tmp_1
+[debug] : 0, encoder_layer_5_multi_head_att_value_fc.w_0
+[debug] : 0, fc_32.tmp_0
+[debug] : 0, encoder_layer_5_multi_head_att_value_fc.b_0
+[debug] : 0, fc_32.tmp_1
+[debug] : 0, reshape2_20.tmp_0
+[debug] : 0, transpose_20.tmp_0
+[debug] : 0, transpose_20.tmp_1
+[debug] : 0, reshape2_21.tmp_0
+[debug] : 0, transpose_21.tmp_0
+[debug] : 0, transpose_21.tmp_1
+[debug] : 0, reshape2_22.tmp_0
+[debug] : 0, transpose_22.tmp_0
+[debug] : 0, transpose_22.tmp_1
+[debug] : 0, scale_6.tmp_0
+[debug] : 0, matmul_11.tmp_0
+[debug] : 0, tmp_29
+[debug] : 0, softmax_5.tmp_0
+[debug] : 0, dropout_16.tmp_0
+[debug] : 0, dropout_16.tmp_1
+[debug] : 0, matmul_12.tmp_0
+[debug] : 0, transpose_23.tmp_0
+[debug] : 0, transpose_23.tmp_1
+[debug] : 0, reshape2_23.tmp_0
+[debug] : 0, encoder_layer_5_multi_head_att_output_fc.w_0
+[debug] : 0, fc_33.tmp_0
+[debug] : 0, encoder_layer_5_multi_head_att_output_fc.b_0
+[debug] : 0, fc_33.tmp_1
+[debug] : 0, dropout_17.tmp_0
+[debug] : 0, dropout_17.tmp_1
+[debug] : 0, tmp_30
+[debug] : 0, reduce_mean_22.tmp_0
+[debug] : 0, elementwise_sub_11
+[debug] : 0, square_11.tmp_0
+[debug] : 0, reduce_mean_23.tmp_0
+[debug] : 0, tmp_31
+[debug] : 0, rsqrt_11.tmp_0
+[debug] : 0, elementwise_mul_22
+[debug] : 0, encoder_layer_5_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_5_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_23
+[debug] : 0, elementwise_add_11
+[debug] : 0, encoder_layer_5_ffn_fc_0.w_0
+[debug] : 0, fc_34.tmp_0
+[debug] : 0, encoder_layer_5_ffn_fc_0.b_0
+[debug] : 0, fc_34.tmp_1
+[debug] : 0, fc_34.tmp_2
+[debug] : 0, encoder_layer_5_ffn_fc_1.w_0
+[debug] : 0, fc_35.tmp_0
+[debug] : 0, encoder_layer_5_ffn_fc_1.b_0
+[debug] : 0, fc_35.tmp_1
+[debug] : 0, dropout_18.tmp_0
+[debug] : 0, dropout_18.tmp_1
+[debug] : 0, tmp_32
+[debug] : 0, reduce_mean_24.tmp_0
+[debug] : 0, elementwise_sub_12
+[debug] : 0, square_12.tmp_0
+[debug] : 0, reduce_mean_25.tmp_0
+[debug] : 0, tmp_33
+[debug] : 0, rsqrt_12.tmp_0
+[debug] : 0, elementwise_mul_24
+[debug] : 0, encoder_layer_5_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_5_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_25
+[debug] : 0, elementwise_add_12
+[debug] : 0, encoder_layer_6_multi_head_att_query_fc.w_0
+[debug] : 0, fc_36.tmp_0
+[debug] : 0, encoder_layer_6_multi_head_att_query_fc.b_0
+[debug] : 0, fc_36.tmp_1
+[debug] : 0, encoder_layer_6_multi_head_att_key_fc.w_0
+[debug] : 0, fc_37.tmp_0
+[debug] : 0, encoder_layer_6_multi_head_att_key_fc.b_0
+[debug] : 0, fc_37.tmp_1
+[debug] : 0, encoder_layer_6_multi_head_att_value_fc.w_0
+[debug] : 0, fc_38.tmp_0
+[debug] : 0, encoder_layer_6_multi_head_att_value_fc.b_0
+[debug] : 0, fc_38.tmp_1
+[debug] : 0, reshape2_24.tmp_0
+[debug] : 0, transpose_24.tmp_0
+[debug] : 0, transpose_24.tmp_1
+[debug] : 0, reshape2_25.tmp_0
+[debug] : 0, transpose_25.tmp_0
+[debug] : 0, transpose_25.tmp_1
+[debug] : 0, reshape2_26.tmp_0
+[debug] : 0, transpose_26.tmp_0
+[debug] : 0, transpose_26.tmp_1
+[debug] : 0, scale_7.tmp_0
+[debug] : 0, matmul_13.tmp_0
+[debug] : 0, tmp_34
+[debug] : 0, softmax_6.tmp_0
+[debug] : 0, dropout_19.tmp_0
+[debug] : 0, dropout_19.tmp_1
+[debug] : 0, matmul_14.tmp_0
+[debug] : 0, transpose_27.tmp_0
+[debug] : 0, transpose_27.tmp_1
+[debug] : 0, reshape2_27.tmp_0
+[debug] : 0, encoder_layer_6_multi_head_att_output_fc.w_0
+[debug] : 0, fc_39.tmp_0
+[debug] : 0, encoder_layer_6_multi_head_att_output_fc.b_0
+[debug] : 0, fc_39.tmp_1
+[debug] : 0, dropout_20.tmp_0
+[debug] : 0, dropout_20.tmp_1
+[debug] : 0, tmp_35
+[debug] : 0, reduce_mean_26.tmp_0
+[debug] : 0, elementwise_sub_13
+[debug] : 0, square_13.tmp_0
+[debug] : 0, reduce_mean_27.tmp_0
+[debug] : 0, tmp_36
+[debug] : 0, rsqrt_13.tmp_0
+[debug] : 0, elementwise_mul_26
+[debug] : 0, encoder_layer_6_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_6_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_27
+[debug] : 0, elementwise_add_13
+[debug] : 0, encoder_layer_6_ffn_fc_0.w_0
+[debug] : 0, fc_40.tmp_0
+[debug] : 0, encoder_layer_6_ffn_fc_0.b_0
+[debug] : 0, fc_40.tmp_1
+[debug] : 0, fc_40.tmp_2
+[debug] : 0, encoder_layer_6_ffn_fc_1.w_0
+[debug] : 0, fc_41.tmp_0
+[debug] : 0, encoder_layer_6_ffn_fc_1.b_0
+[debug] : 0, fc_41.tmp_1
+[debug] : 0, dropout_21.tmp_0
+[debug] : 0, dropout_21.tmp_1
+[debug] : 0, tmp_37
+[debug] : 0, reduce_mean_28.tmp_0
+[debug] : 0, elementwise_sub_14
+[debug] : 0, square_14.tmp_0
+[debug] : 0, reduce_mean_29.tmp_0
+[debug] : 0, tmp_38
+[debug] : 0, rsqrt_14.tmp_0
+[debug] : 0, elementwise_mul_28
+[debug] : 0, encoder_layer_6_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_6_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_29
+[debug] : 0, elementwise_add_14
+[debug] : 0, encoder_layer_7_multi_head_att_query_fc.w_0
+[debug] : 0, fc_42.tmp_0
+[debug] : 0, encoder_layer_7_multi_head_att_query_fc.b_0
+[debug] : 0, fc_42.tmp_1
+[debug] : 0, encoder_layer_7_multi_head_att_key_fc.w_0
+[debug] : 0, fc_43.tmp_0
+[debug] : 0, encoder_layer_7_multi_head_att_key_fc.b_0
+[debug] : 0, fc_43.tmp_1
+[debug] : 0, encoder_layer_7_multi_head_att_value_fc.w_0
+[debug] : 0, fc_44.tmp_0
+[debug] : 0, encoder_layer_7_multi_head_att_value_fc.b_0
+[debug] : 0, fc_44.tmp_1
+[debug] : 0, reshape2_28.tmp_0
+[debug] : 0, transpose_28.tmp_0
+[debug] : 0, transpose_28.tmp_1
+[debug] : 0, reshape2_29.tmp_0
+[debug] : 0, transpose_29.tmp_0
+[debug] : 0, transpose_29.tmp_1
+[debug] : 0, reshape2_30.tmp_0
+[debug] : 0, transpose_30.tmp_0
+[debug] : 0, transpose_30.tmp_1
+[debug] : 0, scale_8.tmp_0
+[debug] : 0, matmul_15.tmp_0
+[debug] : 0, tmp_39
+[debug] : 0, softmax_7.tmp_0
+[debug] : 0, dropout_22.tmp_0
+[debug] : 0, dropout_22.tmp_1
+[debug] : 0, matmul_16.tmp_0
+[debug] : 0, transpose_31.tmp_0
+[debug] : 0, transpose_31.tmp_1
+[debug] : 0, reshape2_31.tmp_0
+[debug] : 0, encoder_layer_7_multi_head_att_output_fc.w_0
+[debug] : 0, fc_45.tmp_0
+[debug] : 0, encoder_layer_7_multi_head_att_output_fc.b_0
+[debug] : 0, fc_45.tmp_1
+[debug] : 0, dropout_23.tmp_0
+[debug] : 0, dropout_23.tmp_1
+[debug] : 0, tmp_40
+[debug] : 0, reduce_mean_30.tmp_0
+[debug] : 0, elementwise_sub_15
+[debug] : 0, square_15.tmp_0
+[debug] : 0, reduce_mean_31.tmp_0
+[debug] : 0, tmp_41
+[debug] : 0, rsqrt_15.tmp_0
+[debug] : 0, elementwise_mul_30
+[debug] : 0, encoder_layer_7_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_7_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_31
+[debug] : 0, elementwise_add_15
+[debug] : 0, encoder_layer_7_ffn_fc_0.w_0
+[debug] : 0, fc_46.tmp_0
+[debug] : 0, encoder_layer_7_ffn_fc_0.b_0
+[debug] : 0, fc_46.tmp_1
+[debug] : 0, fc_46.tmp_2
+[debug] : 0, encoder_layer_7_ffn_fc_1.w_0
+[debug] : 0, fc_47.tmp_0
+[debug] : 0, encoder_layer_7_ffn_fc_1.b_0
+[debug] : 0, fc_47.tmp_1
+[debug] : 0, dropout_24.tmp_0
+[debug] : 0, dropout_24.tmp_1
+[debug] : 0, tmp_42
+[debug] : 0, reduce_mean_32.tmp_0
+[debug] : 0, elementwise_sub_16
+[debug] : 0, square_16.tmp_0
+[debug] : 0, reduce_mean_33.tmp_0
+[debug] : 0, tmp_43
+[debug] : 0, rsqrt_16.tmp_0
+[debug] : 0, elementwise_mul_32
+[debug] : 0, encoder_layer_7_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_7_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_33
+[debug] : 0, elementwise_add_16
+[debug] : 0, encoder_layer_8_multi_head_att_query_fc.w_0
+[debug] : 0, fc_48.tmp_0
+[debug] : 0, encoder_layer_8_multi_head_att_query_fc.b_0
+[debug] : 0, fc_48.tmp_1
+[debug] : 0, encoder_layer_8_multi_head_att_key_fc.w_0
+[debug] : 0, fc_49.tmp_0
+[debug] : 0, encoder_layer_8_multi_head_att_key_fc.b_0
+[debug] : 0, fc_49.tmp_1
+[debug] : 0, encoder_layer_8_multi_head_att_value_fc.w_0
+[debug] : 0, fc_50.tmp_0
+[debug] : 0, encoder_layer_8_multi_head_att_value_fc.b_0
+[debug] : 0, fc_50.tmp_1
+[debug] : 0, reshape2_32.tmp_0
+[debug] : 0, transpose_32.tmp_0
+[debug] : 0, transpose_32.tmp_1
+[debug] : 0, reshape2_33.tmp_0
+[debug] : 0, transpose_33.tmp_0
+[debug] : 0, transpose_33.tmp_1
+[debug] : 0, reshape2_34.tmp_0
+[debug] : 0, transpose_34.tmp_0
+[debug] : 0, transpose_34.tmp_1
+[debug] : 0, scale_9.tmp_0
+[debug] : 0, matmul_17.tmp_0
+[debug] : 0, tmp_44
+[debug] : 0, softmax_8.tmp_0
+[debug] : 0, dropout_25.tmp_0
+[debug] : 0, dropout_25.tmp_1
+[debug] : 0, matmul_18.tmp_0
+[debug] : 0, transpose_35.tmp_0
+[debug] : 0, transpose_35.tmp_1
+[debug] : 0, reshape2_35.tmp_0
+[debug] : 0, encoder_layer_8_multi_head_att_output_fc.w_0
+[debug] : 0, fc_51.tmp_0
+[debug] : 0, encoder_layer_8_multi_head_att_output_fc.b_0
+[debug] : 0, fc_51.tmp_1
+[debug] : 0, dropout_26.tmp_0
+[debug] : 0, dropout_26.tmp_1
+[debug] : 0, tmp_45
+[debug] : 0, reduce_mean_34.tmp_0
+[debug] : 0, elementwise_sub_17
+[debug] : 0, square_17.tmp_0
+[debug] : 0, reduce_mean_35.tmp_0
+[debug] : 0, tmp_46
+[debug] : 0, rsqrt_17.tmp_0
+[debug] : 0, elementwise_mul_34
+[debug] : 0, encoder_layer_8_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_8_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_35
+[debug] : 0, elementwise_add_17
+[debug] : 0, encoder_layer_8_ffn_fc_0.w_0
+[debug] : 0, fc_52.tmp_0
+[debug] : 0, encoder_layer_8_ffn_fc_0.b_0
+[debug] : 0, fc_52.tmp_1
+[debug] : 0, fc_52.tmp_2
+[debug] : 0, encoder_layer_8_ffn_fc_1.w_0
+[debug] : 0, fc_53.tmp_0
+[debug] : 0, encoder_layer_8_ffn_fc_1.b_0
+[debug] : 0, fc_53.tmp_1
+[debug] : 0, dropout_27.tmp_0
+[debug] : 0, dropout_27.tmp_1
+[debug] : 0, tmp_47
+[debug] : 0, reduce_mean_36.tmp_0
+[debug] : 0, elementwise_sub_18
+[debug] : 0, square_18.tmp_0
+[debug] : 0, reduce_mean_37.tmp_0
+[debug] : 0, tmp_48
+[debug] : 0, rsqrt_18.tmp_0
+[debug] : 0, elementwise_mul_36
+[debug] : 0, encoder_layer_8_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_8_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_37
+[debug] : 0, elementwise_add_18
+[debug] : 0, encoder_layer_9_multi_head_att_query_fc.w_0
+[debug] : 0, fc_54.tmp_0
+[debug] : 0, encoder_layer_9_multi_head_att_query_fc.b_0
+[debug] : 0, fc_54.tmp_1
+[debug] : 0, encoder_layer_9_multi_head_att_key_fc.w_0
+[debug] : 0, fc_55.tmp_0
+[debug] : 0, encoder_layer_9_multi_head_att_key_fc.b_0
+[debug] : 0, fc_55.tmp_1
+[debug] : 0, encoder_layer_9_multi_head_att_value_fc.w_0
+[debug] : 0, fc_56.tmp_0
+[debug] : 0, encoder_layer_9_multi_head_att_value_fc.b_0
+[debug] : 0, fc_56.tmp_1
+[debug] : 0, reshape2_36.tmp_0
+[debug] : 0, transpose_36.tmp_0
+[debug] : 0, transpose_36.tmp_1
+[debug] : 0, reshape2_37.tmp_0
+[debug] : 0, transpose_37.tmp_0
+[debug] : 0, transpose_37.tmp_1
+[debug] : 0, reshape2_38.tmp_0
+[debug] : 0, transpose_38.tmp_0
+[debug] : 0, transpose_38.tmp_1
+[debug] : 0, scale_10.tmp_0
+[debug] : 0, matmul_19.tmp_0
+[debug] : 0, tmp_49
+[debug] : 0, softmax_9.tmp_0
+[debug] : 0, dropout_28.tmp_0
+[debug] : 0, dropout_28.tmp_1
+[debug] : 0, matmul_20.tmp_0
+[debug] : 0, transpose_39.tmp_0
+[debug] : 0, transpose_39.tmp_1
+[debug] : 0, reshape2_39.tmp_0
+[debug] : 0, encoder_layer_9_multi_head_att_output_fc.w_0
+[debug] : 0, fc_57.tmp_0
+[debug] : 0, encoder_layer_9_multi_head_att_output_fc.b_0
+[debug] : 0, fc_57.tmp_1
+[debug] : 0, dropout_29.tmp_0
+[debug] : 0, dropout_29.tmp_1
+[debug] : 0, tmp_50
+[debug] : 0, reduce_mean_38.tmp_0
+[debug] : 0, elementwise_sub_19
+[debug] : 0, square_19.tmp_0
+[debug] : 0, reduce_mean_39.tmp_0
+[debug] : 0, tmp_51
+[debug] : 0, rsqrt_19.tmp_0
+[debug] : 0, elementwise_mul_38
+[debug] : 0, encoder_layer_9_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_9_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_39
+[debug] : 0, elementwise_add_19
+[debug] : 0, encoder_layer_9_ffn_fc_0.w_0
+[debug] : 0, fc_58.tmp_0
+[debug] : 0, encoder_layer_9_ffn_fc_0.b_0
+[debug] : 0, fc_58.tmp_1
+[debug] : 0, fc_58.tmp_2
+[debug] : 0, encoder_layer_9_ffn_fc_1.w_0
+[debug] : 0, fc_59.tmp_0
+[debug] : 0, encoder_layer_9_ffn_fc_1.b_0
+[debug] : 0, fc_59.tmp_1
+[debug] : 0, dropout_30.tmp_0
+[debug] : 0, dropout_30.tmp_1
+[debug] : 0, tmp_52
+[debug] : 0, reduce_mean_40.tmp_0
+[debug] : 0, elementwise_sub_20
+[debug] : 0, square_20.tmp_0
+[debug] : 0, reduce_mean_41.tmp_0
+[debug] : 0, tmp_53
+[debug] : 0, rsqrt_20.tmp_0
+[debug] : 0, elementwise_mul_40
+[debug] : 0, encoder_layer_9_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_9_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_41
+[debug] : 0, elementwise_add_20
+[debug] : 0, encoder_layer_10_multi_head_att_query_fc.w_0
+[debug] : 0, fc_60.tmp_0
+[debug] : 0, encoder_layer_10_multi_head_att_query_fc.b_0
+[debug] : 0, fc_60.tmp_1
+[debug] : 0, encoder_layer_10_multi_head_att_key_fc.w_0
+[debug] : 0, fc_61.tmp_0
+[debug] : 0, encoder_layer_10_multi_head_att_key_fc.b_0
+[debug] : 0, fc_61.tmp_1
+[debug] : 0, encoder_layer_10_multi_head_att_value_fc.w_0
+[debug] : 0, fc_62.tmp_0
+[debug] : 0, encoder_layer_10_multi_head_att_value_fc.b_0
+[debug] : 0, fc_62.tmp_1
+[debug] : 0, reshape2_40.tmp_0
+[debug] : 0, transpose_40.tmp_0
+[debug] : 0, transpose_40.tmp_1
+[debug] : 0, reshape2_41.tmp_0
+[debug] : 0, transpose_41.tmp_0
+[debug] : 0, transpose_41.tmp_1
+[debug] : 0, reshape2_42.tmp_0
+[debug] : 0, transpose_42.tmp_0
+[debug] : 0, transpose_42.tmp_1
+[debug] : 0, scale_11.tmp_0
+[debug] : 0, matmul_21.tmp_0
+[debug] : 0, tmp_54
+[debug] : 0, softmax_10.tmp_0
+[debug] : 0, dropout_31.tmp_0
+[debug] : 0, dropout_31.tmp_1
+[debug] : 0, matmul_22.tmp_0
+[debug] : 0, transpose_43.tmp_0
+[debug] : 0, transpose_43.tmp_1
+[debug] : 0, reshape2_43.tmp_0
+[debug] : 0, encoder_layer_10_multi_head_att_output_fc.w_0
+[debug] : 0, fc_63.tmp_0
+[debug] : 0, encoder_layer_10_multi_head_att_output_fc.b_0
+[debug] : 0, fc_63.tmp_1
+[debug] : 0, dropout_32.tmp_0
+[debug] : 0, dropout_32.tmp_1
+[debug] : 0, tmp_55
+[debug] : 0, reduce_mean_42.tmp_0
+[debug] : 0, elementwise_sub_21
+[debug] : 0, square_21.tmp_0
+[debug] : 0, reduce_mean_43.tmp_0
+[debug] : 0, tmp_56
+[debug] : 0, rsqrt_21.tmp_0
+[debug] : 0, elementwise_mul_42
+[debug] : 0, encoder_layer_10_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_10_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_43
+[debug] : 0, elementwise_add_21
+[debug] : 0, encoder_layer_10_ffn_fc_0.w_0
+[debug] : 0, fc_64.tmp_0
+[debug] : 0, encoder_layer_10_ffn_fc_0.b_0
+[debug] : 0, fc_64.tmp_1
+[debug] : 0, fc_64.tmp_2
+[debug] : 0, encoder_layer_10_ffn_fc_1.w_0
+[debug] : 0, fc_65.tmp_0
+[debug] : 0, encoder_layer_10_ffn_fc_1.b_0
+[debug] : 0, fc_65.tmp_1
+[debug] : 0, dropout_33.tmp_0
+[debug] : 0, dropout_33.tmp_1
+[debug] : 0, tmp_57
+[debug] : 0, reduce_mean_44.tmp_0
+[debug] : 0, elementwise_sub_22
+[debug] : 0, square_22.tmp_0
+[debug] : 0, reduce_mean_45.tmp_0
+[debug] : 0, tmp_58
+[debug] : 0, rsqrt_22.tmp_0
+[debug] : 0, elementwise_mul_44
+[debug] : 0, encoder_layer_10_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_10_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_45
+[debug] : 0, elementwise_add_22
+[debug] : 0, encoder_layer_11_multi_head_att_query_fc.w_0
+[debug] : 0, fc_66.tmp_0
+[debug] : 0, encoder_layer_11_multi_head_att_query_fc.b_0
+[debug] : 0, fc_66.tmp_1
+[debug] : 0, encoder_layer_11_multi_head_att_key_fc.w_0
+[debug] : 0, fc_67.tmp_0
+[debug] : 0, encoder_layer_11_multi_head_att_key_fc.b_0
+[debug] : 0, fc_67.tmp_1
+[debug] : 0, encoder_layer_11_multi_head_att_value_fc.w_0
+[debug] : 0, fc_68.tmp_0
+[debug] : 0, encoder_layer_11_multi_head_att_value_fc.b_0
+[debug] : 0, fc_68.tmp_1
+[debug] : 0, reshape2_44.tmp_0
+[debug] : 0, transpose_44.tmp_0
+[debug] : 0, transpose_44.tmp_1
+[debug] : 0, reshape2_45.tmp_0
+[debug] : 0, transpose_45.tmp_0
+[debug] : 0, transpose_45.tmp_1
+[debug] : 0, reshape2_46.tmp_0
+[debug] : 0, transpose_46.tmp_0
+[debug] : 0, transpose_46.tmp_1
+[debug] : 0, scale_12.tmp_0
+[debug] : 0, matmul_23.tmp_0
+[debug] : 0, tmp_59
+[debug] : 0, softmax_11.tmp_0
+[debug] : 0, dropout_34.tmp_0
+[debug] : 0, dropout_34.tmp_1
+[debug] : 0, matmul_24.tmp_0
+[debug] : 0, transpose_47.tmp_0
+[debug] : 0, transpose_47.tmp_1
+[debug] : 0, reshape2_47.tmp_0
+[debug] : 0, encoder_layer_11_multi_head_att_output_fc.w_0
+[debug] : 0, fc_69.tmp_0
+[debug] : 0, encoder_layer_11_multi_head_att_output_fc.b_0
+[debug] : 0, fc_69.tmp_1
+[debug] : 0, dropout_35.tmp_0
+[debug] : 0, dropout_35.tmp_1
+[debug] : 0, tmp_60
+[debug] : 0, reduce_mean_46.tmp_0
+[debug] : 0, elementwise_sub_23
+[debug] : 0, square_23.tmp_0
+[debug] : 0, reduce_mean_47.tmp_0
+[debug] : 0, tmp_61
+[debug] : 0, rsqrt_23.tmp_0
+[debug] : 0, elementwise_mul_46
+[debug] : 0, encoder_layer_11_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_11_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_47
+[debug] : 0, elementwise_add_23
+[debug] : 0, encoder_layer_11_ffn_fc_0.w_0
+[debug] : 0, fc_70.tmp_0
+[debug] : 0, encoder_layer_11_ffn_fc_0.b_0
+[debug] : 0, fc_70.tmp_1
+[debug] : 0, fc_70.tmp_2
+[debug] : 0, encoder_layer_11_ffn_fc_1.w_0
+[debug] : 0, fc_71.tmp_0
+[debug] : 0, encoder_layer_11_ffn_fc_1.b_0
+[debug] : 0, fc_71.tmp_1
+[debug] : 0, dropout_36.tmp_0
+[debug] : 0, dropout_36.tmp_1
+[debug] : 0, tmp_62
+[debug] : 0, reduce_mean_48.tmp_0
+[debug] : 0, elementwise_sub_24
+[debug] : 0, square_24.tmp_0
+[debug] : 0, reduce_mean_49.tmp_0
+[debug] : 0, tmp_63
+[debug] : 0, rsqrt_24.tmp_0
+[debug] : 0, elementwise_mul_48
+[debug] : 0, encoder_layer_11_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_11_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_49
+[debug] : 0, elementwise_add_24
+[debug] : 0, encoder_layer_12_multi_head_att_query_fc.w_0
+[debug] : 0, fc_72.tmp_0
+[debug] : 0, encoder_layer_12_multi_head_att_query_fc.b_0
+[debug] : 0, fc_72.tmp_1
+[debug] : 0, encoder_layer_12_multi_head_att_key_fc.w_0
+[debug] : 0, fc_73.tmp_0
+[debug] : 0, encoder_layer_12_multi_head_att_key_fc.b_0
+[debug] : 0, fc_73.tmp_1
+[debug] : 0, encoder_layer_12_multi_head_att_value_fc.w_0
+[debug] : 0, fc_74.tmp_0
+[debug] : 0, encoder_layer_12_multi_head_att_value_fc.b_0
+[debug] : 0, fc_74.tmp_1
+[debug] : 0, reshape2_48.tmp_0
+[debug] : 0, transpose_48.tmp_0
+[debug] : 0, transpose_48.tmp_1
+[debug] : 0, reshape2_49.tmp_0
+[debug] : 0, transpose_49.tmp_0
+[debug] : 0, transpose_49.tmp_1
+[debug] : 0, reshape2_50.tmp_0
+[debug] : 0, transpose_50.tmp_0
+[debug] : 0, transpose_50.tmp_1
+[debug] : 0, scale_13.tmp_0
+[debug] : 0, matmul_25.tmp_0
+[debug] : 0, tmp_64
+[debug] : 0, softmax_12.tmp_0
+[debug] : 0, dropout_37.tmp_0
+[debug] : 0, dropout_37.tmp_1
+[debug] : 0, matmul_26.tmp_0
+[debug] : 0, transpose_51.tmp_0
+[debug] : 0, transpose_51.tmp_1
+[debug] : 0, reshape2_51.tmp_0
+[debug] : 0, encoder_layer_12_multi_head_att_output_fc.w_0
+[debug] : 0, fc_75.tmp_0
+[debug] : 0, encoder_layer_12_multi_head_att_output_fc.b_0
+[debug] : 0, fc_75.tmp_1
+[debug] : 0, dropout_38.tmp_0
+[debug] : 0, dropout_38.tmp_1
+[debug] : 0, tmp_65
+[debug] : 0, reduce_mean_50.tmp_0
+[debug] : 0, elementwise_sub_25
+[debug] : 0, square_25.tmp_0
+[debug] : 0, reduce_mean_51.tmp_0
+[debug] : 0, tmp_66
+[debug] : 0, rsqrt_25.tmp_0
+[debug] : 0, elementwise_mul_50
+[debug] : 0, encoder_layer_12_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_12_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_51
+[debug] : 0, elementwise_add_25
+[debug] : 0, encoder_layer_12_ffn_fc_0.w_0
+[debug] : 0, fc_76.tmp_0
+[debug] : 0, encoder_layer_12_ffn_fc_0.b_0
+[debug] : 0, fc_76.tmp_1
+[debug] : 0, fc_76.tmp_2
+[debug] : 0, encoder_layer_12_ffn_fc_1.w_0
+[debug] : 0, fc_77.tmp_0
+[debug] : 0, encoder_layer_12_ffn_fc_1.b_0
+[debug] : 0, fc_77.tmp_1
+[debug] : 0, dropout_39.tmp_0
+[debug] : 0, dropout_39.tmp_1
+[debug] : 0, tmp_67
+[debug] : 0, reduce_mean_52.tmp_0
+[debug] : 0, elementwise_sub_26
+[debug] : 0, square_26.tmp_0
+[debug] : 0, reduce_mean_53.tmp_0
+[debug] : 0, tmp_68
+[debug] : 0, rsqrt_26.tmp_0
+[debug] : 0, elementwise_mul_52
+[debug] : 0, encoder_layer_12_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_12_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_53
+[debug] : 0, elementwise_add_26
+[debug] : 0, encoder_layer_13_multi_head_att_query_fc.w_0
+[debug] : 0, fc_78.tmp_0
+[debug] : 0, encoder_layer_13_multi_head_att_query_fc.b_0
+[debug] : 0, fc_78.tmp_1
+[debug] : 0, encoder_layer_13_multi_head_att_key_fc.w_0
+[debug] : 0, fc_79.tmp_0
+[debug] : 0, encoder_layer_13_multi_head_att_key_fc.b_0
+[debug] : 0, fc_79.tmp_1
+[debug] : 0, encoder_layer_13_multi_head_att_value_fc.w_0
+[debug] : 0, fc_80.tmp_0
+[debug] : 0, encoder_layer_13_multi_head_att_value_fc.b_0
+[debug] : 0, fc_80.tmp_1
+[debug] : 0, reshape2_52.tmp_0
+[debug] : 0, transpose_52.tmp_0
+[debug] : 0, transpose_52.tmp_1
+[debug] : 0, reshape2_53.tmp_0
+[debug] : 0, transpose_53.tmp_0
+[debug] : 0, transpose_53.tmp_1
+[debug] : 0, reshape2_54.tmp_0
+[debug] : 0, transpose_54.tmp_0
+[debug] : 0, transpose_54.tmp_1
+[debug] : 0, scale_14.tmp_0
+[debug] : 0, matmul_27.tmp_0
+[debug] : 0, tmp_69
+[debug] : 0, softmax_13.tmp_0
+[debug] : 0, dropout_40.tmp_0
+[debug] : 0, dropout_40.tmp_1
+[debug] : 0, matmul_28.tmp_0
+[debug] : 0, transpose_55.tmp_0
+[debug] : 0, transpose_55.tmp_1
+[debug] : 0, reshape2_55.tmp_0
+[debug] : 0, encoder_layer_13_multi_head_att_output_fc.w_0
+[debug] : 0, fc_81.tmp_0
+[debug] : 0, encoder_layer_13_multi_head_att_output_fc.b_0
+[debug] : 0, fc_81.tmp_1
+[debug] : 0, dropout_41.tmp_0
+[debug] : 0, dropout_41.tmp_1
+[debug] : 0, tmp_70
+[debug] : 0, reduce_mean_54.tmp_0
+[debug] : 0, elementwise_sub_27
+[debug] : 0, square_27.tmp_0
+[debug] : 0, reduce_mean_55.tmp_0
+[debug] : 0, tmp_71
+[debug] : 0, rsqrt_27.tmp_0
+[debug] : 0, elementwise_mul_54
+[debug] : 0, encoder_layer_13_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_13_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_55
+[debug] : 0, elementwise_add_27
+[debug] : 0, encoder_layer_13_ffn_fc_0.w_0
+[debug] : 0, fc_82.tmp_0
+[debug] : 0, encoder_layer_13_ffn_fc_0.b_0
+[debug] : 0, fc_82.tmp_1
+[debug] : 0, fc_82.tmp_2
+[debug] : 0, encoder_layer_13_ffn_fc_1.w_0
+[debug] : 0, fc_83.tmp_0
+[debug] : 0, encoder_layer_13_ffn_fc_1.b_0
+[debug] : 0, fc_83.tmp_1
+[debug] : 0, dropout_42.tmp_0
+[debug] : 0, dropout_42.tmp_1
+[debug] : 0, tmp_72
+[debug] : 0, reduce_mean_56.tmp_0
+[debug] : 0, elementwise_sub_28
+[debug] : 0, square_28.tmp_0
+[debug] : 0, reduce_mean_57.tmp_0
+[debug] : 0, tmp_73
+[debug] : 0, rsqrt_28.tmp_0
+[debug] : 0, elementwise_mul_56
+[debug] : 0, encoder_layer_13_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_13_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_57
+[debug] : 0, elementwise_add_28
+[debug] : 0, encoder_layer_14_multi_head_att_query_fc.w_0
+[debug] : 0, fc_84.tmp_0
+[debug] : 0, encoder_layer_14_multi_head_att_query_fc.b_0
+[debug] : 0, fc_84.tmp_1
+[debug] : 0, encoder_layer_14_multi_head_att_key_fc.w_0
+[debug] : 0, fc_85.tmp_0
+[debug] : 0, encoder_layer_14_multi_head_att_key_fc.b_0
+[debug] : 0, fc_85.tmp_1
+[debug] : 0, encoder_layer_14_multi_head_att_value_fc.w_0
+[debug] : 0, fc_86.tmp_0
+[debug] : 0, encoder_layer_14_multi_head_att_value_fc.b_0
+[debug] : 0, fc_86.tmp_1
+[debug] : 0, reshape2_56.tmp_0
+[debug] : 0, transpose_56.tmp_0
+[debug] : 0, transpose_56.tmp_1
+[debug] : 0, reshape2_57.tmp_0
+[debug] : 0, transpose_57.tmp_0
+[debug] : 0, transpose_57.tmp_1
+[debug] : 0, reshape2_58.tmp_0
+[debug] : 0, transpose_58.tmp_0
+[debug] : 0, transpose_58.tmp_1
+[debug] : 0, scale_15.tmp_0
+[debug] : 0, matmul_29.tmp_0
+[debug] : 0, tmp_74
+[debug] : 0, softmax_14.tmp_0
+[debug] : 0, dropout_43.tmp_0
+[debug] : 0, dropout_43.tmp_1
+[debug] : 0, matmul_30.tmp_0
+[debug] : 0, transpose_59.tmp_0
+[debug] : 0, transpose_59.tmp_1
+[debug] : 0, reshape2_59.tmp_0
+[debug] : 0, encoder_layer_14_multi_head_att_output_fc.w_0
+[debug] : 0, fc_87.tmp_0
+[debug] : 0, encoder_layer_14_multi_head_att_output_fc.b_0
+[debug] : 0, fc_87.tmp_1
+[debug] : 0, dropout_44.tmp_0
+[debug] : 0, dropout_44.tmp_1
+[debug] : 0, tmp_75
+[debug] : 0, reduce_mean_58.tmp_0
+[debug] : 0, elementwise_sub_29
+[debug] : 0, square_29.tmp_0
+[debug] : 0, reduce_mean_59.tmp_0
+[debug] : 0, tmp_76
+[debug] : 0, rsqrt_29.tmp_0
+[debug] : 0, elementwise_mul_58
+[debug] : 0, encoder_layer_14_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_14_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_59
+[debug] : 0, elementwise_add_29
+[debug] : 0, encoder_layer_14_ffn_fc_0.w_0
+[debug] : 0, fc_88.tmp_0
+[debug] : 0, encoder_layer_14_ffn_fc_0.b_0
+[debug] : 0, fc_88.tmp_1
+[debug] : 0, fc_88.tmp_2
+[debug] : 0, encoder_layer_14_ffn_fc_1.w_0
+[debug] : 0, fc_89.tmp_0
+[debug] : 0, encoder_layer_14_ffn_fc_1.b_0
+[debug] : 0, fc_89.tmp_1
+[debug] : 0, dropout_45.tmp_0
+[debug] : 0, dropout_45.tmp_1
+[debug] : 0, tmp_77
+[debug] : 0, reduce_mean_60.tmp_0
+[debug] : 0, elementwise_sub_30
+[debug] : 0, square_30.tmp_0
+[debug] : 0, reduce_mean_61.tmp_0
+[debug] : 0, tmp_78
+[debug] : 0, rsqrt_30.tmp_0
+[debug] : 0, elementwise_mul_60
+[debug] : 0, encoder_layer_14_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_14_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_61
+[debug] : 0, elementwise_add_30
+[debug] : 0, encoder_layer_15_multi_head_att_query_fc.w_0
+[debug] : 0, fc_90.tmp_0
+[debug] : 0, encoder_layer_15_multi_head_att_query_fc.b_0
+[debug] : 0, fc_90.tmp_1
+[debug] : 0, encoder_layer_15_multi_head_att_key_fc.w_0
+[debug] : 0, fc_91.tmp_0
+[debug] : 0, encoder_layer_15_multi_head_att_key_fc.b_0
+[debug] : 0, fc_91.tmp_1
+[debug] : 0, encoder_layer_15_multi_head_att_value_fc.w_0
+[debug] : 0, fc_92.tmp_0
+[debug] : 0, encoder_layer_15_multi_head_att_value_fc.b_0
+[debug] : 0, fc_92.tmp_1
+[debug] : 0, reshape2_60.tmp_0
+[debug] : 0, transpose_60.tmp_0
+[debug] : 0, transpose_60.tmp_1
+[debug] : 0, reshape2_61.tmp_0
+[debug] : 0, transpose_61.tmp_0
+[debug] : 0, transpose_61.tmp_1
+[debug] : 0, reshape2_62.tmp_0
+[debug] : 0, transpose_62.tmp_0
+[debug] : 0, transpose_62.tmp_1
+[debug] : 0, scale_16.tmp_0
+[debug] : 0, matmul_31.tmp_0
+[debug] : 0, tmp_79
+[debug] : 0, softmax_15.tmp_0
+[debug] : 0, dropout_46.tmp_0
+[debug] : 0, dropout_46.tmp_1
+[debug] : 0, matmul_32.tmp_0
+[debug] : 0, transpose_63.tmp_0
+[debug] : 0, transpose_63.tmp_1
+[debug] : 0, reshape2_63.tmp_0
+[debug] : 0, encoder_layer_15_multi_head_att_output_fc.w_0
+[debug] : 0, fc_93.tmp_0
+[debug] : 0, encoder_layer_15_multi_head_att_output_fc.b_0
+[debug] : 0, fc_93.tmp_1
+[debug] : 0, dropout_47.tmp_0
+[debug] : 0, dropout_47.tmp_1
+[debug] : 0, tmp_80
+[debug] : 0, reduce_mean_62.tmp_0
+[debug] : 0, elementwise_sub_31
+[debug] : 0, square_31.tmp_0
+[debug] : 0, reduce_mean_63.tmp_0
+[debug] : 0, tmp_81
+[debug] : 0, rsqrt_31.tmp_0
+[debug] : 0, elementwise_mul_62
+[debug] : 0, encoder_layer_15_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_15_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_63
+[debug] : 0, elementwise_add_31
+[debug] : 0, encoder_layer_15_ffn_fc_0.w_0
+[debug] : 0, fc_94.tmp_0
+[debug] : 0, encoder_layer_15_ffn_fc_0.b_0
+[debug] : 0, fc_94.tmp_1
+[debug] : 0, fc_94.tmp_2
+[debug] : 0, encoder_layer_15_ffn_fc_1.w_0
+[debug] : 0, fc_95.tmp_0
+[debug] : 0, encoder_layer_15_ffn_fc_1.b_0
+[debug] : 0, fc_95.tmp_1
+[debug] : 0, dropout_48.tmp_0
+[debug] : 0, dropout_48.tmp_1
+[debug] : 0, tmp_82
+[debug] : 0, reduce_mean_64.tmp_0
+[debug] : 0, elementwise_sub_32
+[debug] : 0, square_32.tmp_0
+[debug] : 0, reduce_mean_65.tmp_0
+[debug] : 0, tmp_83
+[debug] : 0, rsqrt_32.tmp_0
+[debug] : 0, elementwise_mul_64
+[debug] : 0, encoder_layer_15_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_15_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_65
+[debug] : 0, elementwise_add_32
+[debug] : 0, encoder_layer_16_multi_head_att_query_fc.w_0
+[debug] : 0, fc_96.tmp_0
+[debug] : 0, encoder_layer_16_multi_head_att_query_fc.b_0
+[debug] : 0, fc_96.tmp_1
+[debug] : 0, encoder_layer_16_multi_head_att_key_fc.w_0
+[debug] : 0, fc_97.tmp_0
+[debug] : 0, encoder_layer_16_multi_head_att_key_fc.b_0
+[debug] : 0, fc_97.tmp_1
+[debug] : 0, encoder_layer_16_multi_head_att_value_fc.w_0
+[debug] : 0, fc_98.tmp_0
+[debug] : 0, encoder_layer_16_multi_head_att_value_fc.b_0
+[debug] : 0, fc_98.tmp_1
+[debug] : 0, reshape2_64.tmp_0
+[debug] : 0, transpose_64.tmp_0
+[debug] : 0, transpose_64.tmp_1
+[debug] : 0, reshape2_65.tmp_0
+[debug] : 0, transpose_65.tmp_0
+[debug] : 0, transpose_65.tmp_1
+[debug] : 0, reshape2_66.tmp_0
+[debug] : 0, transpose_66.tmp_0
+[debug] : 0, transpose_66.tmp_1
+[debug] : 0, scale_17.tmp_0
+[debug] : 0, matmul_33.tmp_0
+[debug] : 0, tmp_84
+[debug] : 0, softmax_16.tmp_0
+[debug] : 0, dropout_49.tmp_0
+[debug] : 0, dropout_49.tmp_1
+[debug] : 0, matmul_34.tmp_0
+[debug] : 0, transpose_67.tmp_0
+[debug] : 0, transpose_67.tmp_1
+[debug] : 0, reshape2_67.tmp_0
+[debug] : 0, encoder_layer_16_multi_head_att_output_fc.w_0
+[debug] : 0, fc_99.tmp_0
+[debug] : 0, encoder_layer_16_multi_head_att_output_fc.b_0
+[debug] : 0, fc_99.tmp_1
+[debug] : 0, dropout_50.tmp_0
+[debug] : 0, dropout_50.tmp_1
+[debug] : 0, tmp_85
+[debug] : 0, reduce_mean_66.tmp_0
+[debug] : 0, elementwise_sub_33
+[debug] : 0, square_33.tmp_0
+[debug] : 0, reduce_mean_67.tmp_0
+[debug] : 0, tmp_86
+[debug] : 0, rsqrt_33.tmp_0
+[debug] : 0, elementwise_mul_66
+[debug] : 0, encoder_layer_16_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_16_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_67
+[debug] : 0, elementwise_add_33
+[debug] : 0, encoder_layer_16_ffn_fc_0.w_0
+[debug] : 0, fc_100.tmp_0
+[debug] : 0, encoder_layer_16_ffn_fc_0.b_0
+[debug] : 0, fc_100.tmp_1
+[debug] : 0, fc_100.tmp_2
+[debug] : 0, encoder_layer_16_ffn_fc_1.w_0
+[debug] : 0, fc_101.tmp_0
+[debug] : 0, encoder_layer_16_ffn_fc_1.b_0
+[debug] : 0, fc_101.tmp_1
+[debug] : 0, dropout_51.tmp_0
+[debug] : 0, dropout_51.tmp_1
+[debug] : 0, tmp_87
+[debug] : 0, reduce_mean_68.tmp_0
+[debug] : 0, elementwise_sub_34
+[debug] : 0, square_34.tmp_0
+[debug] : 0, reduce_mean_69.tmp_0
+[debug] : 0, tmp_88
+[debug] : 0, rsqrt_34.tmp_0
+[debug] : 0, elementwise_mul_68
+[debug] : 0, encoder_layer_16_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_16_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_69
+[debug] : 0, elementwise_add_34
+[debug] : 0, encoder_layer_17_multi_head_att_query_fc.w_0
+[debug] : 0, fc_102.tmp_0
+[debug] : 0, encoder_layer_17_multi_head_att_query_fc.b_0
+[debug] : 0, fc_102.tmp_1
+[debug] : 0, encoder_layer_17_multi_head_att_key_fc.w_0
+[debug] : 0, fc_103.tmp_0
+[debug] : 0, encoder_layer_17_multi_head_att_key_fc.b_0
+[debug] : 0, fc_103.tmp_1
+[debug] : 0, encoder_layer_17_multi_head_att_value_fc.w_0
+[debug] : 0, fc_104.tmp_0
+[debug] : 0, encoder_layer_17_multi_head_att_value_fc.b_0
+[debug] : 0, fc_104.tmp_1
+[debug] : 0, reshape2_68.tmp_0
+[debug] : 0, transpose_68.tmp_0
+[debug] : 0, transpose_68.tmp_1
+[debug] : 0, reshape2_69.tmp_0
+[debug] : 0, transpose_69.tmp_0
+[debug] : 0, transpose_69.tmp_1
+[debug] : 0, reshape2_70.tmp_0
+[debug] : 0, transpose_70.tmp_0
+[debug] : 0, transpose_70.tmp_1
+[debug] : 0, scale_18.tmp_0
+[debug] : 0, matmul_35.tmp_0
+[debug] : 0, tmp_89
+[debug] : 0, softmax_17.tmp_0
+[debug] : 0, dropout_52.tmp_0
+[debug] : 0, dropout_52.tmp_1
+[debug] : 0, matmul_36.tmp_0
+[debug] : 0, transpose_71.tmp_0
+[debug] : 0, transpose_71.tmp_1
+[debug] : 0, reshape2_71.tmp_0
+[debug] : 0, encoder_layer_17_multi_head_att_output_fc.w_0
+[debug] : 0, fc_105.tmp_0
+[debug] : 0, encoder_layer_17_multi_head_att_output_fc.b_0
+[debug] : 0, fc_105.tmp_1
+[debug] : 0, dropout_53.tmp_0
+[debug] : 0, dropout_53.tmp_1
+[debug] : 0, tmp_90
+[debug] : 0, reduce_mean_70.tmp_0
+[debug] : 0, elementwise_sub_35
+[debug] : 0, square_35.tmp_0
+[debug] : 0, reduce_mean_71.tmp_0
+[debug] : 0, tmp_91
+[debug] : 0, rsqrt_35.tmp_0
+[debug] : 0, elementwise_mul_70
+[debug] : 0, encoder_layer_17_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_17_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_71
+[debug] : 0, elementwise_add_35
+[debug] : 0, encoder_layer_17_ffn_fc_0.w_0
+[debug] : 0, fc_106.tmp_0
+[debug] : 0, encoder_layer_17_ffn_fc_0.b_0
+[debug] : 0, fc_106.tmp_1
+[debug] : 0, fc_106.tmp_2
+[debug] : 0, encoder_layer_17_ffn_fc_1.w_0
+[debug] : 0, fc_107.tmp_0
+[debug] : 0, encoder_layer_17_ffn_fc_1.b_0
+[debug] : 0, fc_107.tmp_1
+[debug] : 0, dropout_54.tmp_0
+[debug] : 0, dropout_54.tmp_1
+[debug] : 0, tmp_92
+[debug] : 0, reduce_mean_72.tmp_0
+[debug] : 0, elementwise_sub_36
+[debug] : 0, square_36.tmp_0
+[debug] : 0, reduce_mean_73.tmp_0
+[debug] : 0, tmp_93
+[debug] : 0, rsqrt_36.tmp_0
+[debug] : 0, elementwise_mul_72
+[debug] : 0, encoder_layer_17_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_17_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_73
+[debug] : 0, elementwise_add_36
+[debug] : 0, encoder_layer_18_multi_head_att_query_fc.w_0
+[debug] : 0, fc_108.tmp_0
+[debug] : 0, encoder_layer_18_multi_head_att_query_fc.b_0
+[debug] : 0, fc_108.tmp_1
+[debug] : 0, encoder_layer_18_multi_head_att_key_fc.w_0
+[debug] : 0, fc_109.tmp_0
+[debug] : 0, encoder_layer_18_multi_head_att_key_fc.b_0
+[debug] : 0, fc_109.tmp_1
+[debug] : 0, encoder_layer_18_multi_head_att_value_fc.w_0
+[debug] : 0, fc_110.tmp_0
+[debug] : 0, encoder_layer_18_multi_head_att_value_fc.b_0
+[debug] : 0, fc_110.tmp_1
+[debug] : 0, reshape2_72.tmp_0
+[debug] : 0, transpose_72.tmp_0
+[debug] : 0, transpose_72.tmp_1
+[debug] : 0, reshape2_73.tmp_0
+[debug] : 0, transpose_73.tmp_0
+[debug] : 0, transpose_73.tmp_1
+[debug] : 0, reshape2_74.tmp_0
+[debug] : 0, transpose_74.tmp_0
+[debug] : 0, transpose_74.tmp_1
+[debug] : 0, scale_19.tmp_0
+[debug] : 0, matmul_37.tmp_0
+[debug] : 0, tmp_94
+[debug] : 0, softmax_18.tmp_0
+[debug] : 0, dropout_55.tmp_0
+[debug] : 0, dropout_55.tmp_1
+[debug] : 0, matmul_38.tmp_0
+[debug] : 0, transpose_75.tmp_0
+[debug] : 0, transpose_75.tmp_1
+[debug] : 0, reshape2_75.tmp_0
+[debug] : 0, encoder_layer_18_multi_head_att_output_fc.w_0
+[debug] : 0, fc_111.tmp_0
+[debug] : 0, encoder_layer_18_multi_head_att_output_fc.b_0
+[debug] : 0, fc_111.tmp_1
+[debug] : 0, dropout_56.tmp_0
+[debug] : 0, dropout_56.tmp_1
+[debug] : 0, tmp_95
+[debug] : 0, reduce_mean_74.tmp_0
+[debug] : 0, elementwise_sub_37
+[debug] : 0, square_37.tmp_0
+[debug] : 0, reduce_mean_75.tmp_0
+[debug] : 0, tmp_96
+[debug] : 0, rsqrt_37.tmp_0
+[debug] : 0, elementwise_mul_74
+[debug] : 0, encoder_layer_18_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_18_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_75
+[debug] : 0, elementwise_add_37
+[debug] : 0, encoder_layer_18_ffn_fc_0.w_0
+[debug] : 0, fc_112.tmp_0
+[debug] : 0, encoder_layer_18_ffn_fc_0.b_0
+[debug] : 0, fc_112.tmp_1
+[debug] : 0, fc_112.tmp_2
+[debug] : 0, encoder_layer_18_ffn_fc_1.w_0
+[debug] : 0, fc_113.tmp_0
+[debug] : 0, encoder_layer_18_ffn_fc_1.b_0
+[debug] : 0, fc_113.tmp_1
+[debug] : 0, dropout_57.tmp_0
+[debug] : 0, dropout_57.tmp_1
+[debug] : 0, tmp_97
+[debug] : 0, reduce_mean_76.tmp_0
+[debug] : 0, elementwise_sub_38
+[debug] : 0, square_38.tmp_0
+[debug] : 0, reduce_mean_77.tmp_0
+[debug] : 0, tmp_98
+[debug] : 0, rsqrt_38.tmp_0
+[debug] : 0, elementwise_mul_76
+[debug] : 0, encoder_layer_18_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_18_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_77
+[debug] : 0, elementwise_add_38
+[debug] : 0, encoder_layer_19_multi_head_att_query_fc.w_0
+[debug] : 0, fc_114.tmp_0
+[debug] : 0, encoder_layer_19_multi_head_att_query_fc.b_0
+[debug] : 0, fc_114.tmp_1
+[debug] : 0, encoder_layer_19_multi_head_att_key_fc.w_0
+[debug] : 0, fc_115.tmp_0
+[debug] : 0, encoder_layer_19_multi_head_att_key_fc.b_0
+[debug] : 0, fc_115.tmp_1
+[debug] : 0, encoder_layer_19_multi_head_att_value_fc.w_0
+[debug] : 0, fc_116.tmp_0
+[debug] : 0, encoder_layer_19_multi_head_att_value_fc.b_0
+[debug] : 0, fc_116.tmp_1
+[debug] : 0, reshape2_76.tmp_0
+[debug] : 0, transpose_76.tmp_0
+[debug] : 0, transpose_76.tmp_1
+[debug] : 0, reshape2_77.tmp_0
+[debug] : 0, transpose_77.tmp_0
+[debug] : 0, transpose_77.tmp_1
+[debug] : 0, reshape2_78.tmp_0
+[debug] : 0, transpose_78.tmp_0
+[debug] : 0, transpose_78.tmp_1
+[debug] : 0, scale_20.tmp_0
+[debug] : 0, matmul_39.tmp_0
+[debug] : 0, tmp_99
+[debug] : 0, softmax_19.tmp_0
+[debug] : 0, dropout_58.tmp_0
+[debug] : 0, dropout_58.tmp_1
+[debug] : 0, matmul_40.tmp_0
+[debug] : 0, transpose_79.tmp_0
+[debug] : 0, transpose_79.tmp_1
+[debug] : 0, reshape2_79.tmp_0
+[debug] : 0, encoder_layer_19_multi_head_att_output_fc.w_0
+[debug] : 0, fc_117.tmp_0
+[debug] : 0, encoder_layer_19_multi_head_att_output_fc.b_0
+[debug] : 0, fc_117.tmp_1
+[debug] : 0, dropout_59.tmp_0
+[debug] : 0, dropout_59.tmp_1
+[debug] : 0, tmp_100
+[debug] : 0, reduce_mean_78.tmp_0
+[debug] : 0, elementwise_sub_39
+[debug] : 0, square_39.tmp_0
+[debug] : 0, reduce_mean_79.tmp_0
+[debug] : 0, tmp_101
+[debug] : 0, rsqrt_39.tmp_0
+[debug] : 0, elementwise_mul_78
+[debug] : 0, encoder_layer_19_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_19_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_79
+[debug] : 0, elementwise_add_39
+[debug] : 0, encoder_layer_19_ffn_fc_0.w_0
+[debug] : 0, fc_118.tmp_0
+[debug] : 0, encoder_layer_19_ffn_fc_0.b_0
+[debug] : 0, fc_118.tmp_1
+[debug] : 0, fc_118.tmp_2
+[debug] : 0, encoder_layer_19_ffn_fc_1.w_0
+[debug] : 0, fc_119.tmp_0
+[debug] : 0, encoder_layer_19_ffn_fc_1.b_0
+[debug] : 0, fc_119.tmp_1
+[debug] : 0, dropout_60.tmp_0
+[debug] : 0, dropout_60.tmp_1
+[debug] : 0, tmp_102
+[debug] : 0, reduce_mean_80.tmp_0
+[debug] : 0, elementwise_sub_40
+[debug] : 0, square_40.tmp_0
+[debug] : 0, reduce_mean_81.tmp_0
+[debug] : 0, tmp_103
+[debug] : 0, rsqrt_40.tmp_0
+[debug] : 0, elementwise_mul_80
+[debug] : 0, encoder_layer_19_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_19_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_81
+[debug] : 0, elementwise_add_40
+[debug] : 0, encoder_layer_20_multi_head_att_query_fc.w_0
+[debug] : 0, fc_120.tmp_0
+[debug] : 0, encoder_layer_20_multi_head_att_query_fc.b_0
+[debug] : 0, fc_120.tmp_1
+[debug] : 0, encoder_layer_20_multi_head_att_key_fc.w_0
+[debug] : 0, fc_121.tmp_0
+[debug] : 0, encoder_layer_20_multi_head_att_key_fc.b_0
+[debug] : 0, fc_121.tmp_1
+[debug] : 0, encoder_layer_20_multi_head_att_value_fc.w_0
+[debug] : 0, fc_122.tmp_0
+[debug] : 0, encoder_layer_20_multi_head_att_value_fc.b_0
+[debug] : 0, fc_122.tmp_1
+[debug] : 0, reshape2_80.tmp_0
+[debug] : 0, transpose_80.tmp_0
+[debug] : 0, transpose_80.tmp_1
+[debug] : 0, reshape2_81.tmp_0
+[debug] : 0, transpose_81.tmp_0
+[debug] : 0, transpose_81.tmp_1
+[debug] : 0, reshape2_82.tmp_0
+[debug] : 0, transpose_82.tmp_0
+[debug] : 0, transpose_82.tmp_1
+[debug] : 0, scale_21.tmp_0
+[debug] : 0, matmul_41.tmp_0
+[debug] : 0, tmp_104
+[debug] : 0, softmax_20.tmp_0
+[debug] : 0, dropout_61.tmp_0
+[debug] : 0, dropout_61.tmp_1
+[debug] : 0, matmul_42.tmp_0
+[debug] : 0, transpose_83.tmp_0
+[debug] : 0, transpose_83.tmp_1
+[debug] : 0, reshape2_83.tmp_0
+[debug] : 0, encoder_layer_20_multi_head_att_output_fc.w_0
+[debug] : 0, fc_123.tmp_0
+[debug] : 0, encoder_layer_20_multi_head_att_output_fc.b_0
+[debug] : 0, fc_123.tmp_1
+[debug] : 0, dropout_62.tmp_0
+[debug] : 0, dropout_62.tmp_1
+[debug] : 0, tmp_105
+[debug] : 0, reduce_mean_82.tmp_0
+[debug] : 0, elementwise_sub_41
+[debug] : 0, square_41.tmp_0
+[debug] : 0, reduce_mean_83.tmp_0
+[debug] : 0, tmp_106
+[debug] : 0, rsqrt_41.tmp_0
+[debug] : 0, elementwise_mul_82
+[debug] : 0, encoder_layer_20_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_20_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_83
+[debug] : 0, elementwise_add_41
+[debug] : 0, encoder_layer_20_ffn_fc_0.w_0
+[debug] : 0, fc_124.tmp_0
+[debug] : 0, encoder_layer_20_ffn_fc_0.b_0
+[debug] : 0, fc_124.tmp_1
+[debug] : 0, fc_124.tmp_2
+[debug] : 0, encoder_layer_20_ffn_fc_1.w_0
+[debug] : 0, fc_125.tmp_0
+[debug] : 0, encoder_layer_20_ffn_fc_1.b_0
+[debug] : 0, fc_125.tmp_1
+[debug] : 0, dropout_63.tmp_0
+[debug] : 0, dropout_63.tmp_1
+[debug] : 0, tmp_107
+[debug] : 0, reduce_mean_84.tmp_0
+[debug] : 0, elementwise_sub_42
+[debug] : 0, square_42.tmp_0
+[debug] : 0, reduce_mean_85.tmp_0
+[debug] : 0, tmp_108
+[debug] : 0, rsqrt_42.tmp_0
+[debug] : 0, elementwise_mul_84
+[debug] : 0, encoder_layer_20_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_20_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_85
+[debug] : 0, elementwise_add_42
+[debug] : 0, encoder_layer_21_multi_head_att_query_fc.w_0
+[debug] : 0, fc_126.tmp_0
+[debug] : 0, encoder_layer_21_multi_head_att_query_fc.b_0
+[debug] : 0, fc_126.tmp_1
+[debug] : 0, encoder_layer_21_multi_head_att_key_fc.w_0
+[debug] : 0, fc_127.tmp_0
+[debug] : 0, encoder_layer_21_multi_head_att_key_fc.b_0
+[debug] : 0, fc_127.tmp_1
+[debug] : 0, encoder_layer_21_multi_head_att_value_fc.w_0
+[debug] : 0, fc_128.tmp_0
+[debug] : 0, encoder_layer_21_multi_head_att_value_fc.b_0
+[debug] : 0, fc_128.tmp_1
+[debug] : 0, reshape2_84.tmp_0
+[debug] : 0, transpose_84.tmp_0
+[debug] : 0, transpose_84.tmp_1
+[debug] : 0, reshape2_85.tmp_0
+[debug] : 0, transpose_85.tmp_0
+[debug] : 0, transpose_85.tmp_1
+[debug] : 0, reshape2_86.tmp_0
+[debug] : 0, transpose_86.tmp_0
+[debug] : 0, transpose_86.tmp_1
+[debug] : 0, scale_22.tmp_0
+[debug] : 0, matmul_43.tmp_0
+[debug] : 0, tmp_109
+[debug] : 0, softmax_21.tmp_0
+[debug] : 0, dropout_64.tmp_0
+[debug] : 0, dropout_64.tmp_1
+[debug] : 0, matmul_44.tmp_0
+[debug] : 0, transpose_87.tmp_0
+[debug] : 0, transpose_87.tmp_1
+[debug] : 0, reshape2_87.tmp_0
+[debug] : 0, encoder_layer_21_multi_head_att_output_fc.w_0
+[debug] : 0, fc_129.tmp_0
+[debug] : 0, encoder_layer_21_multi_head_att_output_fc.b_0
+[debug] : 0, fc_129.tmp_1
+[debug] : 0, dropout_65.tmp_0
+[debug] : 0, dropout_65.tmp_1
+[debug] : 0, tmp_110
+[debug] : 0, reduce_mean_86.tmp_0
+[debug] : 0, elementwise_sub_43
+[debug] : 0, square_43.tmp_0
+[debug] : 0, reduce_mean_87.tmp_0
+[debug] : 0, tmp_111
+[debug] : 0, rsqrt_43.tmp_0
+[debug] : 0, elementwise_mul_86
+[debug] : 0, encoder_layer_21_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_21_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_87
+[debug] : 0, elementwise_add_43
+[debug] : 0, encoder_layer_21_ffn_fc_0.w_0
+[debug] : 0, fc_130.tmp_0
+[debug] : 0, encoder_layer_21_ffn_fc_0.b_0
+[debug] : 0, fc_130.tmp_1
+[debug] : 0, fc_130.tmp_2
+[debug] : 0, encoder_layer_21_ffn_fc_1.w_0
+[debug] : 0, fc_131.tmp_0
+[debug] : 0, encoder_layer_21_ffn_fc_1.b_0
+[debug] : 0, fc_131.tmp_1
+[debug] : 0, dropout_66.tmp_0
+[debug] : 0, dropout_66.tmp_1
+[debug] : 0, tmp_112
+[debug] : 0, reduce_mean_88.tmp_0
+[debug] : 0, elementwise_sub_44
+[debug] : 0, square_44.tmp_0
+[debug] : 0, reduce_mean_89.tmp_0
+[debug] : 0, tmp_113
+[debug] : 0, rsqrt_44.tmp_0
+[debug] : 0, elementwise_mul_88
+[debug] : 0, encoder_layer_21_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_21_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_89
+[debug] : 0, elementwise_add_44
+[debug] : 0, encoder_layer_22_multi_head_att_query_fc.w_0
+[debug] : 0, fc_132.tmp_0
+[debug] : 0, encoder_layer_22_multi_head_att_query_fc.b_0
+[debug] : 0, fc_132.tmp_1
+[debug] : 0, encoder_layer_22_multi_head_att_key_fc.w_0
+[debug] : 0, fc_133.tmp_0
+[debug] : 0, encoder_layer_22_multi_head_att_key_fc.b_0
+[debug] : 0, fc_133.tmp_1
+[debug] : 0, encoder_layer_22_multi_head_att_value_fc.w_0
+[debug] : 0, fc_134.tmp_0
+[debug] : 0, encoder_layer_22_multi_head_att_value_fc.b_0
+[debug] : 0, fc_134.tmp_1
+[debug] : 0, reshape2_88.tmp_0
+[debug] : 0, transpose_88.tmp_0
+[debug] : 0, transpose_88.tmp_1
+[debug] : 0, reshape2_89.tmp_0
+[debug] : 0, transpose_89.tmp_0
+[debug] : 0, transpose_89.tmp_1
+[debug] : 0, reshape2_90.tmp_0
+[debug] : 0, transpose_90.tmp_0
+[debug] : 0, transpose_90.tmp_1
+[debug] : 0, scale_23.tmp_0
+[debug] : 0, matmul_45.tmp_0
+[debug] : 0, tmp_114
+[debug] : 0, softmax_22.tmp_0
+[debug] : 0, dropout_67.tmp_0
+[debug] : 0, dropout_67.tmp_1
+[debug] : 0, matmul_46.tmp_0
+[debug] : 0, transpose_91.tmp_0
+[debug] : 0, transpose_91.tmp_1
+[debug] : 0, reshape2_91.tmp_0
+[debug] : 0, encoder_layer_22_multi_head_att_output_fc.w_0
+[debug] : 0, fc_135.tmp_0
+[debug] : 0, encoder_layer_22_multi_head_att_output_fc.b_0
+[debug] : 0, fc_135.tmp_1
+[debug] : 0, dropout_68.tmp_0
+[debug] : 0, dropout_68.tmp_1
+[debug] : 0, tmp_115
+[debug] : 0, reduce_mean_90.tmp_0
+[debug] : 0, elementwise_sub_45
+[debug] : 0, square_45.tmp_0
+[debug] : 0, reduce_mean_91.tmp_0
+[debug] : 0, tmp_116
+[debug] : 0, rsqrt_45.tmp_0
+[debug] : 0, elementwise_mul_90
+[debug] : 0, encoder_layer_22_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_22_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_91
+[debug] : 0, elementwise_add_45
+[debug] : 0, encoder_layer_22_ffn_fc_0.w_0
+[debug] : 0, fc_136.tmp_0
+[debug] : 0, encoder_layer_22_ffn_fc_0.b_0
+[debug] : 0, fc_136.tmp_1
+[debug] : 0, fc_136.tmp_2
+[debug] : 0, encoder_layer_22_ffn_fc_1.w_0
+[debug] : 0, fc_137.tmp_0
+[debug] : 0, encoder_layer_22_ffn_fc_1.b_0
+[debug] : 0, fc_137.tmp_1
+[debug] : 0, dropout_69.tmp_0
+[debug] : 0, dropout_69.tmp_1
+[debug] : 0, tmp_117
+[debug] : 0, reduce_mean_92.tmp_0
+[debug] : 0, elementwise_sub_46
+[debug] : 0, square_46.tmp_0
+[debug] : 0, reduce_mean_93.tmp_0
+[debug] : 0, tmp_118
+[debug] : 0, rsqrt_46.tmp_0
+[debug] : 0, elementwise_mul_92
+[debug] : 0, encoder_layer_22_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_22_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_93
+[debug] : 0, elementwise_add_46
+[debug] : 0, encoder_layer_23_multi_head_att_query_fc.w_0
+[debug] : 0, fc_138.tmp_0
+[debug] : 0, encoder_layer_23_multi_head_att_query_fc.b_0
+[debug] : 0, fc_138.tmp_1
+[debug] : 0, encoder_layer_23_multi_head_att_key_fc.w_0
+[debug] : 0, fc_139.tmp_0
+[debug] : 0, encoder_layer_23_multi_head_att_key_fc.b_0
+[debug] : 0, fc_139.tmp_1
+[debug] : 0, encoder_layer_23_multi_head_att_value_fc.w_0
+[debug] : 0, fc_140.tmp_0
+[debug] : 0, encoder_layer_23_multi_head_att_value_fc.b_0
+[debug] : 0, fc_140.tmp_1
+[debug] : 0, reshape2_92.tmp_0
+[debug] : 0, transpose_92.tmp_0
+[debug] : 0, transpose_92.tmp_1
+[debug] : 0, reshape2_93.tmp_0
+[debug] : 0, transpose_93.tmp_0
+[debug] : 0, transpose_93.tmp_1
+[debug] : 0, reshape2_94.tmp_0
+[debug] : 0, transpose_94.tmp_0
+[debug] : 0, transpose_94.tmp_1
+[debug] : 0, scale_24.tmp_0
+[debug] : 0, matmul_47.tmp_0
+[debug] : 0, tmp_119
+[debug] : 0, softmax_23.tmp_0
+[debug] : 0, dropout_70.tmp_0
+[debug] : 0, dropout_70.tmp_1
+[debug] : 0, matmul_48.tmp_0
+[debug] : 0, transpose_95.tmp_0
+[debug] : 0, transpose_95.tmp_1
+[debug] : 0, reshape2_95.tmp_0
+[debug] : 0, encoder_layer_23_multi_head_att_output_fc.w_0
+[debug] : 0, fc_141.tmp_0
+[debug] : 0, encoder_layer_23_multi_head_att_output_fc.b_0
+[debug] : 0, fc_141.tmp_1
+[debug] : 0, dropout_71.tmp_0
+[debug] : 0, dropout_71.tmp_1
+[debug] : 0, tmp_120
+[debug] : 0, reduce_mean_94.tmp_0
+[debug] : 0, elementwise_sub_47
+[debug] : 0, square_47.tmp_0
+[debug] : 0, reduce_mean_95.tmp_0
+[debug] : 0, tmp_121
+[debug] : 0, rsqrt_47.tmp_0
+[debug] : 0, elementwise_mul_94
+[debug] : 0, encoder_layer_23_post_att_layer_norm_scale
+[debug] : 0, encoder_layer_23_post_att_layer_norm_bias
+[debug] : 0, elementwise_mul_95
+[debug] : 0, elementwise_add_47
+[debug] : 0, encoder_layer_23_ffn_fc_0.w_0
+[debug] : 0, fc_142.tmp_0
+[debug] : 0, encoder_layer_23_ffn_fc_0.b_0
+[debug] : 0, fc_142.tmp_1
+[debug] : 0, fc_142.tmp_2
+[debug] : 0, encoder_layer_23_ffn_fc_1.w_0
+[debug] : 0, fc_143.tmp_0
+[debug] : 0, encoder_layer_23_ffn_fc_1.b_0
+[debug] : 0, fc_143.tmp_1
+[debug] : 0, dropout_72.tmp_0
+[debug] : 0, dropout_72.tmp_1
+[debug] : 0, tmp_122
+[debug] : 0, reduce_mean_96.tmp_0
+[debug] : 0, elementwise_sub_48
+[debug] : 0, square_48.tmp_0
+[debug] : 0, reduce_mean_97.tmp_0
+[debug] : 0, tmp_123
+[debug] : 0, rsqrt_48.tmp_0
+[debug] : 0, elementwise_mul_96
+[debug] : 0, encoder_layer_23_post_ffn_layer_norm_scale
+[debug] : 0, encoder_layer_23_post_ffn_layer_norm_bias
+[debug] : 0, elementwise_mul_97
+[debug] : 0, elementwise_add_48
+[debug] : 0, slice_0.tmp_0
+[debug] : 0, reshape2_96.tmp_0
+[debug] : 0, reshape2_96.tmp_1
+[debug] : 0, pooled_fc.w_0
+[debug] : 0, fc_144.tmp_0
+[debug] : 0, pooled_fc.b_0
+[debug] : 0, fc_144.tmp_1
+[debug] : 0, fc_144.tmp_2
+[debug] : 0, senti_cls.senti_cls.dropout_0.tmp_0
+[debug] : 0, senti_cls.senti_cls.dropout_0.tmp_1
+[debug] : 0, senti_cls.cls_out_w
+[debug] : 0, senti_cls.senti_cls.fc_0.tmp_0
+[debug] : 0, senti_cls.cls_out_b
+[debug] : 0, senti_cls.senti_cls.fc_0.tmp_1
+[debug] : 0, senti_cls.senti_cls.softmax_0.tmp_0
+[debug] : 0, senti_cls.senti_cls.cross_entropy2_0.tmp_0
+[debug] : 0, senti_cls.senti_cls.cross_entropy2_0.tmp_1
+[debug] : 0, senti_cls.senti_cls.cross_entropy2_0.tmp_2
+[debug] : 0, senti_cls.senti_cls.mean_0.tmp_0
+[debug] : 0, reduce_sum_0.tmp_0
+0
+preparing data...ok!
+61
+30
+name: "reduce_sum_0.tmp_0"
+type {
+  type: LOD_TENSOR
+  lod_tensor {
+    tensor {
+      data_type: FP32
+      dims: 1
+    }
+  }
+}
+persistable: false
+
+random init params...
+Loading pretraining parameters from pretrain/ernie/params...

demo/demo3/pretrain

-../../pretrain/
\ No newline at end of file

demo/demo3/run.py

@@ -11,11 +11,11 @@ if __name__ == '__main__':
     vocab_path = './pretrain/ernie/vocab.txt'
 
     train_file = './data/cls4mrqa/train.tsv'
+    predict_file = './data/cls4mrqa/dev.tsv'
 
     config = json.load(open('./pretrain/ernie/ernie_config.json'))
     # ernie = palm.backbone.ERNIE(...)
     ernie = palm.backbone.ERNIE.from_config(config)
-    # pred_ernie = palm.backbone.ERNIE.from_config(config, phase='pred')
 
     # cls_reader2 = palm.reader.cls(train_file_topic, vocab_path, batch_size, max_seqlen)
     # cls_reader3 = palm.reader.cls(train_file_subj, vocab_path, batch_size, max_seqlen)
@@ -24,16 +24,25 @@ if __name__ == '__main__':
 
     # 创建该分类任务的reader，由诸多参数控制数据集读入格式、文件数量、预处理规则等
     cls_reader = palm.reader.ClassifyReader(vocab_path, max_seqlen)
+    predict_cls_reader = palm.reader.ClassifyReader(vocab_path, max_seqlen, phase='predict')
     print(cls_reader.outputs_attr)
+    print(predict_cls_reader.outputs_attr)
     # 不同的backbone会对任务reader有不同的特征要求，例如对于分类任务，基本的输入feature为token_ids和label_ids，但是对于BERT，还要求从输入中额外提取position、segment、input_mask等特征，因此经过register后，reader会自动补充backbone所要求的字段
     cls_reader.register_with(ernie)
     print(cls_reader.outputs_attr)
+    print(predict_cls_reader.outputs_attr)
+
+    print("preparing data...")
+    print(cls_reader.num_examples)
+    cls_reader.load_data(train_file, batch_size, num_epochs=num_epochs)
+    print(cls_reader.num_examples)
+    print('done!')
+
     # 创建任务头（task head），如分类、匹配、机器阅读理解等。每个任务头有跟该任务相关的必选/可选参数。注意，任务头与reader是解耦合的，只要任务头依赖的数据集侧的字段能被reader提供，那么就是合法的
     cls_head = palm.head.Classify(4, 1024, 0.1)
-    # cls_pred_head = palm.head.Classify(4, 1024, 0.1, phase='pred')
 
     # 根据reader和任务头来创建一个训练器trainer，trainer代表了一个训练任务，内部维护着训练进程、和任务的关键信息，并完成合法性校验，该任务的模型保存、载入等相关规则控制
-    trainer = palm.Trainer('senti_cls', cls_reader, cls_head)
+    trainer = palm.Trainer('senti_cls')
 
     # match4mrqa.reuse_head_with(mrc4mrqa)
 
@@ -41,19 +50,16 @@ if __name__ == '__main__':
     # output_vars = ernie.build(data_vars)
     # cls_head.build({'backbone': output_vars, 'reader': data_vars})
 
-    loss_var = trainer.build_forward(ernie)
+    loss_var = trainer.build_forward(ernie, cls_head)
 
     # controller.build_forward()
     # Error! a head/backbone can be only build once! Try NOT to call build_forward method for any Trainer!
 
-    print(trainer.num_examples)
-    iterator_fn = trainer.load_data(train_file, 'csv', num_epochs=num_epochs, batch_size=batch_size)
-    print(trainer.num_examples)
-
-    n_steps = trainer.num_examples * num_epochs // batch_size
-    warmup_steps = int(0.1 * n_steps)
-    print(warmup_steps)
-    sched = palm.lr_sched.TriangularSchedualer(warmup_steps, n_steps)
+    # n_steps = cls_reader.num_examples * num_epochs // batch_size
+    # warmup_steps = int(0.1 * n_steps)
+    # print(warmup_steps)
+    # sched = palm.lr_sched.TriangularSchedualer(warmup_steps, n_steps)
+    sched = None
 
     adam = palm.optimizer.Adam(loss_var, lr, sched)
 
@@ -62,11 +68,22 @@ if __name__ == '__main__':
     trainer.random_init_params()
     trainer.load_pretrain('pretrain/ernie/params')
 
-    # print(trainer.train_one_step(next(iterator_fn())))
-    # trainer.train_one_epoch()
-    trainer.train(iterator_fn, print_steps=1, save_steps=5, save_path='outputs/ckpt')
+    # trainer.train(iterator_fn, print_steps=1, save_steps=5, save_path='outputs', save_type='ckpt,predict')
+    trainer.fit_reader(cls_reader)
+    trainer.train(print_steps=1)
     # trainer.save()
 
+    print('prepare to predict...')
+    pred_ernie = palm.backbone.ERNIE.from_config(config, phase='pred')
+    cls_pred_head = palm.head.Classify(4, 1024, phase='pred')
+    trainer.build_predict_forward(pred_ernie, cls_pred_head)
+
+    predict_cls_reader.load_data(predict_file, 8)
+    print(predict_cls_reader.num_examples)
+    predict_cls_reader.register_with(pred_ernie)
+    trainer.fit_reader(predict_cls_reader, phase='predict')
+    print('predicting..')
+    trainer.predict(print_steps=20)
 
 
 

download_models.py

+# -*- coding: UTF-8 -*-
+#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import paddlepalm as palm
+import sys
+import argparse
+ 
+# create parser
+parser = argparse.ArgumentParser(prog='download_models.py', usage='python %(prog)s -l | -d <model_name> [-h]\n\nFor example,\n\tpython %(prog)s -d bert-en-uncased-large ',description = 'Download pretrain models for initializing params of backbones. ')
+parser1= parser.add_argument_group("required arguments")
+parser1.add_argument('-l','--list', action = 'store_true', help = 'show the list of available pretrain models', default = False)
+parser1.add_argument('-d','--download', action = 'store', help = 'download pretrain models. The available pretrain models can be listed by run "python download_models.py -l"') 
+args = parser.parse_args()
+
+if(args.list):
+  palm.downloader.ls('pretrain')
+elif(args.download):
+  print('download~~~')
+  print(args.download)
+  palm.downloader.download('pretrain', args.download)
+else:
+  print (parser.parse_args(['-h']))

interface.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""v1.1"""
-
-class reader(object):
-    """interface of data manager."""
-
-    def __init__(self, config):
-        assert isinstance(config, dict)
-
-    # @property
-    # def inputs_attr(self):
-    #     """描述reader输入对象的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1.
-    #     Return:
-    #         dict类型。对各个输入对象的属性描述。例如，
-    #         对于文本分类任务，可能需要包含输入文本和所属标签的id
-    #             {"text": ([], 'str'),
-    #              "label": ([], 'int')}
-    #         对于标注任务，可能需要输入词序列和对应的标签
-    #             {"tokens", ([-1], 'str'),
-    #              "tags", ([-1], 'str')}
-    #         对于机器阅读理解任务，可能需要包含上下文、问题、回答、答案区域的起止位置等
-    #             {"paragraph", ([], 'str'),
-    #              "question", ([], 'str'),
-    #              "start_position", ([], 'int')
-    #         """
-    #     raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述reader输出对象（被yield出的对象）的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        注意：当使用mini-batch梯度下降学习策略时，，应为常规的输入对象设置batch_size维度（一般为-1）
-        Return:
-            dict类型。对各个输入对象的属性描述。例如，
-            对于文本分类和匹配任务，yield的输出内容可能包含如下的对象（下游backbone和task可按需访问其中的对象）
-                {"token_ids": ([-1, max_len], 'int64'),
-                 "input_ids": ([-1, max_len], 'int64'),
-                 "segment_ids": ([-1, max_len], 'int64'),
-                 "input_mask": ([-1, max_len], 'float32'),
-                 "label": ([-1], 'int')}
-        """
-        raise NotImplementedError()
-
-    # def parse_line(self):
-    #     """框架内部使用字典描述每个样本，字典的key为inputs_attr，value为每个input对应的符合attr描述的值。
-    #         该函数负责将文本行解析成符合inputs_attr描述的字典类型的样本。默认的parse_line方法会读取json格式的数据集文件，数据集的每一行为json格式描述的样本。
-    #         用户可通过对该方法的继承改写来适配不同格式的数据集，例如csv格式甚至tfrecord文件。
-    #         """
-    #     raise NotImplementedError()
-    # 
-    # def tokenize(self, line):
-    #     """框架中内置了word piece tokenizer等分词器，用户可通过修改tokenizer超参数来制定使用的分词器，若内置的分词器均无法满足需求，用户可通过对该方法的继承改写来自定义分词器。
-    #         Args:
-    #             - line: a unicode string. 
-    #         Return:
-    #             a list of tokens
-    #         """
-    #     raise NotImplementedError()
-    
-    def iterator(self):
-        """数据集遍历接口，注意，当数据集遍历到尾部时该接口应自动完成指针重置，即重新从数据集头部开始新的遍历。
-        Yield:
-            (dict) elements that meet the requirements in output_templete
-        """
-        raise NotImplementedError()
-
-    @property
-    def num_examples(self):
-        """数据集中的样本数量，即每个epoch中iterator所生成的样本数。注意，使用滑动窗口等可能导致数据集样本数发生变化的策略时，该接口应返回runtime阶段的实际样本数。"""
-        raise NotImplementedError()
-
-
-
-class backbone(object):
-    """interface of backbone model."""
-
-    def __init__(self, config, phase):
-        """
-        Args:
-            config: dict类型。描述了 多任务配置文件+预训练模型配置文件 中定义超参数
-            phase: str类型。运行阶段，目前支持train和predict
-            """
-        assert isinstance(config, dict)
-
-    @property
-    def inputs_attr(self):
-        """描述backbone从reader处需要得到的输入对象的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输入对象的属性描述。例如，
-            对于文本分类和匹配任务，bert backbone依赖的reader对象主要包含如下的对象
-                {"token_ids": ([-1, max_len], 'int64'),
-                 "input_ids": ([-1, max_len], 'int64'),
-                 "segment_ids": ([-1, max_len], 'int64'),
-                 "input_mask": ([-1, max_len], 'float32')}"""
-        raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述backbone输出对象的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输出对象的属性描述。例如，
-            对于文本分类和匹配任务，bert backbone的输出内容可能包含如下的对象
-                {"word_emb": ([-1, max_seqlen, word_emb_size], 'float32'),
-                 "sentence_emb": ([-1, hidden_size], 'float32'),
-                 "sim_vec": ([-1, hidden_size], 'float32')}""" 
-        raise NotImplementedError()
-
-    def build(self, inputs):
-        """建立backbone的计算图。将符合inputs_attr描述的静态图Variable输入映射成符合outputs_attr描述的静态图Variable输出。
-        Args:
-            inputs: dict类型。字典中包含inputs_attr中的对象名到计算图Variable的映射，inputs中至少会包含inputs_attr中定义的对象
-        Return:
-           需要输出的计算图变量，输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-            """
-        raise NotImplementedError()
-
-
-
-
-class task_paradigm(object):
-
-    def __init__(self, config, phase, backbone_config):
-        """
-            config: dict类型。描述了 任务实例(task instance)+多任务配置文件 中定义超参数
-            phase: str类型。运行阶段，目前支持train和predict
-            """
-
-    @property
-    def inputs_attrs(self):
-        """描述task_layer需要从reader, backbone等输入对象集合所读取到的输入对象的属性，第一级key为对象集和的名字，如backbone，reader等（后续会支持更灵活的输入），第二级key为对象集和中各对象的属性，包括对象的名字，shape和dtype。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个对象集及其输入对象的属性描述。"""
-        raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述task输出对象的属性，包括对象的名字，shape和dtype。输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-        当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输入对象的属性描述。注意，训练阶段必须包含名为loss的输出对象。
-            """
-
-        raise NotImplementedError()
-
-    @property
-    def epoch_inputs_attrs(self):
-        return {}
-
-    def build(self, inputs, scope_name=""):
-        """建立task_layer的计算图。将符合inputs_attrs描述的来自各个对象集的静态图Variables映射成符合outputs_attr描述的静态图Variable输出。
-        Args:
-            inputs: dict类型。字典中包含inputs_attrs中的对象名到计算图Variable的映射，inputs中至少会包含inputs_attr中定义的对象
-        Return:
-           需要输出的计算图变量，输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-
-        """
-        raise NotImplementedError()
-
-    def postprocess(self, rt_outputs):
-        """每个训练或推理step后针对当前batch的task_layer的runtime计算结果进行相关后处理。注意，rt_outputs除了包含build方法，还自动包含了loss的计算结果。"""
-        pass
-        
-    def epoch_postprocess(self, post_inputs):
-        pass
-

paddlepalm/__init__.py

@@ -9,6 +9,7 @@ import head
 
 
 from trainer import Trainer
+from multihead_trainer import MultiHeadTrainer
 
 del interface
 del task_instance

paddlepalm/_downloader.py

@@ -34,13 +34,16 @@ _items = {
     'pretrain': {'ernie-en-uncased-large': 'https://ernie.bj.bcebos.com/ERNIE_Large_en_stable-2.0.0.tar.gz',
                  'bert-en-uncased-large': 'https://bert-models.bj.bcebos.com/uncased_L-24_H-1024_A-16.tar.gz',
                  'bert-en-uncased-base': 'https://bert-models.bj.bcebos.com/uncased_L-12_H-768_A-12.tar.gz',
+                 'ernie-ch-uncased-base':'https://ernie.bj.bcebos.com/ERNIE_1.0_max-len-512.tar.gz',
+                 'roberta-cn-base': 'https://bert-models.bj.bcebos.com/chinese_roberta_wwm_ext_L-12_H-768_A-12.tar.gz',
+                 'roberta-cn-large': 'https://bert-models.bj.bcebos.com/chinese_roberta_wwm_large_ext_L-24_H-1024_A-16.tar.gz',
                  'utils': None},
     'reader': {'utils': None},
     'backbone': {'utils': None},
     'tasktype': {'utils': None},
 }
 
-def _download(item, scope, path, silent=False):
+def _download(item, scope, path, silent=False, convert=False):
     data_url = _items[item][scope]
     if data_url == None:
         return
@@ -100,6 +103,7 @@ def _download(item, scope, path, silent=False):
             os.removedirs(source_path)
         if not silent:
             print ('done!')
+        if convert:
             if not silent:
                 print ('Converting params...', end=" ")
             _convert(data_dir, silent)

paddlepalm/backbone/base_backbone.py

@@ -15,7 +15,7 @@
 """v1.1"""
 
 
-class BaseBackbone(object):
+class Backbone(object):
     """interface of backbone model."""
 
     def __init__(self, config, phase):
@@ -58,52 +58,3 @@ class BaseBackbone(object):
             """
         raise NotImplementedError()
 
-
-
-
-class task_paradigm(object):
-
-    def __init__(self, config, phase, backbone_config):
-        """
-            config: dict类型。描述了 任务实例(task instance)+多任务配置文件 中定义超参数
-            phase: str类型。运行阶段，目前支持train和predict
-            """
-
-    @property
-    def inputs_attrs(self):
-        """描述task_layer需要从reader, backbone等输入对象集合所读取到的输入对象的属性，第一级key为对象集和的名字，如backbone，reader等（后续会支持更灵活的输入），第二级key为对象集和中各对象的属性，包括对象的名字，shape和dtype。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个对象集及其输入对象的属性描述。"""
-        raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述task输出对象的属性，包括对象的名字，shape和dtype。输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-        当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输入对象的属性描述。注意，训练阶段必须包含名为loss的输出对象。
-            """
-
-        raise NotImplementedError()
-
-    @property
-    def epoch_inputs_attrs(self):
-        return {}
-
-    def build(self, inputs, scope_name=""):
-        """建立task_layer的计算图。将符合inputs_attrs描述的来自各个对象集的静态图Variables映射成符合outputs_attr描述的静态图Variable输出。
-        Args:
-            inputs: dict类型。字典中包含inputs_attrs中的对象名到计算图Variable的映射，inputs中至少会包含inputs_attr中定义的对象
-        Return:
-           需要输出的计算图变量，输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-
-        """
-        raise NotImplementedError()
-
-    def postprocess(self, rt_outputs):
-        """每个训练或推理step后针对当前batch的task_layer的runtime计算结果进行相关后处理。注意，rt_outputs除了包含build方法，还自动包含了loss的计算结果。"""
-        pass
-        
-    def epoch_postprocess(self, post_inputs):
-        pass
-

paddlepalm/backbone/bert.py

@@ -23,28 +23,37 @@ from paddle import fluid
 from paddle.fluid import layers
 
 from paddlepalm.backbone.utils.transformer import pre_process_layer, encoder
-from paddlepalm.backbone.base_backbone import BaseBackbone
+from paddlepalm.backbone.base_backbone import Backbone
 
 
-class BERT(BaseBackbone):
+class BERT(Backbone):
 
 
-    def __init__(hidden_size, num_hidden_layers, num_attention_heads, vocab_size, \
+    def __init__(self, hidden_size, num_hidden_layers, num_attention_heads, vocab_size, \
           max_position_embeddings, type_vocab_size, hidden_act, hidden_dropout_prob, \
-          attention_probs_dropout_prob, initializer_range, phase='train'):
-        config = {}
-        config['hidden_size'] = hidden_size
-        config['num_hidden_layers'] = num_hidden_layers
-        config['num_attention_heads'] = num_attention_heads
-        config['vocab_size'] = vocab_size
-        config['max_position_embeddings'] = max_position_embeddings
-        config['type_vocab_size'] = sent_type_vocab_size
-        config['hidden_act'] = hidden_act
-        config['hidden_dropout_prob'] = hidden_dropout_prob
-        config['attention_probs_dropout_prob'] = attention_probs_dropout_prob
-        config['initializer_range'] = initializer_range
-
-        self.from_config(config, phase=phase)
+          attention_probs_dropout_prob, initializer_range, is_pairwise=False, phase='train'):
+     
+        self._emb_size = hidden_size
+        self._n_layer = num_hidden_layers
+        self._n_head = num_attention_heads
+        self._voc_size = vocab_size
+        self._max_position_seq_len = max_position_embeddings
+        self._sent_types = type_vocab_size
+
+       
+        self._hidden_act = hidden_act
+        self._prepostprocess_dropout = hidden_dropout_prob
+        self._attention_dropout = attention_probs_dropout_prob
+
+        self._word_emb_name = "word_embedding"
+        self._pos_emb_name = "pos_embedding"
+        self._sent_emb_name = "sent_embedding"
+        self._task_emb_name = "task_embedding"
+        self._emb_dtype = "float32"
+        self._phase = phase
+        self._is_pairwise = is_pairwise
+        self._param_initializer = fluid.initializer.TruncatedNormal(
+            scale=initializer_range)
 
     @classmethod
     def from_config(self, config, phase='train'):
@@ -62,40 +71,57 @@ class BERT(BaseBackbone):
             "{} is required to initialize ERNIE".format('attention_probs_dropout_prob')
         assert 'initializer_range' in config, "{} is required to initialize ERNIE".format('initializer_range')
 
-        # self._is_training = phase == 'train' # backbone一般不用关心运行阶段，因为outputs在任何阶段基本不会变
-        self._emb_size = config["hidden_size"]
-        self._n_layer = config["num_hidden_layers"]
-        self._n_head = config["num_attention_heads"]
-        self._voc_size = config["vocab_size"]
-        self._max_position_seq_len = config["max_position_embeddings"]
-        self._sent_types = config["type_vocab_size"]
-        self._hidden_act = config["hidden_act"]
-        self._prepostprocess_dropout = config["hidden_dropout_prob"]
-        self._attention_dropout = config["attention_probs_dropout_prob"]
-
-        self._word_emb_name = "word_embedding"
-        self._pos_emb_name = "pos_embedding"
-        self._sent_emb_name = "sent_embedding"
-
-        # Initialize all weigths by truncated normal initializer, and all biases 
-        # will be initialized by constant zero by default.
-        self._param_initializer = fluid.initializer.TruncatedNormal(
-            scale=config["initializer_range"])
+        hidden_size = config['hidden_size']
+        num_hidden_layers = config['num_hidden_layers']
+        num_attention_heads = config['num_attention_heads']
+        vocab_size = config['vocab_size']
+        max_position_embeddings = config['max_position_embeddings']
+        if 'sent_type_vocab_size' in config:
+            sent_type_vocab_size = config['sent_type_vocab_size']
+        else:
+            sent_type_vocab_size = config['type_vocab_size']
+
+        hidden_act = config['hidden_act']
+        hidden_dropout_prob = config['hidden_dropout_prob']
+        attention_probs_dropout_prob = config['attention_probs_dropout_prob']
+        initializer_range = config['initializer_range']
+        if 'is_pairwise' in config:
+            is_pairwise = config['is_pairwise']
+        else:
+            is_pairwise = False
+
+        return self(hidden_size, num_hidden_layers, num_attention_heads, vocab_size, \
+          max_position_embeddings, sent_type_vocab_size, \
+          hidden_act, hidden_dropout_prob, attention_probs_dropout_prob, initializer_range, is_pairwise, phase)
 
     @property
     def inputs_attr(self):
-        return {"token_ids": [[-1, -1], 'int64'],
+        ret = {"token_ids": [[-1, -1], 'int64'],
                "position_ids": [[-1, -1], 'int64'],
                "segment_ids": [[-1, -1], 'int64'],
-                "input_mask": [[-1, -1, 1], 'float32']}
+               "input_mask": [[-1, -1, 1], 'float32'],
+               }
+        if self._is_pairwise and self._phase=='train':
+            ret.update({"token_ids_neg": [[-1, -1], 'int64'],
+                        "position_ids_neg": [[-1, -1], 'int64'],
+                        "segment_ids_neg": [[-1, -1], 'int64'],
+                        "input_mask_neg": [[-1, -1, 1], 'float32'],
+                        })
+        return ret
 
     @property
     def outputs_attr(self):
-        return {"word_embedding": [[-1, -1, self._emb_size], 'float32'],
+        ret = {"word_embedding": [[-1, -1, self._emb_size], 'float32'],
                "embedding_table": [[-1, self._voc_size, self._emb_size], 'float32'],
                "encoder_outputs": [[-1, -1, self._emb_size], 'float32'],
                "sentence_embedding": [[-1, self._emb_size], 'float32'],
                "sentence_pair_embedding": [[-1, self._emb_size], 'float32']}
+        if self._is_pairwise and self._phase == 'train':
+            ret.update({"word_embedding_neg": [[-1, -1, self._emb_size], 'float32'],
+                        "encoder_outputs_neg": [[-1, -1, self._emb_size], 'float32'],
+                        "sentence_embedding_neg": [[-1, self._emb_size], 'float32'],
+                        "sentence_pair_embedding_neg": [[-1, self._emb_size], 'float32']})
+        return ret 
 
     def build(self, inputs, scope_name=""):
         src_ids = inputs['token_ids']
@@ -104,6 +130,21 @@ class BERT(BaseBackbone):
         input_mask = inputs['input_mask']
 
         self._emb_dtype = 'float32'
+
+        input_buffer = {}
+        output_buffer = {}
+        input_buffer['base'] = [src_ids, pos_ids, sent_ids, input_mask]
+        output_buffer['base'] = {}
+
+        if self._is_pairwise and self._phase =='train':
+            src_ids = inputs['token_ids_neg']
+            pos_ids = inputs['position_ids_neg']
+            sent_ids = inputs['segment_ids_neg']
+            input_mask = inputs['input_mask_neg']
+            input_buffer['neg'] = [src_ids, pos_ids, sent_ids, input_mask]
+            output_buffer['neg'] = {}
+        
+        for key, (src_ids, pos_ids, sent_ids, input_mask) in input_buffer.items():
             # padding id in vocabulary must be set to 0
             emb_out = fluid.embedding(
                 input=src_ids,
@@ -174,12 +215,25 @@ class BERT(BaseBackbone):
                 param_attr=fluid.ParamAttr(
                     name=scope_name+"pooled_fc.w_0", initializer=self._param_initializer),
                 bias_attr=scope_name+"pooled_fc.b_0")
-
-        return {'embedding_table': embedding_table,
-                'word_embedding': emb_out,
-                'encoder_outputs': enc_out,
-                'sentence_embedding': next_sent_feat,
-                'sentence_pair_embedding': next_sent_feat}
+            output_buffer[key]['word_embedding'] = emb_out
+            output_buffer[key]['encoder_outputs'] = enc_out
+            output_buffer[key]['sentence_embedding'] = next_sent_feat
+            output_buffer[key]['sentence_pair_embedding'] = next_sent_feat
+        
+        ret = {}
+        ret['embedding_table'] = embedding_table
+        ret['word_embedding'] = output_buffer['base']['word_embedding']
+        ret['encoder_outputs'] = output_buffer['base']['encoder_outputs']
+        ret['sentence_embedding'] = output_buffer['base']['sentence_embedding']
+        ret['sentence_pair_embedding'] = output_buffer['base']['sentence_pair_embedding']
+
+        if self._is_pairwise and self._phase == 'train':
+            ret['word_embedding_neg'] = output_buffer['neg']['word_embedding']
+            ret['encoder_outputs_neg'] = output_buffer['neg']['encoder_outputs']
+            ret['sentence_embedding_neg'] = output_buffer['neg']['sentence_embedding']
+            ret['sentence_pair_embedding_neg'] = output_buffer['neg']['sentence_pair_embedding']
+        
+        return ret
                     
     def postprocess(self, rt_outputs):
         pass

paddlepalm/backbone/ernie.py

@@ -24,14 +24,14 @@ from paddle import fluid
 from paddle.fluid import layers
 
 from paddlepalm.backbone.utils.transformer import pre_process_layer, encoder
-from paddlepalm.backbone.base_backbone import BaseBackbone
+from paddlepalm.backbone.base_backbone import Backbone
 
 
-class ERNIE(BaseBackbone):
+class ERNIE(Backbone):
     
     def __init__(self, hidden_size, num_hidden_layers, num_attention_heads, vocab_size, \
           max_position_embeddings, sent_type_vocab_size, task_type_vocab_size, \
-          hidden_act, hidden_dropout_prob, attention_probs_dropout_prob, initializer_range, phase='train'):
+          hidden_act, hidden_dropout_prob, attention_probs_dropout_prob, initializer_range, is_pairwise=False, phase='train'):
 
         # self._is_training = phase == 'train' # backbone一般不用关心运行阶段，因为outputs在任何阶段基本不会变
  
@@ -53,7 +53,8 @@ class ERNIE(BaseBackbone):
         self._sent_emb_name = "sent_embedding"
         self._task_emb_name = "task_embedding"
         self._emb_dtype = "float32"
-
+        self._is_pairwise = is_pairwise
+        self._phase=phase
         self._param_initializer = fluid.initializer.TruncatedNormal(
             scale=initializer_range)
 
@@ -65,7 +66,7 @@ class ERNIE(BaseBackbone):
         assert 'vocab_size' in config, "{} is required to initialize ERNIE".format('vocab_size')
         assert 'max_position_embeddings' in config, "{} is required to initialize ERNIE".format('max_position_embeddings')
         assert 'sent_type_vocab_size' in config or 'type_vocab_size' in config, "{} is required to initialize ERNIE".format('sent_type_vocab_size')
-        assert 'task_type_vocab_size' in config, "{} is required to initialize ERNIE".format('task_type_vocab_size')
+        # assert 'task_type_vocab_size' in config, "{} is required to initialize ERNIE".format('task_type_vocab_size')
         assert 'hidden_act' in config, "{} is required to initialize ERNIE".format('hidden_act')
         assert 'hidden_dropout_prob' in config, "{} is required to initialize ERNIE".format('hidden_dropout_prob')
         assert 'attention_probs_dropout_prob' in config, "{} is required to initialize ERNIE".format('attention_probs_dropout_prob')
@@ -80,40 +81,76 @@ class ERNIE(BaseBackbone):
             sent_type_vocab_size = config['sent_type_vocab_size']
         else:
             sent_type_vocab_size = config['type_vocab_size']
+        if 'task_type_vocab_size' in config:
             task_type_vocab_size = config['task_type_vocab_size']
+        else:
+            task_type_vocab_size = config['type_vocab_size']
         hidden_act = config['hidden_act']
         hidden_dropout_prob = config['hidden_dropout_prob']
         attention_probs_dropout_prob = config['attention_probs_dropout_prob']
         initializer_range = config['initializer_range']
+        if 'is_pairwise' in config:
+            is_pairwise = config['is_pairwise']
+        else:
+            is_pairwise = False
         
         return cls(hidden_size, num_hidden_layers, num_attention_heads, vocab_size, \
           max_position_embeddings, sent_type_vocab_size, task_type_vocab_size, \
-          hidden_act, hidden_dropout_prob, attention_probs_dropout_prob, initializer_range, phase=phase)
+          hidden_act, hidden_dropout_prob, attention_probs_dropout_prob, initializer_range, is_pairwise, phase=phase)
 
     @property
     def inputs_attr(self):
-        return {"token_ids": [[-1, -1], 'int64'],
+        ret = {"token_ids": [[-1, -1], 'int64'],
                "position_ids": [[-1, -1], 'int64'],
                "segment_ids": [[-1, -1], 'int64'],
                "input_mask": [[-1, -1, 1], 'float32'],
                "task_ids": [[-1,-1], 'int64']}
+        if self._is_pairwise and self._phase=='train':
+            ret.update({"token_ids_neg": [[-1, -1], 'int64'],
+                        "position_ids_neg": [[-1, -1], 'int64'],
+                        "segment_ids_neg": [[-1, -1], 'int64'],
+                        "input_mask_neg": [[-1, -1, 1], 'float32'],
+                        "task_ids_neg": [[-1,-1], 'int64']
+                        })
+        return ret
+                
 
     @property
     def outputs_attr(self):
-        return {"word_embedding": [[-1, -1, self._emb_size], 'float32'],
+        ret = {"word_embedding": [[-1, -1, self._emb_size], 'float32'],
                "embedding_table": [[-1, self._voc_size, self._emb_size], 'float32'],
                "encoder_outputs": [[-1, -1, self._emb_size], 'float32'],
                "sentence_embedding": [[-1, self._emb_size], 'float32'],
                "sentence_pair_embedding": [[-1, self._emb_size], 'float32']}
+        if self._is_pairwise and self._phase == 'train':
+            ret.update({"word_embedding_neg": [[-1, -1, self._emb_size], 'float32'],
+                        "encoder_outputs_neg": [[-1, -1, self._emb_size], 'float32'],
+                        "sentence_embedding_neg": [[-1, self._emb_size], 'float32'],
+                        "sentence_pair_embedding_neg": [[-1, self._emb_size], 'float32']})
+        return ret 
 
     def build(self, inputs, scope_name=""):
-
         src_ids = inputs['token_ids']
         pos_ids = inputs['position_ids']
         sent_ids = inputs['segment_ids']
         input_mask = inputs['input_mask']
         task_ids = inputs['task_ids']
 
+        input_buffer = {}
+        output_buffer = {}
+        input_buffer['base'] = [src_ids, pos_ids, sent_ids, input_mask, task_ids]
+        output_buffer['base'] = {}
+
+        if self._is_pairwise and self._phase =='train':
+            src_ids = inputs['token_ids_neg']
+            pos_ids = inputs['position_ids_neg']
+            sent_ids = inputs['segment_ids_neg']
+            input_mask = inputs['input_mask_neg']
+            task_ids = inputs['task_ids_neg']
+            input_buffer['neg'] = [src_ids, pos_ids, sent_ids, input_mask, task_ids]
+            output_buffer['neg'] = {}
+
+        for key, (src_ids, pos_ids, sent_ids, input_mask, task_ids) in input_buffer.items():
             # padding id in vocabulary must be set to 0
             emb_out = fluid.embedding(
                 input=src_ids,
@@ -183,7 +220,6 @@ class ERNIE(BaseBackbone):
                 param_initializer=self._param_initializer,
                 name=scope_name+'encoder')
 
-        
             next_sent_feat = fluid.layers.slice(
                 input=enc_out, axes=[1], starts=[0], ends=[1])
             next_sent_feat = fluid.layers.reshape(next_sent_feat, [-1, next_sent_feat.shape[-1]])
@@ -195,11 +231,25 @@ class ERNIE(BaseBackbone):
                     name=scope_name+"pooled_fc.w_0", initializer=self._param_initializer),
                 bias_attr=scope_name+"pooled_fc.b_0")
             
-        return {'embedding_table': embedding_table,
-                'word_embedding': emb_out,
-                'encoder_outputs': enc_out,
-                'sentence_embedding': next_sent_feat,
-                'sentence_pair_embedding': next_sent_feat}
+            output_buffer[key]['word_embedding'] = emb_out
+            output_buffer[key]['encoder_outputs'] = enc_out
+            output_buffer[key]['sentence_embedding'] = next_sent_feat
+            output_buffer[key]['sentence_pair_embedding'] = next_sent_feat
+        
+        ret = {}
+        ret['embedding_table'] = embedding_table
+        ret['word_embedding'] = output_buffer['base']['word_embedding']
+        ret['encoder_outputs'] = output_buffer['base']['encoder_outputs']
+        ret['sentence_embedding'] = output_buffer['base']['sentence_embedding']
+        ret['sentence_pair_embedding'] = output_buffer['base']['sentence_pair_embedding']
+
+        if self._is_pairwise and self._phase == 'train':
+            ret['word_embedding_neg'] = output_buffer['neg']['word_embedding']
+            ret['encoder_outputs_neg'] = output_buffer['neg']['encoder_outputs']
+            ret['sentence_embedding_neg'] = output_buffer['neg']['sentence_embedding']
+            ret['sentence_pair_embedding_neg'] = output_buffer['neg']['sentence_pair_embedding']
+        
+        return ret
 
     def postprocess(self, rt_outputs):
         pass

paddlepalm/distribute/__init__.py

@@ -5,5 +5,5 @@ import multiprocessing
 gpu_dev_count = int(fluid.core.get_cuda_device_count())
 cpu_dev_count = int(os.environ.get('CPU_NUM', multiprocessing.cpu_count()))
 
-from reader import yield_pieces, data_feeder 
+from reader import yield_pieces, data_feeder, decode_fake
 

paddlepalm/distribute/reader.py

@@ -11,8 +11,8 @@ def yield_pieces(data, distribute_strategy, batch_size):
         distribute_strategy: support s=split, c=copy, u=unstack,
         """
     assert batch_size % dev_count == 0, "batch_size need to be integer times larger than dev_count."
-    print('data in yield pieces')
-    print(len(data))
+    # print('data in yield pieces')
+    # print(len(data))
 
     assert type(data) == type(distribute_strategy), [type(data), type(distribute_strategy)]
     assert len(data) == len(distribute_strategy), [len(data), len(distribute_strategy)]
@@ -24,7 +24,6 @@ def yield_pieces(data, distribute_strategy, batch_size):
         assert isinstance(data, list), "the input data must be a list or dict, and contained with multiple tensors."
         data_list = data
         ds_list = distribute_strategy
-
     stride = batch_size // dev_count
     p = stride
     # while p < len(data_list) + stride:
@@ -53,12 +52,11 @@ def yield_pieces(data, distribute_strategy, batch_size):
         if type(data) == dict:
             yield dict(zip(*[keys, temp]))
         else:
-            print('yielded pieces')
-            print(len(temp))
+            # print('yielded pieces')
+            # print(len(temp))
             yield temp
 
 def data_feeder(reader, postprocess_fn=None, prefetch_steps=2):
-
     if postprocess_fn is None:
         def postprocess_fn(batch):
             return batch
@@ -98,6 +96,7 @@ def data_feeder(reader, postprocess_fn=None, prefetch_steps=2):
                 flag = idx-len(batches) < -num_pad
                 # if num_pad > 0:
                 #     num_pad -= 1
+                # batch = postprocess_fn(batch, id)
                 batch = postprocess_fn(batch)
                 batch_buf.append(batch)
                 flag_buf.append(flag)
@@ -107,3 +106,15 @@ def data_feeder(reader, postprocess_fn=None, prefetch_steps=2):
     queue.join()
 
 
+def decode_fake(nums, mask, bs):
+    n_t = 0
+    for flag in mask: 
+        if not flag:
+            break
+        n_t = n_t + 1
+    
+    n_f = len(mask) - n_t
+    p1 = nums - (n_t-1) * bs
+    each_f = p1 / (n_f+1)
+    return each_f * n_f
+

paddlepalm/head/__init__.py

 
 from cls import Classify
-# from match import Match
-# from mrc import MRC
-# from mlm import MaskLM
+from match import Match
+from ner import SequenceLabel
+from mrc import MRC
+from mlm import MaskLM

paddlepalm/head/base_head.py

@@ -13,16 +13,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import os
+import json
 
-class BaseHead(object):
+class Head(object):
 
-    def __init__(self, config, phase, backbone_config):
+    def __init__(self, phase='train'):
         """
             config: dict类型。描述了 任务实例(task instance)+多任务配置文件 中定义超参数
             phase: str类型。运行阶段，目前支持train和predict
             """
         self._stop_gradient = {}
+        self._phase = phase
         self._prog = None
+        self._results_buffer = []
 
     @property
     def inputs_attrs(self):
@@ -67,10 +71,31 @@ class BaseHead(object):
         raise NotImplementedError()
         
 
-    def postprocess(self, rt_outputs):
+    def batch_postprocess(self, rt_outputs):
         """每个训练或推理step后针对当前batch的task_layer的runtime计算结果进行相关后处理。注意，rt_outputs除了包含build方法，还自动包含了loss的计算结果。"""
-        pass
+        if isinstance(rt_outputs, dict):
+            keys = rt_outputs.keys()
+            vals = [rt_outputs[k] for k in keys]
+            lens = [len(v) for v in vals]
+            if len(set(lens)) == 1:
+                results = [dict(zip(*[keys, i])) for i in zip(*vals)]
+                self._results_buffer.extend(results)
+                return results
+            else:
+                print('WARNING: irregular output results. visualize failed.')
+                self._results_buffer.append(rt_outputs)
+        return None
+        
+    def epoch_postprocess(self, post_inputs, output_dir=None):
+        if output_dir is not None:
+            for i in self._results_buffer:
+                print(i)
+        else:
+            if not os.path.exists(output_dir):
+                os.makedirs(output_dir)
+            with open(os.path.join(output_dir, self._phase), 'w') as writer:
+                for i in self._results_buffer:
+                    writer.write(json.dumps(i)+'\n')
+            
         
-    def epoch_postprocess(self, post_inputs):
-        pass
 

paddlepalm/head/cls.py

@@ -87,11 +87,13 @@ class Classify(BaseHead):
             self._preds.extend(preds.tolist())
             return preds
 
-    def epoch_postprocess(self, post_inputs):
+    def epoch_postprocess(self, post_inputs, output_dir=None):
         # there is no post_inputs needed and not declared in epoch_inputs_attrs, hence no elements exist in post_inputs
         if not self._is_training:
-            if self._pred_output_path is None:
-                raise ValueError('argument pred_output_path not found in config. Please add it into config dict/file.')
+            if output_dir is None:
+                for p in self._preds:
+                    print(p)
+            else:
                 with open(os.path.join(self._pred_output_path, 'predictions.json'), 'w') as writer:
                     for p in self._preds:
                         writer.write(str(p)+'\n')

paddlepalm/head/match.py

@@ -13,41 +13,66 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+
 import paddle.fluid as fluid
 from paddle.fluid import layers
-from paddlepalm.interface import task_paradigm
+from paddlepalm.head.base_head import Head
 import numpy as np
 import os
+import json
+
+
+def computeHingeLoss(pos, neg, margin):
+    loss_part1 = fluid.layers.elementwise_sub(
+        fluid.layers.fill_constant_batch_size_like(
+            input=pos, shape=[-1, 1], value=margin, dtype='float32'), pos)
+    loss_part2 = fluid.layers.elementwise_add(loss_part1, neg)
+    loss_part3 = fluid.layers.elementwise_max(
+        fluid.layers.fill_constant_batch_size_like(
+            input=loss_part2, shape=[-1, 1], value=0.0, dtype='float32'), loss_part2)
+    return loss_part3
 
-class TaskParadigm(task_paradigm):
+
+class Match(Head):
     '''
     matching
     '''
-    def __init__(self, config, phase, backbone_config=None):
+   
+    def __init__(self, num_classes, input_dim, dropout_prob=0.0, param_initializer_range=0.02, \
+        learning_strategy='pointwise', margin=0.5, phase='train'):
+
+        """  
+        Args:
+            phase: train, eval, pred
+            lang: en, ch, ...
+            learning_strategy: pointwise, pairwise
+        """
+        
         self._is_training = phase == 'train'
-        self._hidden_size = backbone_config['hidden_size']
+        self._hidden_size = input_dim
     
-        if 'initializer_range' in config:
-            self._param_initializer = config['initializer_range']
-        else:
+        self._num_classes = num_classes
+
+        self._dropout_prob = dropout_prob if phase == 'train' else 0.0
         self._param_initializer = fluid.initializer.TruncatedNormal(
-                scale=backbone_config.get('initializer_range', 0.02))
-        if 'dropout_prob' in config:
-            self._dropout_prob = config['dropout_prob']
-        else:
-            self._dropout_prob = backbone_config.get('hidden_dropout_prob', 0.0)
+            scale=param_initializer_range)
+        self._learning_strategy = learning_strategy 
+        self._margin = margin
 
-        self._pred_output_path = config.get('pred_output_path', None)
-        self._preds = []
     
+        self._preds = []
+        self._preds_logits = []
     
     @property
     def inputs_attrs(self):
-        if self._is_training:
-            reader = {"label_ids": [[-1, 1], 'int64']}
-        else:
         reader = {}
         bb = {"sentence_pair_embedding": [[-1, self._hidden_size], 'float32']}
+        if self._is_training:
+            if self._learning_strategy == 'pointwise':
+                reader["label_ids"] = [[-1], 'int64']
+            elif self._learning_strategy == 'pairwise':
+                bb["sentence_pair_embedding_neg"] = [[-1, self._hidden_size], 'float32']
+
         return {'reader': reader, 'backbone': bb}
 
     @property
@@ -55,51 +80,110 @@ class TaskParadigm(task_paradigm):
         if self._is_training:
             return {"loss": [[1], 'float32']}
         else:
-            return {"logits": [[-1, 2], 'float32']}
+            if self._learning_strategy=='paiwise':
+                return {"probs": [[-1, 1], 'float32']}
+            else:
+                return {"logits": [[-1, 2], 'float32'],
+                        "probs": [[-1, 2], 'float32']}
 
     def build(self, inputs, scope_name=""):
-        if self._is_training:
-            labels = inputs["reader"]["label_ids"] 
-        cls_feats = inputs["backbone"]["sentence_pair_embedding"]
 
+        # inputs          
+        cls_feats = inputs["backbone"]["sentence_pair_embedding"] 
         if self._is_training:
             cls_feats = fluid.layers.dropout(
                 x=cls_feats,
                 dropout_prob=self._dropout_prob,
                 dropout_implementation="upscale_in_train")
+            if self._learning_strategy == 'pairwise':
+                cls_feats_neg = inputs["backbone"]["sentence_pair_embedding_neg"]
+                cls_feats_neg = fluid.layers.dropout(
+                x=cls_feats_neg,
+                dropout_prob=self._dropout_prob,
+                dropout_implementation="upscale_in_train")
+            elif self._learning_strategy == 'pointwise':
+                labels = inputs["reader"]["label_ids"] 
         
+        # loss
+        # for pointwise
+        if self._learning_strategy == 'pointwise':
             logits = fluid.layers.fc(
                 input=cls_feats,
-            size=2,
+                size=self._num_classes,
                 param_attr=fluid.ParamAttr(
                     name=scope_name+"cls_out_w",
                     initializer=self._param_initializer),
                 bias_attr=fluid.ParamAttr(
                     name=scope_name+"cls_out_b",
                     initializer=fluid.initializer.Constant(0.)))
-
+            probs = fluid.layers.softmax(logits)
             if self._is_training:
-            ce_loss, probs = fluid.layers.softmax_with_cross_entropy(
-                logits=logits, label=labels, return_softmax=True)
+                ce_loss = fluid.layers.cross_entropy(
+                    input=probs, label=labels)
                 loss = fluid.layers.mean(x=ce_loss)
                 return {'loss': loss}
+            # for pred
             else:
-            return {'logits': logits}
+                return {'logits': logits,
+                        'probs': probs}
+        # for pairwise
+        elif self._learning_strategy == 'pairwise':
+            pos_score = fluid.layers.fc(
+                input=cls_feats,
+                size=1,
+                act = "sigmoid",
+                param_attr=fluid.ParamAttr(
+                    name=scope_name+"cls_out_w_pr",
+                    initializer=self._param_initializer),
+                bias_attr=fluid.ParamAttr(
+                    name=scope_name+"cls_out_b_pr",
+                    initializer=fluid.initializer.Constant(0.)))
+            pos_score = fluid.layers.reshape(x=pos_score, shape=[-1, 1], inplace=True)
 
-    def postprocess(self, rt_outputs):
+            if self._is_training:
+                neg_score = fluid.layers.fc(
+                    input=cls_feats_neg,
+                    size=1,
+                    act = "sigmoid",
+                    param_attr=fluid.ParamAttr(
+                        name=scope_name+"cls_out_w_pr",
+                        initializer=self._param_initializer),
+                    bias_attr=fluid.ParamAttr(
+                        name=scope_name+"cls_out_b_pr",
+                        initializer=fluid.initializer.Constant(0.)))        
+                neg_score = fluid.layers.reshape(x=neg_score, shape=[-1, 1], inplace=True)
+        
+                loss = fluid.layers.mean(computeHingeLoss(pos_score, neg_score, self._margin))
+                return {'loss': loss}
+            # for pred
+            else:
+                return {'probs': pos_score}
+        
+
+
+    def batch_postprocess(self, rt_outputs):
         if not self._is_training:
+            probs = []
+            logits = []
+            probs = rt_outputs['probs']
+            self._preds.extend(probs.tolist())
+            if self._learning_strategy == 'pointwise':
                 logits = rt_outputs['logits']
-            preds = np.argmax(logits, -1)
-            self._preds.extend(preds.tolist())
+                self._preds_logits.extend(logits.tolist())
         
-    def epoch_postprocess(self, post_inputs):
+    def epoch_postprocess(self, post_inputs, output_dir=None):
         # there is no post_inputs needed and not declared in epoch_inputs_attrs, hence no elements exist in post_inputs
         if not self._is_training:
-            if self._pred_output_path is None:
-                raise ValueError('argument pred_output_path not found in config. Please add it into config dict/file.')
-            with open(os.path.join(self._pred_output_path, 'predictions.json'), 'w') as writer:
-                for p in self._preds:
-                    writer.write(str(p)+'\n')
-            print('Predictions saved at '+os.path.join(self._pred_output_path, 'predictions.json'))
-
-                
+            if output_dir is None:
+                raise ValueError('argument output_dir not found in config. Please add it into config dict/file.')
+            with open(os.path.join(output_dir, 'predictions.json'), 'w') as writer:
+                for i in range(len(self._preds)):
+                    if self._learning_strategy == 'pointwise':
+                        label = 0 if self._preds[i][0] > self._preds[i][1] else 1
+                        result = {'index': i, 'label': label, 'logits': self._preds_logits[i], 'probs': self._preds[i]}
+                    elif self._learning_strategy == 'pairwise':
+                        label = 0 if self._preds[i][0] < 0.5 else 1
+                        result = {'index': i, 'label': label, 'probs': self._preds[i][0]}
+                    result = json.dumps(result)
+                    writer.write(result+'\n')
+            print('Predictions saved at '+os.path.join(output_dir, 'predictions.json'))
\ No newline at end of file

paddlepalm/head/mlm.py

@@ -14,30 +14,39 @@
 # limitations under the License.
 
 import paddle.fluid as fluid
-from paddlepalm.interface import task_paradigm
+from paddlepalm.head.base_head import Head
 from paddle.fluid import layers
+import numpy as np
+import os
 from paddlepalm.backbone.utils.transformer import pre_process_layer
 
-class TaskParadigm(task_paradigm):
+class MaskLM(Head):
     '''
-    matching
+    mlm
     '''
-    def __init__(self, config, phase, backbone_config=None):
+    def __init__(self, input_dim, vocab_size, hidden_act, initializer_range, dropout_prob=0.0, \
+                 param_initializer_range=0.02, phase='train'):
         self._is_training = phase == 'train'
-        self._emb_size = backbone_config['hidden_size']
-        self._hidden_size = backbone_config['hidden_size']
-        self._vocab_size = backbone_config['vocab_size']
-        self._hidden_act = backbone_config['hidden_act']
-        self._initializer_range = backbone_config['initializer_range']
+        self._emb_size = input_dim
+        self._hidden_size = input_dim
+        self._dropout_prob = dropout_prob if phase == 'train' else 0.0
+        self._param_initializer = fluid.initializer.TruncatedNormal(
+            scale=param_initializer_range)
+        self._preds = []
+
+        self._vocab_size = vocab_size
+        self._hidden_act = hidden_act
+        self._initializer_range = initializer_range
     
     @property
     def inputs_attrs(self):
         reader = {
-            "mask_label": [[-1, 1], 'int64'],
-            "mask_pos": [[-1, 1], 'int64']}
+            "token_ids":[[-1, -1], 'int64'],
+            "mask_label": [[-1], 'int64'],
+            "mask_pos": [[-1], 'int64'],
+            }
         if not self._is_training:
             del reader['mask_label']
-            del reader['batchsize_x_seqlen']
         bb = {
             "encoder_outputs": [[-1, -1, self._hidden_size], 'float32'],
             "embedding_table": [[-1, self._vocab_size, self._emb_size], 'float32']}
@@ -54,7 +63,13 @@ class TaskParadigm(task_paradigm):
         mask_pos = inputs["reader"]["mask_pos"]
         if self._is_training:
             mask_label = inputs["reader"]["mask_label"] 
-            max_position = inputs["reader"]["batchsize_x_seqlen"] - 1
+            l1 = fluid.layers.shape(inputs["reader"]["token_ids"] )[0]
+            # bxs = inputs["reader"]["token_ids"].shape[2].value
+            l2 = fluid.layers.shape(inputs["reader"]["token_ids"][0])[0]
+            bxs = (l1*l2).astype(np.int64)
+            # max_position = inputs["reader"]["batchsize_x_seqlen"] - 1
+            max_position = bxs - 1
+
             mask_pos = fluid.layers.elementwise_min(mask_pos, max_position)
             mask_pos.stop_gradient = True
 
@@ -100,11 +115,31 @@ class TaskParadigm(task_paradigm):
             is_bias=True)
 
         if self._is_training:
-            mask_lm_loss = fluid.layers.softmax_with_cross_entropy(
-                logits=fc_out, label=mask_label)
+            inputs = fluid.layers.softmax(fc_out)
+            mask_lm_loss = fluid.layers.cross_entropy(
+                input=inputs, label=mask_label)
             loss = fluid.layers.mean(mask_lm_loss)
             return {'loss': loss}
         else:
             return {'logits': fc_out}
 
+    def batch_postprocess(self, rt_outputs):
+        if not self._is_training:
+            logits = rt_outputs['logits']
+            preds = np.argmax(logits, -1)
+            self._preds.extend(preds.tolist())
+            return preds
+
+    def epoch_postprocess(self, post_inputs, output_dir=None):
+        # there is no post_inputs needed and not declared in epoch_inputs_attrs, hence no elements exist in post_inputs
+        if not self._is_training:
+            if output_dir is None:
+                for p in self._preds:
+                    print(p)
+            else:
+                with open(os.path.join(output_dir, 'predictions.json'), 'w') as writer:
+                    for p in self._preds:
+                        writer.write(str(p)+'\n')
+                print('Predictions saved at '+os.path.join(output_dir, 'predictions.json'))
+
 

paddlepalm/head/mrc.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 import paddle.fluid as fluid
-from paddlepalm.interface import task_paradigm
+from paddlepalm.head.base_head import Head
 import collections
 import numpy as np
 import os
@@ -26,34 +26,37 @@ import json
 RawResult = collections.namedtuple("RawResult",
                                    ["unique_id", "start_logits", "end_logits"])
 
-class TaskParadigm(task_paradigm):
-    """"""
+class MRC(Head):
+    """
+    Machine Reading Comprehension
+    """
 
-    def __init__(self, config, phase, backbone_config=None):
+    def __init__(self, max_query_len, input_dim, pred_output_path=None, verbose=False, with_negative=False, do_lower_case=False, max_ans_len=None, null_score_diff_threshold=0.0, n_best_size=20, phase='train'):
 
         self._is_training = phase == 'train'
-        self._max_sequence_length = config['max_seq_len']
-        self._hidden_size = backbone_config['hidden_size']
+        self._hidden_size = input_dim
+        self._max_sequence_length = max_query_len
+ 
         self._pred_results = []
         
-        if phase == 'pred':
-            self._max_answer_length = config.get('max_answer_len', None)
-            self._null_score_diff_threshold = config.get('null_score_diff_threshold', 0.0)
-            self._n_best_size = config.get('n_best_size', 20)
-            self._pred_output_path = config.get('pred_output_path', None)
-            self._verbose = config.get('verbose', False)
-            self._with_negative = config.get('with_negative', False)
-            self._do_lower_case = config.get('do_lower_case', False)
+        output_dir = pred_output_path
+        self._max_answer_length = max_ans_len
+        self._null_score_diff_threshold = null_score_diff_threshold
+        self._n_best_size = n_best_size
+        output_dir = pred_output_path
+        self._verbose = verbose
+        self._with_negative = with_negative
+        self._do_lower_case = do_lower_case
 
 
     @property
     def inputs_attrs(self):
         if self._is_training:
-            reader = {"start_positions": [[-1, 1], 'int64'],
-                      "end_positions": [[-1, 1], 'int64'],
+            reader = {"start_positions": [[-1], 'int64'],
+                      "end_positions": [[-1], 'int64'],
                       }
         else:
-            reader = {'unique_ids': [[-1, 1], 'int64']}
+            reader = {'unique_ids': [[-1], 'int64']}
         bb = {"encoder_outputs": [[-1, -1, self._hidden_size], 'float32']}
         return {'reader': reader, 'backbone': bb}
         
@@ -70,21 +73,26 @@ class TaskParadigm(task_paradigm):
         else:
             return {'start_logits': [[-1, -1, 1], 'float32'],
                     'end_logits': [[-1, -1, 1], 'float32'],
-                    'unique_ids': [[-1, 1], 'int64']}
+                    'unique_ids': [[-1], 'int64']}
 
 
     def build(self, inputs, scope_name=""):
         if self._is_training:
             start_positions = inputs['reader']['start_positions']
             end_positions = inputs['reader']['end_positions']
-            max_position = inputs["reader"]["seqlen"] - 1
-            start_positions = fluid.layers.elementwise_min(start_positions, max_position)
-            end_positions = fluid.layers.elementwise_min(end_positions, max_position)
+            # max_position = inputs["reader"]["seqlen"] - 1
+            # start_positions = fluid.layers.elementwise_min(start_positions, max_position)
+            # end_positions = fluid.layers.elementwise_min(end_positions, max_position)
             start_positions.stop_gradient = True
             end_positions.stop_gradient = True
         else:
             unique_id = inputs['reader']['unique_ids']
 
+            # It's used to help fetch variable 'unique_ids' that will be removed in the future
+            helper_constant = fluid.layers.fill_constant(shape=[1], value=1, dtype='int64')
+            fluid.layers.elementwise_mul(unique_id, helper_constant)  
+            
+
         enc_out = inputs['backbone']['encoder_outputs']
         logits = fluid.layers.fc(
             input=enc_out,
@@ -100,9 +108,11 @@ class TaskParadigm(task_paradigm):
         start_logits, end_logits = fluid.layers.unstack(x=logits, axis=0)
 
         def _compute_single_loss(logits, positions):
-            """Compute start/end loss for mrc model"""
-            loss = fluid.layers.softmax_with_cross_entropy(
-                logits=logits, label=positions)
+            """Compute start/en
+            d loss for mrc model"""
+            inputs = fluid.layers.softmax(logits)
+            loss = fluid.layers.cross_entropy(
+                input=inputs, label=positions)
             loss = fluid.layers.mean(x=loss)
             return loss
 
@@ -117,10 +127,10 @@ class TaskParadigm(task_paradigm):
                     'unique_ids': unique_id}
 
 
-    def postprocess(self, rt_outputs):
+    def batch_postprocess(self, rt_outputs):
         """this func will be called after each step(batch) of training/evaluating/predicting process."""
         if not self._is_training:
-            unique_ids = np.squeeze(rt_outputs['unique_ids'], -1)
+            unique_ids = rt_outputs['unique_ids']
             start_logits = rt_outputs['start_logits']
             end_logits = rt_outputs['end_logits']
             for idx in range(len(unique_ids)):
@@ -139,19 +149,19 @@ class TaskParadigm(task_paradigm):
                         start_logits=s,
                         end_logits=e))
 
-    def epoch_postprocess(self, post_inputs):
+    def epoch_postprocess(self, post_inputs, output_dir=None):
         """(optional interface) this func will be called after evaluation/predicting process and each epoch during training process."""
 
         if not self._is_training:
-            if self._pred_output_path is None:
-                raise ValueError('argument pred_output_path not found in config. Please add it into config dict/file.')
+            if output_dir is None:
+                raise ValueError('argument output_dir not found in config. Please add it into config dict/file.')
             examples = post_inputs['reader']['examples']
             features = post_inputs['reader']['features']
-            if not os.path.exists(self._pred_output_path):
-                os.makedirs(self._pred_output_path)
-            output_prediction_file = os.path.join(self._pred_output_path, "predictions.json")
-            output_nbest_file = os.path.join(self._pred_output_path, "nbest_predictions.json")
-            output_null_log_odds_file = os.path.join(self._pred_output_path, "null_odds.json")
+            if not os.path.exists(output_dir):
+                os.makedirs(output_dir)
+            output_prediction_file = os.path.join(output_dir, "predictions.json")
+            output_nbest_file = os.path.join(output_dir, "nbest_predictions.json")
+            output_null_log_odds_file = os.path.join(output_dir, "null_odds.json")
             _write_predictions(examples, features, self._pred_results,
                               self._n_best_size, self._max_answer_length,
                               self._do_lower_case, output_prediction_file,
@@ -194,8 +204,9 @@ def _write_predictions(all_examples, all_features, all_results, n_best_size,
         # keep track of the minimum score of null start+end of position 0
         score_null = 1000000  # large and positive
         min_null_feature_index = 0  # the paragraph slice with min mull score
-        null_start_logit = 0  # the start logit at the slice with min null score
+        ull_start_logit = 0  # the start logit at the slice with min null score
         null_end_logit = 0  # the end logit at the slice with min null score
+    
         for (feature_index, feature) in enumerate(features):
             result = unique_id_to_result[feature.unique_id]
             start_indexes = _get_best_indexes(result.start_logits, n_best_size)

tasktype/cls.py → paddlepalm/head/ner.py

@@ -15,38 +15,44 @@
 
 import paddle.fluid as fluid
 from paddle.fluid import layers
-from paddlepalm.interface import task_paradigm
+from paddlepalm.head.base_head import Head
 import numpy as np
 import os
+import math
 
-class TaskParadigm(task_paradigm):
+class SequenceLabel(Head):
     '''
-    classification
+    Sequence label
     '''
-    def __init__(self, config, phase, backbone_config=None):
+    def __init__(self, num_classes, input_dim, dropout_prob=0.0, learning_rate=1e-3,  \
+                 param_initializer_range=0.02, phase='train'):
+        
+        """  
+        Args:
+            phase: train, eval, pred
+            lang: en, ch, ...
+        """
+
         self._is_training = phase == 'train'
-        self._hidden_size = backbone_config['hidden_size']
-        self.num_classes = config['n_classes']
+        self._hidden_size = input_dim
 
-        if 'initializer_range' in config:
-            self._param_initializer = config['initializer_range']
-        else:
+        self.num_classes = num_classes
+    
+        self._dropout_prob = dropout_prob if phase == 'train' else 0.0
         self._param_initializer = fluid.initializer.TruncatedNormal(
-                scale=backbone_config.get('initializer_range', 0.02))
-        if 'dropout_prob' in config:
-            self._dropout_prob = config['dropout_prob']
-        else:
-            self._dropout_prob = backbone_config.get('hidden_dropout_prob', 0.0)
-        self._pred_output_path = config.get('pred_output_path', None)
+            scale=param_initializer_range)
+
+        self.learning_rate = learning_rate
         self._preds = []
 
+
     @property
     def inputs_attrs(self):
-        if self._is_training:
-            reader = {"label_ids": [[-1, 1], 'int64']}
-        else:
         reader = {}
-        bb = {"sentence_embedding": [[-1, self._hidden_size], 'float32']}
+        bb = {"encoder_outputs": [[-1, -1, -1], 'float32']}
+        if self._is_training:
+            reader["label_ids"] = [[-1, -1], 'int64']
+            reader["seq_lens"] = [[-1], 'int64']
         return {'reader': reader, 'backbone': bb}
 
     @property
@@ -54,48 +60,67 @@ class TaskParadigm(task_paradigm):
         if self._is_training:
             return {'loss': [[1], 'float32']}
         else:
-            return {'logits': [[-1, self.num_classes], 'float32']}
+            return {'emission': [[-1, self.num_classes], 'float32']}
 
     def build(self, inputs, scope_name=''):
-        sent_emb = inputs['backbone']['sentence_embedding']
+        token_emb = inputs['backbone']['encoder_outputs']
         if self._is_training:
             label_ids = inputs['reader']['label_ids']
-            cls_feats = fluid.layers.dropout(
-                x=sent_emb,
-                dropout_prob=self._dropout_prob,
-                dropout_implementation="upscale_in_train")
+            seq_lens = inputs['reader']['seq_lens']
 
-        logits = fluid.layers.fc(
-            input=sent_emb,
+        emission = fluid.layers.fc(
             size=self.num_classes,
+            input=token_emb,
             param_attr=fluid.ParamAttr(
-                name=scope_name+"cls_out_w",
-                initializer=self._param_initializer),
+                initializer=self._param_initializer,
+                regularizer=fluid.regularizer.L2DecayRegularizer(
+                    regularization_coeff=1e-4)),
             bias_attr=fluid.ParamAttr(
-                name=scope_name+"cls_out_b", initializer=fluid.initializer.Constant(0.)))
+                name=scope_name+"cls_out_b", initializer=fluid.initializer.Constant(0.)),
+            num_flatten_dims=2)
 
         if self._is_training:
-            loss = fluid.layers.softmax_with_cross_entropy(
-                logits=logits, label=label_ids)
-            loss = layers.mean(loss)
-            return {"loss": loss}
+
+            # compute loss
+            crf_cost = fluid.layers.linear_chain_crf(  
+                input=emission,
+                label=label_ids,
+                param_attr=fluid.ParamAttr(
+                    name=scope_name+'crfw', learning_rate=self.learning_rate),
+                length=seq_lens)
+
+            avg_cost = fluid.layers.mean(x=crf_cost)
+            crf_decode = fluid.layers.crf_decoding(
+                input=emission,
+                param_attr=fluid.ParamAttr(name=scope_name+'crfw'),
+                length=seq_lens)
+
+            (precision, recall, f1_score, num_infer_chunks, num_label_chunks,
+            num_correct_chunks) = fluid.layers.chunk_eval(
+                input=crf_decode,
+                label=label_ids,
+                chunk_scheme="IOB",
+                num_chunk_types=int(math.ceil((self.num_classes - 1) / 2.0)),
+                seq_length=seq_lens)
+            chunk_evaluator = fluid.metrics.ChunkEvaluator()
+            chunk_evaluator.reset()
+
+            return {"loss": avg_cost}
         else:
-            return {"logits":logits}
+            return {"emission": emission} 
 
-    def postprocess(self, rt_outputs):
+    def batch_postprocess(self, rt_outputs):
         if not self._is_training:
-            logits = rt_outputs['logits']
-            preds = np.argmax(logits, -1)
+            emission = rt_outputs['emission']
+            preds = np.argmax(emission, -1)
             self._preds.extend(preds.tolist())
 
-    def epoch_postprocess(self, post_inputs):
+    def epoch_postprocess(self, post_inputs, output_dir=None):
         # there is no post_inputs needed and not declared in epoch_inputs_attrs, hence no elements exist in post_inputs
         if not self._is_training:
-            if self._pred_output_path is None:
-                raise ValueError('argument pred_output_path not found in config. Please add it into config dict/file.')
-            with open(os.path.join(self._pred_output_path, 'predictions.json'), 'w') as writer:
+            if output_dir is None:
+                raise ValueError('argument output_dir not found in config. Please add it into config dict/file.')
+            with open(os.path.join(output_dir, 'predictions.json'), 'w') as writer:
                 for p in self._preds:
                     writer.write(str(p)+'\n')
-            print('Predictions saved at '+os.path.join(self._pred_output_path, 'predictions.json'))
-
-                
+            print('Predictions saved at '+os.path.join(output_dir, 'predictions.json'))

paddlepalm/mtl_controller.py

@@ -31,7 +31,7 @@ from paddlepalm.utils.saver import init_pretraining_params, init_checkpoint
 from paddlepalm.utils.config_helper import PDConfig
 from paddlepalm.utils.print_helper import print_dict
 from paddlepalm.utils.reader_helper import create_net_inputs, create_iterator_fn, create_joint_iterator_fn, merge_input_attrs 
-from paddlepalm.distribute import data_feeder
+from paddlepalm.distribute import data_feeder, decode_fake
 
 from default_settings import *
 from task_instance import TaskInstance, check_instances
@@ -186,13 +186,20 @@ def _fit_attr(conf, fit_attr, strict=False):
 
 def create_feed_batch_process_fn(net_inputs):
 
-    def feed_batch_process_fn(data):
+
+    def feed_batch_process_fn(data, id=-1):
+        # temps = {}
+        # for i in range(len(net_inputs)):
         temp = {}
-        for q, var in net_inputs.items():
+        inputs = net_inputs[id] if id != -1 else net_inputs
+        
+        for q, var in inputs.items():
             if isinstance(var, str) or isinstance(var, unicode):
                 temp[var] = data[q]
             else:
                 temp[var.name] = data[q]
+            # temps[i] = temp
+            
         return temp
 
     return feed_batch_process_fn
@@ -221,6 +228,7 @@ class Controller(object):
         exe, dev_count = _init_env(use_gpu=mtl_conf.get('use_gpu', True))
         self.exe = exe
         self.dev_count = dev_count
+        self.batch_size = mtl_conf.get('batch_size')
 
         print_dict(mtl_conf, title='global configuration')
 
@@ -343,6 +351,7 @@ class Controller(object):
         dev_count = self.dev_count
         num_instances = len(instances)
         mrs = self.mrs
+        branch = fluid.data(name="branch",shape=[1],dtype='int64')
 
         # set first_target/main task instance
         main_inst = None
@@ -362,35 +371,51 @@ class Controller(object):
 
         # create reader, task
         # then check i/o across reader, backbone and task_layer
-        task_attrs = []
+        
+        # check_fns = {}
+        task_attrs = {}
         pred_task_attrs = []
-        for inst in instances:
-            train_reader = inst.Reader(inst.config, phase='train')
-            inst.reader['train'] = train_reader
-            train_parad = inst.Paradigm(inst.config, phase='train', backbone_config=bb_conf)
-            inst.task_layer['train'] = train_parad
-            task_attr_from_reader = _encode_inputs(train_parad.inputs_attrs['reader'], inst.name)
-            task_attrs.append(task_attr_from_reader)
+        joint_input_names = {}
+        joint_shape_and_dtypes = {}
+        name_to_position = {}
+        for i in range(num_instances):
+            # def check_tasks():
+            #     i = s 
+            #     def checkeach():
+                    
+            train_reader = instances[i].Reader(instances[i].config, phase='train')
+            instances[i].reader['train'] = train_reader
+            train_parad = instances[i].Paradigm(instances[i].config, phase='train', backbone_config=bb_conf)
+            instances[i].task_layer['train'] = train_parad
+            task_attr_from_reader = _encode_inputs(train_parad.inputs_attrs['reader'], instances[i].name)
+            task_attrs[i] = task_attr_from_reader
 
             _check_io(train_backbone.inputs_attr, train_reader.outputs_attr, in_name=bb_name+'_backbone', out_name='reader.train')
             _check_io(train_parad.inputs_attrs['reader'], train_reader.outputs_attr, in_name='task_paradigm.train.reader', out_name='reader.train')
             _check_io(train_parad.inputs_attrs['backbone'], train_backbone.outputs_attr, in_name='task_paradigm.train.backbone', out_name=bb_name+'_backbone')
-
-            if inst.is_target:
-                if 'pred_file' not in inst.config:
-                    inst.config['pred_file'] = ''
-                pred_reader = inst.Reader(inst.config, phase='pred')
-                pred_parad = inst.Paradigm(inst.config, phase='pred', backbone_config=bb_conf)
-                inst.task_layer['pred'] = pred_parad
-                task_attr_from_reader = _encode_inputs(pred_parad.inputs_attrs['reader'], inst.name)
+            # merge reader input attrs from backbone and task_instances
+            # pred_joint_input_names = []
+            # pred_joint_shape_and_dtypes = []
+            if instances[i].is_target:
+                if 'pred_file' not in instances[i].config:
+                    instances[i].config['pred_file'] = ''
+                pred_reader = instances[i].Reader(instances[i].config, phase='pred')
+                pred_parad = instances[i].Paradigm(instances[i].config, phase='pred', backbone_config=bb_conf)
+                instances[i].task_layer['pred'] = pred_parad
+                task_attr_from_reader = _encode_inputs(pred_parad.inputs_attrs['reader'], instances[i].name)
                 pred_task_attrs.append(task_attr_from_reader)
                 _check_io(pred_backbone.inputs_attr, pred_reader.outputs_attr, in_name=bb_name+'_backbone', out_name='reader.pred')
                 _check_io(pred_parad.inputs_attrs['reader'], pred_reader.outputs_attr, in_name='task_paradigm.pred.reader', out_name='reader.pred')
                 _check_io(pred_parad.inputs_attrs['backbone'], pred_backbone.outputs_attr, in_name='task_paradigm.pred.backbone', out_name=bb_name+'_backbone')
+                # pred_joint_input_names, pred_joint_shape_and_dtypes, _ = merge_input_attrs(pred_backbone.inputs_attr, pred_task_attrs, insert_taskid=False, insert_batchsize=False, insert_seqlen=False, insert_batchsize_x_seqlen=False)
+                #     return joint_input_names[i], joint_shape_and_dtypes[i], name_to_position[i], pred_joint_input_names, pred_joint_shape_and_dtypes
+                #   return checkeach
+                # check_fns[i] = check_tasks()
+            joint_input_names[i], joint_shape_and_dtypes[i], name_to_position[i] = merge_input_attrs(train_backbone.inputs_attr, task_attrs[i])
            
-        # merge reader input attrs from backbone and task_instances
-        joint_input_names, joint_shape_and_dtypes, name_to_position = merge_input_attrs(train_backbone.inputs_attr, task_attrs)
         pred_joint_input_names, pred_joint_shape_and_dtypes, _ = merge_input_attrs(pred_backbone.inputs_attr, pred_task_attrs, insert_taskid=False, insert_batchsize=False, insert_seqlen=False, insert_batchsize_x_seqlen=False)
+      
+            
         # shapes: [task_id, shapes_of_backbone, shapes_of_inst1, ..., shapes_of_instN]
 
         if DEBUG:
@@ -401,15 +426,17 @@ class Controller(object):
             print(joint_shape_and_dtypes)
 
         # load data 
-        for inst in instances:
-            print(inst.name+": preparing data...", end='')
-            inst.reader['train'].load_data()
+        data_fns={}
+        for i in range(num_instances):
+            print(instances[i].name+": preparing data...", end='')
+            instances[i].reader['train'].load_data()
             print('ok!')
 
         # merge dataset iterators and create net input vars
         iterators = []
         prefixes = []
         mrs = []
+
         for inst in instances:
             iterators.append(inst.reader['train'].iterator())
             prefixes.append(inst.name)
@@ -418,65 +445,65 @@ class Controller(object):
         joint_iterator_fn = create_joint_iterator_fn(iterators, prefixes, joint_shape_and_dtypes, mrs, name_to_position, dev_count=dev_count, verbose=VERBOSE, return_type='dict')
         self._joint_iterator_fn = joint_iterator_fn
 
-        input_attrs = [[i, j, k] for i, (j,k) in zip(joint_input_names, joint_shape_and_dtypes)]
-        pred_input_attrs = [[i, j, k] for i, (j,k) in zip(pred_joint_input_names, pred_joint_shape_and_dtypes)]
-        # net_inputs = create_net_inputs(input_attrs, async=True, iterator_fn=joint_iterator_fn, dev_count=dev_count, n_prefetch=3)
-        net_inputs = create_net_inputs(input_attrs, async=False)
-        self._net_inputs = net_inputs
-
-        # build backbone and task layers
-        train_prog = fluid.default_main_program()
-        train_init_prog = fluid.default_startup_program()
-        bb_output_vars = train_backbone.build(net_inputs, scope_name='__paddlepalm_')
-        assert sorted(bb_output_vars.keys()) == sorted(train_backbone.outputs_attr.keys())
+        input_attrs = {}
+        net_inputs = {}
+        bb_output_vars = {}
+        bb_output_fns = {}
 
+        # prepare predict vars for saving inference model
+        pred_input_attrs = [[i, j, k] for i, (j,k) in zip(pred_joint_input_names, pred_joint_shape_and_dtypes)]
         pred_prog = fluid.Program()
         pred_init_prog = fluid.Program()
+        self._pred_prog = pred_prog
 
         with fluid.program_guard(main_program = pred_prog, startup_program = pred_init_prog):
            pred_net_inputs = create_net_inputs(pred_input_attrs)
            pred_bb_output_vars = pred_backbone.build(pred_net_inputs, scope_name='__paddlepalm_')
 
-        fluid.framework.switch_main_program(train_prog)
-        fluid.framework.switch_startup_program(train_init_prog)
-
+        task_inputs = {}
         task_output_vars = {}
-        for inst in instances:
-            task_inputs = {'backbone': bb_output_vars}
-            task_inputs_from_reader = _decode_inputs(net_inputs, inst.name)
-            task_inputs['reader'] = task_inputs_from_reader
+        task_fns = {}
 
-            scope = inst.task_reuse_scope + '/'
-            with fluid.unique_name.guard(scope):
-                output_vars = inst.build_task_layer(task_inputs, phase='train', scope=scope)
-                output_vars = {inst.name+'/'+key: val for key, val in output_vars.items()}
-                old = len(task_output_vars) # for debug
-                task_output_vars.update(output_vars)
-                assert len(task_output_vars) - old == len(output_vars) # for debug
+        def get_loss(i):
+            input_attrs[i] = [[m, j, k] for m, (j,k) in zip(joint_input_names[i], joint_shape_and_dtypes[i])]
+            net_inputs[i] = create_net_inputs(input_attrs[i], async=False)
+            # net_inputs = create_net_inputs(input_attrs, async=True, iterator_fn=joint_iterator_fn, dev_count=dev_count, n_prefetch=3)
+            bb_output_vars[i] = train_backbone.build(net_inputs[i], scope_name='__paddlepalm_')
+            assert sorted(bb_output_vars[i].keys()) == sorted(train_backbone.outputs_attr.keys())
 
-            # prepare predict vars for saving inference model
-            if inst.is_target:
+            # build backbone and task layers
+            task_inputs[i] = {'backbone': bb_output_vars[i]}
+            task_inputs_from_reader = _decode_inputs(net_inputs[i], instances[i].name)
+            task_inputs[i]['reader'] = task_inputs_from_reader
+        
+            scope = instances[i].task_reuse_scope + '/'
+            with fluid.unique_name.guard(scope):
+                output_vars = instances[i].build_task_layer(task_inputs[i], phase='train', scope=scope)
+                output_vars = {instances[i].name+'/'+key: val for key, val in output_vars.items()}
+                loss_var = output_vars[instances[i].name+'/loss']
+                task_output_vars[i] = output_vars
 
+            if instances[i].is_target:
                 with fluid.program_guard(pred_prog, pred_init_prog):
-                    cur_inputs = _decode_inputs(pred_net_inputs, inst.name)
-                    inst.pred_input = cur_inputs
+                    cur_inputs = _decode_inputs(pred_net_inputs, instances[i].name)
+                    instances[i].pred_input = cur_inputs
                     pred_task_inputs = {'backbone': pred_bb_output_vars, 'reader': cur_inputs}
-                    scope = inst.task_reuse_scope + '/'
+                    scope = instances[i].task_reuse_scope + '/'
                     with fluid.unique_name.guard(scope):
-                        inst.build_task_layer(pred_task_inputs, phase='pred', scope=scope)
-
-
-        bb_fetches = {k: v.name for k,v in bb_output_vars.items()}
-        task_fetches = {k: v.name for k,v in task_output_vars.items()}
-        fetches = task_fetches
-        fetches['__task_id'] = net_inputs['__task_id'].name
-
-        # compute loss
-        task_id_var = net_inputs['__task_id']
-        task_id_vec = fluid.one_hot(task_id_var, num_instances)
-        losses = fluid.layers.concat([task_output_vars[inst.name+'/loss'] for inst in instances], axis=0)
-        loss = layers.reduce_sum(task_id_vec * losses)
-
+                        instances[i].build_task_layer(pred_task_inputs, phase='pred', scope=scope)
+            return loss_var
+
+        for i in range(num_instances):
+            def task_loss():
+                task_id = i
+                return lambda: get_loss(task_id)
+            task_fns[i] = task_loss()
+
+        loss = layers.switch_case(
+            branch_index=branch,
+            branch_fns=task_fns
+        )
+        self._switched_loss = loss.name
         main_reader = main_inst.reader['train']
 
         num_examples = main_reader.num_examples
@@ -514,9 +541,9 @@ class Controller(object):
         self.saver_program = fluid.default_main_program()
 
         self.main_inst = main_inst
-        self.fetches = fetches
         self.has_init_train = True
         self.has_init_pred = True
+        self._net_inputs = net_inputs
 
         self.exe.run(fluid.default_startup_program())
         print("\nRandomly initialize parameters...\n")
@@ -569,8 +596,6 @@ class Controller(object):
         backbone = self.train_backbone
         train_program = self.train_program
         saver_program = self.saver_program
-        fetches = self.fetches
-
         finish = []
         for inst in instances:
             if inst.is_target:
@@ -588,9 +613,10 @@ class Controller(object):
                         return False
             return True
 
-        # do training
-        fetch_names, fetch_list = zip(*fetches.items())
 
+        # do training
+        fetch_names = {}
+        fetch_list = []
         main_step = 0 # only count for main task
         global_step = 0 # count for all tasks
         epoch = 0
@@ -600,34 +626,32 @@ class Controller(object):
         feed_batch_process_fn = create_feed_batch_process_fn(self._net_inputs)
         distribute_feeder = data_feeder(self._joint_iterator_fn, feed_batch_process_fn)
         
-        # palm.distribute.reader(self._joint_iterator_fn, self._net_inputs, prefetch_steps=2)
-        
         while not train_finish():
-            feed, mask = next(distribute_feeder)
+            feed, mask, id = next(distribute_feeder)
+            for i in range(self.dev_count):
+                feed[i].update({'branch':np.array([id],dtype='int64')})
+            fetch_list.append(self._switched_loss)
             rt_outputs = self.exe.run(train_program, feed=feed, fetch_list=fetch_list)
-            while mask.pop() == False:
-                rt_outputs.pop()
+            rt_loss = rt_outputs.pop()
 
             rt_outputs = {k:v for k,v in zip(fetch_names, rt_outputs)}
-            rt_task_id = np.squeeze(rt_outputs['__task_id']).tolist()
-            rt_task_id = rt_task_id[0] if isinstance(rt_task_id, list) else rt_task_id
-            cur_task = instances[rt_task_id]
+            cur_task = instances[id]
 
-            backbone_rt_outputs = {k:v for k,v in rt_outputs.items() if '/' not in k}
-            backbone_buffer.append(backbone.postprocess(backbone_rt_outputs))
+            # backbone_rt_outputs = {k:v for k,v in rt_outputs.items() if '/' not in k}
+            # backbone_buffer.append(backbone.postprocess(backbone_rt_outputs))
             
-            task_rt_outputs = {k[len(cur_task.name+'/'):]: v for k,v in rt_outputs.items() if k.startswith(cur_task.name+'/')}
-            instances[rt_task_id].task_layer['train'].postprocess(task_rt_outputs)
+            # task_rt_outputs = {k[len(cur_task.name+'/'):]: v for k,v in rt_outputs.items() if k.startswith(cur_task.name+'/')}
+            # instances[rt_task_id].task_layer['train'].postprocess(task_rt_outputs)
 
             global_step += 1
             cur_task.cur_train_step += 1
 
             cur_task_global_step = cur_task.cur_train_step + cur_task.cur_train_epoch * cur_task.steps_pur_epoch
             if cur_task.is_target and cur_task.save_infermodel_every_n_steps > 0 and cur_task_global_step % cur_task.save_infermodel_every_n_steps == 0:
-                cur_task.save(suffix='.step'+str(cur_task_global_step))
+                cur_task.save(suffix='.step'+str(cur_task_global_step), prog=self._pred_prog)
 
             if global_step % main_conf.get('print_every_n_steps', 5) == 0:
-                loss = rt_outputs[cur_task.name+'/loss']
+                loss = rt_loss
                 loss = np.mean(np.squeeze(loss)).tolist()
 
                 time_end = time.time()
@@ -640,7 +664,7 @@ class Controller(object):
 
             if cur_task.train_finish and cur_task.cur_train_step + cur_task.cur_train_epoch * cur_task.steps_pur_epoch == cur_task.expected_train_steps:
                 print(cur_task.name+': train finished!')
-                cur_task.save()
+                cur_task.save(prog=self._pred_prog)
 
             if 'save_ckpt_every_n_steps' in main_conf and global_step % main_conf['save_ckpt_every_n_steps'] == 0:
                 save_path = os.path.join(main_conf['save_path'], 'ckpt', 
@@ -686,37 +710,26 @@ class Controller(object):
 
         print('predicting...')
         feed_batch_process_fn = create_feed_batch_process_fn(inst.pred_input)
-        distribute_feeder = data_feeder(inst.reader['pred'].iterator, feed_batch_process_fn, prefetch_steps=1)
+        distribute_feeder = data_feeder(inst.reader['pred'].iterator, feed_batch_process_fn, prefetch_steps=1, phase='pred')
 
         buf = []
-        for feed, mask in distribute_feeder:
-            print('before run')
-            rt_outputs = self.exe.run(pred_prog, feed, fetch_vars)
-            print('after run')
-            splited_rt_outputs = []
-            for item in rt_outputs:
-                splited_rt_outputs.append(np.split(item, len(mask)))
+        for feed, mask, id in distribute_feeder:
         
-            # assert len(rt_outputs) == len(mask), [len(rt_outputs), len(mask)]
-            print(mask)
+            rt_outputs = self.exe.run(pred_prog, feed, fetch_vars)
  
-            while mask.pop() == False:
-                print(mask)
-                for item in splited_rt_outputs:
+            num_fakes = decode_fake(len(rt_outputs[0]), mask, self.batch_size)
+            for _ in range(num_fakes):
+                for item in rt_outputs:
                     item.pop()
-            rt_outputs = []
-            print('cancat')
-            for item in splited_rt_outputs:
-                rt_outputs.append(np.concatenate(item))
     
             rt_outputs = {k:v for k,v in zip(fetch_names, rt_outputs)}
             inst.postprocess(rt_outputs, phase='pred')
-        print('leave feeder')
+
         if inst.task_layer['pred'].epoch_inputs_attrs:
             reader_outputs = inst.reader['pred'].get_epoch_outputs()
         else:
             reader_outputs = None
-        print('epoch postprocess')
+    
         inst.epoch_postprocess({'reader':reader_outputs}, phase='pred')
 
 
@@ -731,6 +744,3 @@ if __name__ == '__main__':
 
 
 __all__ = ["Controller"]
-
-            
-

paddlepalm/multihead_trainer.py

+
+from paddle import fluid
+from paddle.fluid import layers
+from paddlepalm.distribute import gpu_dev_count, cpu_dev_count
+from paddlepalm import Trainer
+from paddlepalm.utils import reader_helper
+import numpy as np
+import time
+
+dev_count = 1 if gpu_dev_count <= 1 else gpu_dev_count
+VERBOSE=False
+
+
+class MultiHeadTrainer(Trainer):
+    
+    def __init__(self, trainers, reuse_flags=None):
+        if reuse_flags is not None:
+            assert len(reuse_flags) == len(trainers)
+
+        self._trainers = trainers
+
+        name_maxlen = max([len(i.name) for i in self._trainers])
+        self._name_pads = {i.name: name_maxlen-len(i.name) for i in self._trainers}
+
+        self._train_init = False
+        self._predict_init = False
+        self._feeded_var_names = None
+        self._cur_train_step = 0
+        self._target_vars = None
+
+        self._inputname_to_varname = {}
+        self._pred_input_name_list = []
+        self._pred_input_varname_list = []
+        self._pred_fetch_name_list = []
+        self._pred_fetch_var_list = []
+
+        self._exe = None
+
+        self._save_protocol = {
+            'input_names': 'self._pred_input_name_list',
+            'input_varnames': 'self._pred_input_varname_list',
+            'fetch_list': 'self._pred_fetch_name_list'}
+
+        self._check_save = lambda: False
+        for t in self._trainers:
+            t._set_multitask()
+
+    def build_forward(self, backbone, heads):
+
+        if isinstance(heads, list):
+            head_dict = {k.name: v for k,v in zip(self._trainers, heads)}
+        elif isinstance(heads, dict):
+            head_dict = heads
+        else:
+            raise ValueError()
+
+        num_heads = len(self._trainers)
+        assert len(head_dict) == num_heads
+
+        for t in self._trainers:
+            assert t.name in head_dict, "expected: {}, exists: {}".format(t.name, head_dict.keys())
+        
+        train_prog = fluid.Program()
+        train_init_prog = fluid.Program()
+        self._train_prog = train_prog
+        self._train_init_prog = train_init_prog
+
+        def get_loss(i):
+            head = head_dict[self._trainers[i].name]
+            # loss_var = self._trainers[i].build_forward(backbone, head, train_prog, train_init_prog)
+            loss_var = self._trainers[i].build_forward(backbone, head)
+            return loss_var
+      
+        # task_fns = {}
+        # for i in range(num_heads):
+
+        #     def task_loss():
+        #         task_id = i
+        #         return lambda: get_loss(task_id)
+
+        #     task_fns[i] = task_loss()
+
+
+        # task_fns = {i: lambda: get_loss(i) for i in range(num_heads)}
+        task_fns = {i: lambda i=i: get_loss(i) for i in range(num_heads)}
+
+        with fluid.program_guard(train_prog, train_init_prog):
+            task_id_var = fluid.data(name="__task_id",shape=[1],dtype='int64')
+            # task_id_var = fluid.layers.fill_constant(shape=[1],dtype='int64', value=1)
+            # print(task_id_var.name)
+
+            loss_var = layers.switch_case(
+                branch_index=task_id_var,
+                branch_fns=task_fns
+            )
+        self._task_id_var = task_id_var
+        self._loss_var = loss_var
+        self._fetch_list = [loss_var.name]
+        # for b in train_prog.blocks:
+        #     for var in b.vars:
+        #         pass
+                # if 'task_id' in var:
+                #     print(var)
+                #     exit()
+                # print(var)
+        return loss_var
+
+    def fit_readers(self, reader_dict):
+        raise NotImplementedError()
+
+    def fit_readers_with_mixratio(self, readers, sampling_reference, num_epochs, phase='train'):
+
+        if isinstance(readers, list):
+            reader_dict = {k.name: v for k,v in zip(self._trainers, readers)}
+        elif isinstance(readers, dict):
+            reader_dict = readers
+        else:
+            raise ValueError()
+        
+        num_heads = len(self._trainers)
+        assert len(reader_dict) == num_heads
+
+        trainer_dict = {t.name: t for t in self._trainers}
+        assert sampling_reference in trainer_dict
+
+        trainer_dict[sampling_reference].fit_reader(reader_dict[sampling_reference], task_id=self._task_id_var)
+        base_steps_pur_epoch = trainer_dict[sampling_reference]._steps_pur_epoch
+
+        self._finish_steps = {}
+        self._finish = {}
+        input_names = []
+        name_to_pos = []
+        joint_shape_and_dtypes = []
+        iterators = []
+        prefixes = []
+        mrs = []
+        net_inputs = []
+        global_steps = 0
+        for t in self._trainers:
+            assert t.name in reader_dict
+            assert reader_dict[t.name].num_epochs is None, "{}: num_epochs is not None. \
+                To run with multi-head mode, num_epochs of each Trainer should be set as None.".format(t.name)
+            # print(num_epochs, t.mix_ratio, base_steps_pur_epoch)
+            max_train_steps = int(num_epochs * t.mix_ratio * base_steps_pur_epoch)
+            if not t._as_auxilary:
+                print('{}: expected train steps {}.'.format(t.name, max_train_steps))
+                self._finish_steps[t.name] = max_train_steps
+                self._finish[t.name] = False
+            else:
+                self._finish_steps[t.name] = 9999999999
+                self._finish[t.name] = True
+
+            global_steps += max_train_steps
+            if t.name != sampling_reference:
+                t.fit_reader(reader_dict[t.name], task_id=self._task_id_var)
+            net_inputs.append(t._net_inputs)
+            prefixes.append(t.name)
+            mrs.append(t.mix_ratio)
+            iterators.append(t._raw_iterator_fn())
+            input_names.append(t._input_names)
+            name_to_pos.append(t._name_to_position)
+            joint_shape_and_dtypes.append(t._shape_and_dtypes)
+
+        print('Estimated overall train steps {}.'.format(global_steps))
+        self._overall_train_steps = global_steps
+
+        iterator_fn = reader_helper.create_multihead_iterator_fn(iterators, prefixes, joint_shape_and_dtypes, \
+            mrs, input_names, name_to_pos, dev_count=dev_count)
+        feed_batch_process_fn = reader_helper.create_feed_batch_process_fn(net_inputs)
+
+        if gpu_dev_count > 1:
+            distribute_feeder_fn = data_feeder(iterator_fn, feed_batch_process_fn)
+        else:
+            distribute_feeder_fn = iterator_fn
+
+        if phase == 'train':
+            self._train_reader = distribute_feeder_fn()
+            self._feed_batch_process_fn = feed_batch_process_fn
+        elif phase == 'predict':
+            self._predict_reader = distribute_feeder_fn()
+            self._pred_feed_batch_process_fn = feed_batch_process_fn
+
+    def check_finish(self, task_name, silent=False):
+        trainers = {t.name:t for t in self._trainers}
+        if trainers[task_name]._cur_train_step == self._finish_steps[task_name]:
+            if not silent:
+                print(task_name+' train finish!')
+            self._finish[task_name]=True
+        flags = list(set(self._finish.values()))
+        return len(flags) == 1 and flags[0] == True
+        
+    def train(self, save_path=None, save_steps=None, save_type='ckpt', print_steps=5):
+        iterator = self._train_reader
+        self._distribute_train_prog = fluid.CompiledProgram(self._train_prog).with_data_parallel(loss_name=self._loss_var.name)
+
+        save_type = save_type.split(',')
+        if 'predict' in save_type:
+            assert self._pred_head is not None, "Predict head not found! You should build_predict_head first if you want to save predict model."
+            assert save_path is not None and save_steps is not None, 'save_path and save_steps is required to save model.'
+            save_predict = True
+            if not os.path.exists(save_path):
+                os.makedirs(save_path)
+        else:
+            save_predict = False
+
+        if 'ckpt' in save_type:
+            if save_path is not None and save_steps is not None:
+                save_ckpt = True
+                if not os.path.exists(save_path):
+                    os.makedirs(save_path)
+            else:
+                "WARNING: save_path or save_steps is not set, model will not be saved during training."
+                save_ckpt = False
+        else:
+            save_ckpt = False
+
+        time_begin = time.time()
+        for feed in iterator:
+            # batch, task_id = feed
+            rt_outputs, task_id = self.train_one_step(feed)
+
+            task_rt_outputs = {k[len(self._trainers[task_id].name+'.'):]: v for k,v in rt_outputs.items() if k.startswith(self._trainers[task_id].name+'.')}
+            self._trainers[task_id]._task_head.batch_postprocess(task_rt_outputs)
+
+            if print_steps > 0 and self._cur_train_step % print_steps == 0:
+                loss = rt_outputs[self._trainers[task_id].name+'.loss']
+                loss = np.mean(np.squeeze(loss)).tolist()
+
+                time_end = time.time()
+                time_cost = time_end - time_begin
+
+                print("global step: {}, {}: step {}/{} (epoch {}), loss: {:.3f}, speed: {:.2f} steps/s".format(
+                       self._cur_train_step, ' '*self._name_pads[self._trainers[task_id].name]+self._trainers[task_id].name, \
+                       (self._trainers[task_id]._cur_train_step-1) % self._trainers[task_id]._steps_pur_epoch + 1, \
+                       self._trainers[task_id]._steps_pur_epoch, self._trainers[task_id]._cur_train_epoch, \
+                       loss, print_steps / time_cost))
+                time_begin = time.time()
+
+            self._check_save()
+            finish = self.check_finish(self._trainers[task_id].name)
+            if finish:
+                break
+
+            # if cur_task.train_finish and cur_task.cur_train_step + cur_task.cur_train_epoch * cur_task.steps_pur_epoch == cur_task.expected_train_steps:
+            #     print(cur_task.name+': train finished!')
+            #     cur_task.save()
+
+            # if (save_predict or save_ckpt) and self._cur_train_step % save_steps == 0:
+            #     if save_predict:
+            #         self.save(save_path, suffix='pred.step'+str(self._cur_train_step))
+            #     if save_ckpt:
+            #         fluid.io.save_persistables(self._exe, os.path.join(save_path, 'ckpt.step'+str(self._cur_train_step)), self._train_prog)
+            #         print('checkpoint has been saved at '+os.path.join(save_path, 'ckpt.step'+str(self._cur_train_step)))
+
+
+    def train_one_step(self, batch):
+
+        if dev_count > 1:
+            assert isinstance(batch, list)
+            task_id = batch[0]['__task_id'][0]
+        else:
+            assert isinstance(batch, dict)
+            task_id = batch['__task_id'][0]
+            
+        rt_outputs = self._trainers[task_id].train_one_step(batch, self._exe, self._distribute_train_prog, self._fetch_list)
+
+        self._cur_train_step += 1
+        return rt_outputs, task_id
+        
+        # if dev_count > 1:
+        #     # feed, mask, task_id = batch
+        #     for f in feed:
+        #         f['branch'] = np.array([task_id], dtype='int64')
+        #     rt_outputs = self.exe.run(self._distribute_train_prog, feed=feed, fetch_list=self._trainers[task_id]._fetch_list)
+        #     num_fakes = decode_fake(len(rt_outputs[0]), mask, self._trainers[task_id]._batch_size)
+        #     for _ in range(num_fakes):
+        #         for item in rt_outputs:
+        #             item.pop()
+        # else:
+        #     feed, task_id = batch
+        #     feed['branch'] = np.array([task_id], dtype='int64')
+        #     rt_outputs = self._exe.run(self._distribute_train_prog, feed=feed, fetch_list=self._trainers[task_id]._fetch_list)
+
+    def predict_one_batch(self, batch):
+        raise NotImplementedError()
+
+    def predict(self, output_dir=None, print_steps=1000):
+        raise NotImplementedError()
+
+    @property
+    def overall_train_steps(self):
+        return self._overall_train_steps

paddlepalm/optimizer/base_optimizer.py

@@ -8,8 +8,9 @@ class BaseOptimizer():
     def build(self, grad_clip=None):
         pass
 
-    def _set_prog(self, prog):
+    def _set_prog(self, prog, init_prog):
         self._prog = prog
+        self._init_prog = prog
         if self._lr_schedualer is not None:
             self._lr_schedualer._set_prog(prog)
 

paddlepalm/reader/__init__.py

 
 from cls import ClassifyReader
-
+from match import MatchReader
+from ner import SequenceLabelReader
+from mrc import MrcReader
+from mlm import MaskLMReader

paddlepalm/reader/base_reader.py

@@ -14,13 +14,15 @@
 # limitations under the License.
 """v1.1"""
 from copy import copy
-class BaseReader(object):
+class Reader(object):
     """interface of data manager."""
 
     def __init__(self, phase='train'):
         # assert isinstance(config, dict)
         # self._config = config
         self._phase = phase
+        self._batch_size = None
+        self._num_epochs = 1
         self._register = set()
         self._registered_backbone = None
 
@@ -40,7 +42,6 @@ class BaseReader(object):
         self._register.add(attr_name)
             
     def register_with(self, backbone):
-        print(backbone)
         for attr in backbone.inputs_attr:
             self.require_attr(attr)
         self._registered_backbone = backbone
@@ -117,4 +118,8 @@ class BaseReader(object):
         """数据集中的样本数量，即每个epoch中iterator所生成的样本数。注意，使用滑动窗口等可能导致数据集样本数发生变化的策略时，该接口应返回runtime阶段的实际样本数。"""
         raise NotImplementedError()
 
+    @property
+    def num_epochs(self):
+        """"""
+        raise NotImplementedError()
 

paddlepalm/reader/cls.py

@@ -13,11 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from paddlepalm.reader.base_reader import BaseReader
+from paddlepalm.reader.base_reader import Reader
 from paddlepalm.reader.utils.reader4ernie import ClassifyReader as CLSReader
 
 
-class ClassifyReader(BaseReader):
+class ClassifyReader(Reader):
     
     def __init__(self, vocab_path, max_len, tokenizer='wordpiece', \
              lang='en', seed=None, do_lower_case=False, phase='train'):
@@ -29,10 +29,10 @@ class ClassifyReader(BaseReader):
 
         """
 
-        BaseReader.__init__(self, phase)
+        Reader.__init__(self, phase)
 
         assert lang.lower() in ['en', 'cn', 'english', 'chinese'], "supported language: en (English), cn (Chinese)."
-        assert phase in ['train', 'pred'], "supported phase: train, pred."
+        assert phase in ['train', 'predict'], "supported phase: train, predict."
 
         for_cn = lang.lower() == 'cn' or lang.lower() == 'chinese'
 
@@ -66,10 +66,13 @@ class ClassifyReader(BaseReader):
         return self._get_registed_attrs(attrs)
 
 
-    def _load_data(self, input_file, batch_size, num_epochs=None, \
+    def load_data(self, input_file, batch_size, num_epochs=None, \
                   file_format='csv', shuffle_train=True):
-        self._data_generator = self._reader.data_generator(input_file, batch_size, \
-            num_epochs, shuffle=shuffle_train if self._phase == 'train' else False, \
+        self._batch_size = batch_size
+        self._num_epochs = num_epochs
+        self._data_generator = self._reader.data_generator( \
+            input_file, batch_size, num_epochs if self._phase == 'train' else 1, \
+            shuffle=shuffle_train if self._phase == 'train' else False, \
             phase=self._phase)
 
     def _iterator(self): 
@@ -92,4 +95,8 @@ class ClassifyReader(BaseReader):
     def num_examples(self):
         return self._reader.get_num_examples(phase=self._phase)
 
+    @property
+    def num_epochs(self):
+        return self._num_epochs
+
 

paddlepalm/reader/match.py

@@ -13,87 +13,104 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from paddlepalm.interface import reader
-from paddlepalm.reader.utils.reader4ernie import ClassifyReader
+from paddlepalm.reader.base_reader import Reader
+from paddlepalm.reader.utils.reader4ernie import ClassifyReader as CLSReader
 
-class Reader(reader):
 
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
+class MatchReader(Reader):
+    
+    def __init__(self, vocab_path, max_len, tokenizer='wordpiece', lang='en', seed=None, \
+        do_lower_case=False, learning_strategy='pointwise', phase='train', dev_count=1, print_prefix=''):  # 需要什么加什么
         """  
         Args:
             phase: train, eval, pred
+            lang: en, ch, ...
+            learning_strategy: pointwise, pairwise
         """
 
-        self._is_training = phase == 'train'
+        Reader.__init__(self, phase)
 
-        reader = ClassifyReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', True),
-            for_cn=config.get('for_cn', False),
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
+        assert lang.lower() in ['en', 'cn', 'english', 'chinese'], "supported language: en (English), cn (Chinese)."
+        assert phase in ['train', 'predict'], "supported phase: train, predict."
 
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
+        for_cn = lang.lower() == 'cn' or lang.lower() == 'chinese'
+
+        self._register.add('token_ids')
         if phase == 'train':
-            self._input_file = config['train_file']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        elif phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
+            if learning_strategy == 'pointwise':
+                self._register.add('label_ids')
+            if learning_strategy == 'pairwise':
+                self._register.add('token_ids_neg')
+                self._register.add('position_ids_neg')
+                self._register.add('segment_ids_neg')
+                self._register.add('input_mask_neg')
+                self._register.add('task_ids_neg')
+
+        self._is_training = phase == 'train'
+        self._learning_strategy = learning_strategy
+
+
+        match_reader = CLSReader(vocab_path,
+                                max_seq_len=max_len,
+                                do_lower_case=do_lower_case,
+                                for_cn=for_cn,
+                                random_seed=seed,
+                                learning_strategy = learning_strategy)
             
+        self._reader = match_reader
+        self._dev_count = dev_count
         self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 1)
 
 
     @property
     def outputs_attr(self):
-        if self._is_training:
-            return {"token_ids": [[-1, -1], 'int64'],
+        attrs = {"token_ids": [[-1, -1], 'int64'],
                 "position_ids": [[-1, -1], 'int64'],
                 "segment_ids": [[-1, -1], 'int64'],
                 "input_mask": [[-1, -1, 1], 'float32'],
-                    "label_ids": [[-1], 'int64'],
-                    "task_ids": [[-1, -1], 'int64']
-                    }
-        else:
-            return {"token_ids": [[-1, -1], 'int64'],
-                    "position_ids": [[-1, -1], 'int64'],
-                    "segment_ids": [[-1, -1], 'int64'],
                 "task_ids": [[-1, -1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32']
+                "label_ids": [[-1], 'int64'],
+                "token_ids_neg": [[-1, -1], 'int64'],
+                "position_ids_neg": [[-1, -1], 'int64'],
+                "segment_ids_neg": [[-1, -1], 'int64'],
+                "input_mask_neg": [[-1, -1, 1], 'float32'],
+                "task_ids_neg": [[-1, -1], 'int64']
                 }
+        return self._get_registed_attrs(attrs)
+
 
+    def load_data(self, input_file, batch_size, num_epochs=None, \
+                  file_format='tsv', shuffle_train=True):
+        self._batch_size = batch_size
+        self._num_epochs = num_epochs
+        self._data_generator = self._reader.data_generator( \
+            input_file, batch_size, num_epochs if self._phase == 'train' else 1, \
+            shuffle=shuffle_train if self._phase == 'train' else False, \
+            phase=self._phase)
 
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
+    def _iterator(self): 
 
-    def iterator(self): 
         
-        def list_to_dict(x):
-            names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 
-                'label_ids', 'unique_ids']
-            outputs = {n: i for n,i in zip(names, x)}
-            del outputs['unique_ids']
-            if not self._is_training:
-                del outputs['label_ids']
-            return outputs
+        names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 'label_ids', \
+            'token_ids_neg', 'segment_ids_neg', 'position_ids_neg', 'task_ids_neg', 'input_mask_neg']
+        
+        if self._learning_strategy == 'pairwise':
+            names.remove('label_ids')
+
 
         for batch in self._data_generator():
-            yield list_to_dict(batch)
+            outputs = {n: i for n,i in zip(names, batch)}
+            ret = {}
+            # TODO: move runtime shape check here
+            for attr in self.outputs_attr.keys():
+                ret[attr] = outputs[attr]
+            yield ret
 
     @property
     def num_examples(self):
         return self._reader.get_num_examples(phase=self._phase)
 
+    @property
+    def num_epochs(self):
+        return self._num_epochs
+

paddlepalm/reader/mlm.py

@@ -13,79 +13,79 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from paddlepalm.interface import reader
-from paddlepalm.reader.utils.reader4ernie import MaskLMReader
+from paddlepalm.reader.base_reader import Reader
+from paddlepalm.reader.utils.reader4ernie import MaskLMReader as MLMReader
 import numpy as np
 
-class Reader(reader):
+class MaskLMReader(Reader):
     
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
+    def __init__(self, vocab_path, max_len, tokenizer='wordpiece', \
+             lang='en', seed=None, do_lower_case=False, phase='train', dev_count=1, print_prefix=''):
         """
         Args:
             phase: train, eval, pred
         """
 
-        self._is_training = phase == 'train'
 
-        reader = MaskLMReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', False),
-            for_cn=config.get('for_cn', False),
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
+        Reader.__init__(self, phase)
+
+        assert lang.lower() in ['en', 'cn', 'english', 'chinese'], "supported language: en (English), cn (Chinese)."
+        assert phase in ['train', 'predict'], "supported phase: train, predict."
 
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
+        for_cn = lang.lower() == 'cn' or lang.lower() == 'chinese'
+
+        self._register.add('token_ids')
+        self._register.add('mask_pos')
         if phase == 'train':
-            self._input_file = config['train_file']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        elif phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
+            self._register.add('mask_label')
+        self._is_training = phase == 'train'
+
+        mlm_reader = MLMReader(vocab_path,
+                                max_seq_len=max_len,
+                                do_lower_case=do_lower_case,
+                                for_cn=for_cn,
+                                random_seed=seed)
+        self._reader = mlm_reader
 
         self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 1)
+        self._dev_count = dev_count
 
 
     @property
     def outputs_attr(self):
-        return {"token_ids": [[-1, -1], 'int64'],
+        attrs = {"token_ids": [[-1, -1], 'int64'],
                 "position_ids": [[-1, -1], 'int64'],
                 "segment_ids": [[-1, -1], 'int64'],
                 "input_mask": [[-1, -1, 1], 'float32'],
                 "task_ids": [[-1, -1], 'int64'],
                 "mask_label": [[-1], 'int64'],
-                "mask_pos": [[-1], 'int64'],
+                "mask_pos": [[-1], 'int64']
                 }
 
+        return self._get_registed_attrs(attrs)
+
 
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
+    def load_data(self, input_file, batch_size, num_epochs=None, \
+                  file_format='csv', shuffle_train=True):
+        self._batch_size = batch_size
+        self._num_epochs = num_epochs
+        self._data_generator = self._reader.data_generator( \
+            input_file, batch_size, num_epochs if self._phase == 'train' else 1, \
+            shuffle=shuffle_train if self._phase == 'train' else False, \
+            phase=self._phase)
 
-    def iterator(self): 
+    def _iterator(self): 
 
-        def list_to_dict(x):
         names = ['token_ids', 'position_ids', 'segment_ids', 'input_mask', 
             'task_ids', 'mask_label', 'mask_pos']
-            outputs = {n: i for n,i in zip(names, x)}
-            # outputs['batchsize_x_seqlen'] = [self._batch_size * len(outputs['token_ids'][0]) - 1]
-            return outputs
-
         for batch in self._data_generator():
-            # print(np.shape(list_to_dict(batch)['token_ids']))
-            # print(list_to_dict(batch)['mask_label'].tolist())
-            yield list_to_dict(batch)
+            outputs = {n: i for n,i in zip(names, batch)}
+            ret = {}
+            # TODO: move runtime shape check here
+            for attr in self.outputs_attr.keys():
+                ret[attr] = outputs[attr]
+
+            yield ret
 
     def get_epoch_outputs(self):
         return {'examples': self._reader.get_examples(self._phase),
@@ -95,3 +95,7 @@ class Reader(reader):
     def num_examples(self):
         return self._reader.get_num_examples(phase=self._phase)
 
+    @property
+    def num_epochs(self):
+        return self._num_epochs
+

paddlepalm/reader/mrc.py

@@ -13,77 +13,66 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from paddlepalm.interface import reader
+from paddlepalm.reader.base_reader import Reader
 from paddlepalm.reader.utils.reader4ernie import MRCReader
+import numpy as np
 
-class Reader(reader):
+class MrcReader(Reader):
 
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
+    def __init__(self, vocab_path, max_len, max_query_len, doc_stride, tokenizer='FullTokenizer', lang='en', seed=None, do_lower_case=False, \
+        remove_noanswer=True, phase='train', dev_count=1, print_prefix=''):
         """
         Args:
             phase: train, eval, pred
+            lang: en, ch, ...
         """
 
-        self._is_training = phase == 'train'
+        Reader.__init__(self, phase)
 
-        reader = MRCReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', False),
-            tokenizer='FullTokenizer',
-            for_cn=config.get('for_cn', False),
-            doc_stride=config['doc_stride'],
-            remove_noanswer=config.get('remove_noanswer', True),
-            max_query_length=config['max_query_len'],
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
 
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
+        assert lang.lower() in ['en', 'cn', 'english', 'chinese'], "supported language: en (English), cn (Chinese)."
+        assert phase in ['train', 'predict'], "supported phase: train, predict."
+
+        for_cn = lang.lower() == 'cn' or lang.lower() == 'chinese'
+
+
+        self._register.add('token_ids')
         if phase == 'train':
-            self._input_file = config['train_file']
-            # self._num_epochs = config['num_epochs']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        if phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
+            self._register.add("start_positions")
+            self._register.add("end_positions")
+        else:
+            self._register.add("unique_ids")
             
-        self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 1)
 
-        # TODO: without slide window version
-        self._with_slide_window = config.get('with_slide_window', False)
+        self._is_training = phase == 'train'
+
+        mrc_reader = MRCReader(vocab_path,
+                                max_seq_len=max_len,
+                                do_lower_case=do_lower_case,
+                                tokenizer=tokenizer,
+                                doc_stride=doc_stride,
+                                remove_noanswer=remove_noanswer,
+                                max_query_length=max_query_len,
+                                for_cn=for_cn,
+                                random_seed=seed)
+        self._reader = mrc_reader
+
+        self._phase = phase
+        self._dev_count = dev_count
  
 
     @property
     def outputs_attr(self):
-        if self._is_training:
-            return {"token_ids": [[-1, -1], 'int64'],
+        attrs = {"token_ids": [[-1, -1], 'int64'],
                 "position_ids": [[-1, -1], 'int64'],
                 "segment_ids": [[-1, -1], 'int64'],
                 "input_mask": [[-1, -1, 1], 'float32'],
                 "start_positions": [[-1], 'int64'],
                 "end_positions": [[-1], 'int64'],
-                    "task_ids": [[-1, -1], 'int64']
-                    }
-        else:
-            return {"token_ids": [[-1, -1], 'int64'],
-                    "position_ids": [[-1, -1], 'int64'],
-                    "segment_ids": [[-1, -1], 'int64'],
                 "task_ids": [[-1, -1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32'],
                 "unique_ids": [[-1], 'int64']
                 }
+        return self._get_registed_attrs(attrs)
 
     @property
     def epoch_outputs_attr(self):
@@ -91,26 +80,34 @@ class Reader(reader):
             return {"examples": None,
                     "features": None}
 
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
-
-    def iterator(self): 
+    def load_data(self, input_file, batch_size, num_epochs=None, file_format='csv', shuffle_train=True):
+        self._batch_size = batch_size
+        self._num_epochs = num_epochs
+        self._data_generator = self._reader.data_generator( \
+            input_file, batch_size, num_epochs if self._phase == 'train' else 1, \
+            shuffle=shuffle_train if self._phase == 'train' else False, \
+            phase=self._phase)
+    def _iterator(self): 
 
-        def list_to_dict(x):
         names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 
             'start_positions', 'end_positions', 'unique_ids']
-            outputs = {n: i for n,i in zip(names, x)}
+        
         if self._is_training:
-                del outputs['unique_ids']
-            else:
-                del outputs['start_positions']
-                del outputs['end_positions']
-            return outputs
+            names.remove('unique_ids')
         
         for batch in self._data_generator():
-            yield list_to_dict(batch)
+            outputs = {n: i for n,i in zip(names, batch)}
+            ret = {}
+            # TODO: move runtime shape check here
+            for attr in self.outputs_attr.keys():
+                ret[attr] = outputs[attr]
+            if not self._is_training:
+                assert 'unique_ids' in ret, ret
+            yield ret
+    
 
     def get_epoch_outputs(self):
+
         return {'examples': self._reader.get_examples(self._phase),
                 'features': self._reader.get_features(self._phase)}
 
@@ -118,3 +115,7 @@ class Reader(reader):
     def num_examples(self):
         return self._reader.get_num_examples(phase=self._phase)
 
+    @property
+    def num_epochs(self):
+        return self._num_epochs
+

reader/mlm.py → paddlepalm/reader/ner.py

@@ -13,79 +13,76 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from paddlepalm.interface import reader
-from paddlepalm.reader.utils.reader4ernie import MaskLMReader
-import numpy as np
+from paddlepalm.reader.base_reader import Reader
+from paddlepalm.reader.utils.reader4ernie import SequenceLabelReader as SLReader
 
-class Reader(reader):
+class SequenceLabelReader(Reader):
     
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
+    def __init__(self, vocab_path, max_len, label_map_config, tokenizer='wordpiece', \
+             lang='en', seed=None, do_lower_case=False, phase='train', dev_count=1, print_prefix=''):
         """  
         Args:
             phase: train, eval, pred
+            lang: en, ch, ...
         """
         
-        self._is_training = phase == 'train'
+        Reader.__init__(self, phase)
 
-        reader = MaskLMReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', False),
-            for_cn=config.get('for_cn', False),
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
+        assert lang.lower() in ['en', 'cn', 'english', 'chinese'], "supported language: en (English), cn (Chinese)."
+        assert phase in ['train', 'predict'], "supported phase: train, predict."
 
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
+        for_cn = lang.lower() == 'cn' or lang.lower() == 'chinese'
+
+        self._register.add('token_ids')
+        self._register.add('seq_lens')
         if phase == 'train':
-            self._input_file = config['train_file']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        elif phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
+            self._register.add('label_ids')
+
+        self._is_training = phase == 'train'
 
+        ner_reader = SLReader(vocab_path,
+                                max_seq_len=max_len,
+                                do_lower_case=do_lower_case,
+                                for_cn=for_cn,
+                                random_seed=seed,
+                                label_map_config=label_map_config)
+        self._reader = ner_reader
         self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 1)
+        self._dev_count = dev_count
 
  
     @property
     def outputs_attr(self):
-        return {"token_ids": [[-1, -1, 1], 'int64'],
-                "position_ids": [[-1, -1, 1], 'int64'],
-                "segment_ids": [[-1, -1, 1], 'int64'],
+        attrs = {"token_ids": [[-1, -1], 'int64'],
+                "position_ids": [[-1, -1], 'int64'],
+                "segment_ids": [[-1, -1], 'int64'],
+                "task_ids": [[-1, -1], 'int64'],
                 "input_mask": [[-1, -1, 1], 'float32'],
-                "task_ids": [[-1, -1, 1], 'int64'],
-                "mask_label": [[-1, 1], 'int64'],
-                "mask_pos": [[-1, 1], 'int64'],
-                }
+                "seq_lens": [[-1], 'int64'],
+                "label_ids": [[-1, -1], 'int64']}
+        return self._get_registed_attrs(attrs)
 
 
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
+    def load_data(self, input_file, batch_size, num_epochs=None, \
+                  file_format='tsv', shuffle_train=True):
+        self._batch_size = batch_size
+        self._num_epochs = num_epochs
+        self._data_generator = self._reader.data_generator( \
+            input_file, batch_size, num_epochs if self._phase == 'train' else 1, \
+            shuffle=shuffle_train if self._phase == 'train' else False, \
+            phase=self._phase)
 
-    def iterator(self): 
-
-        def list_to_dict(x):
-            names = ['token_ids', 'position_ids', 'segment_ids', 'input_mask', 
-                'task_ids', 'mask_label', 'mask_pos']
-            outputs = {n: i for n,i in zip(names, x)}
-            # outputs['batchsize_x_seqlen'] = [self._batch_size * len(outputs['token_ids'][0]) - 1]
-            return outputs
+    def _iterator(self): 
 
+        names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 
+            'label_ids', 'seq_lens', 'label_ids']
         for batch in self._data_generator():
-            # print(np.shape(list_to_dict(batch)['token_ids']))
-            # print(list_to_dict(batch)['mask_label'].tolist())
-            yield list_to_dict(batch)
+            outputs = {n: i for n,i in zip(names, batch)}
+            ret = {}
+            # TODO: move runtime shape check here
+            for attr in self.outputs_attr.keys():
+                ret[attr] = outputs[attr]
+            yield ret
 
     def get_epoch_outputs(self):
         return {'examples': self._reader.get_examples(self._phase),
@@ -95,3 +92,6 @@ class Reader(reader):
     def num_examples(self):
         return self._reader.get_num_examples(phase=self._phase)
 
+    @property
+    def num_epochs(self):
+        return self._num_epochs

paddlepalm/reader/utils/mlm_batching.py

@@ -19,12 +19,26 @@ from __future__ import print_function
 import numpy as np
 
 
-def mask(batch_tokens, total_token_num, vocab_size, CLS=1, SEP=2, MASK=3):
+def mask(batch_tokens, total_token_num, vocab_size, CLS=1, SEP=2, MASK=3, dev_count=1):
     """
     Add mask for batch_tokens, return out, mask_label, mask_pos;
     Note: mask_pos responding the batch_tokens after padded;
     """
     max_len = max([len(sent) for sent in batch_tokens])
+
+    multidev_batch_tokens = []
+    multidev_mask_label = []
+    multidev_mask_pos = []
+
+    big_batch_tokens = batch_tokens
+    stride = len(batch_tokens) // dev_count
+    if stride == 0:
+        return None, None, None
+    p = stride
+
+    for i in range(dev_count):
+        batch_tokens = big_batch_tokens[p-stride:p]
+        p += stride
         mask_label = []
         mask_pos = []
         prob_mask = np.random.rand(total_token_num)
@@ -69,7 +83,12 @@ def mask(batch_tokens, total_token_num, vocab_size, CLS=1, SEP=2, MASK=3):
                     mask_pos.append(sent_index * max_len + token_index)
         mask_label = np.array(mask_label).astype("int64").reshape([-1])
         mask_pos = np.array(mask_pos).astype("int64").reshape([-1])
-    return batch_tokens, mask_label, mask_pos
+
+        multidev_batch_tokens.extend(batch_tokens)
+        multidev_mask_label.append(mask_label)
+        multidev_mask_pos.append(mask_pos)
+    
+    return multidev_batch_tokens, multidev_mask_label, multidev_mask_pos
 
 
 def prepare_batch_data(insts,
@@ -83,7 +102,8 @@ def prepare_batch_data(insts,
                        task_id=0,
                        return_input_mask=True,
                        return_max_len=True,
-                       return_num_token=False):
+                       return_num_token=False, 
+                       dev_count=1):
     """
     1. generate Tensor of data
     2. generate Tensor of position
@@ -101,7 +121,8 @@ def prepare_batch_data(insts,
         vocab_size=voc_size,
         CLS=cls_id,
         SEP=sep_id,
-        MASK=mask_id)
+        MASK=mask_id,
+        dev_count=dev_count)
     # Second step: padding
     src_id, self_input_mask = pad_batch_data(
         out, 
@@ -125,7 +146,7 @@ def prepare_batch_data(insts,
     return_list = [
         src_id, pos_id, sent_id, self_input_mask, task_ids, mask_label, mask_pos
     ]
-    return return_list if len(return_list) > 1 else return_list[0]
+    return return_list
 
 
 def pad_batch_data(insts,

paddlepalm/reader/utils/reader4ernie.py

@@ -29,11 +29,15 @@ import six
 from io import open
 from collections import namedtuple
 
+# from . import gpu_dev_count
+gpu_dev_count=1
+import paddlepalm as palm
 import paddlepalm.tokenizer.ernie_tokenizer as tokenization
 from paddlepalm.reader.utils.batching4ernie import pad_batch_data
 from paddlepalm.reader.utils.mlm_batching import prepare_batch_data
 
 
+
 log = logging.getLogger(__name__)
 
 if six.PY3:
@@ -49,20 +53,23 @@ def csv_reader(fd, delimiter='\t'):
     return gen()
 
 
-class BaseReader(object):
+class Reader(object):
     def __init__(self,
                  vocab_path,
                  label_map_config=None,
                  max_seq_len=512,
-                 do_lower_case=False,
+                 do_lower_case=True,
                  in_tokens=False,
                  is_inference=False,
+                 learning_strategy='pointwise',
                  random_seed=None,
                  tokenizer="FullTokenizer",
+                 phase='train',
                  is_classify=True,
                  is_regression=False,
-                 for_cn=False,
+                 for_cn=True,
                  task_id=0):
+        assert phase in ['train', 'predict'], "supported phase: train, predict."
         self.max_seq_len = max_seq_len
         self.tokenizer = tokenization.FullTokenizer(
             vocab_file=vocab_path, do_lower_case=do_lower_case)
@@ -72,7 +79,9 @@ class BaseReader(object):
         self.sep_id = self.vocab["[SEP]"]
         self.mask_id = self.vocab["[MASK]"]
         self.in_tokens = in_tokens
+        self.phase = phase
         self.is_inference = is_inference
+        self.learning_strategy = learning_strategy
         self.for_cn = for_cn
         self.task_id = task_id
 
@@ -83,7 +92,6 @@ class BaseReader(object):
         self.current_example = 0
         self.current_epoch = 0
         self.num_examples = 0
-
         self.examples = {}
 
         if label_map_config:
@@ -125,33 +133,41 @@ class BaseReader(object):
             else:
                 tokens_b.pop()
     
+
     def _convert_example_to_record(self, example, max_seq_length, tokenizer):
         """Converts a single `Example` into a single `Record`."""
 
         text_a = tokenization.convert_to_unicode(example.text_a)
         tokens_a = tokenizer.tokenize(text_a)
         tokens_b = None
-
         has_text_b = False
+        has_text_b_neg = False
         if isinstance(example, dict):
             has_text_b = "text_b" in example.keys()
+            has_text_b_neg = "text_b_neg" in example.keys()
         else:
             has_text_b = "text_b" in example._fields
+            has_text_b_neg = "text_b_neg" in example._fields
 
         if has_text_b:
             text_b = tokenization.convert_to_unicode(example.text_b)
             tokens_b = tokenizer.tokenize(text_b)
-
-        if tokens_b:
             # Modifies `tokens_a` and `tokens_b` in place so that the total
             # length is less than the specified length.
             # Account for [CLS], [SEP], [SEP] with "- 3"
             self._truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
+           
+            if has_text_b_neg and self.phase == 'train':
+                tokens_a_neg = tokenizer.tokenize(text_a)
+                text_b_neg = tokenization.convert_to_unicode(example.text_b_neg)
+                tokens_b_neg = tokenizer.tokenize(text_b_neg)
+                self._truncate_seq_pair(tokens_a_neg, tokens_b_neg, max_seq_length - 3)
         else:
             # Account for [CLS] and [SEP] with "- 2"
             if len(tokens_a) > max_seq_length - 2:
                 tokens_a = tokens_a[0:(max_seq_length - 2)]
         
+
         # The convention in BERT/ERNIE is:
         # (a) For sequence pairs:
         #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
@@ -173,6 +189,7 @@ class BaseReader(object):
         tokens = []
         text_type_ids = []
         tokens.append("[CLS]")
+        
         text_type_ids.append(0)
         for token in tokens_a:
             tokens.append(token)
@@ -190,6 +207,29 @@ class BaseReader(object):
         token_ids = tokenizer.convert_tokens_to_ids(tokens)
         position_ids = list(range(len(token_ids)))
 
+
+        if has_text_b_neg and self.phase == 'train':
+            tokens_neg = []
+            text_type_ids_neg = []
+            tokens_neg.append("[CLS]")
+            text_type_ids_neg.append(0)
+            for token in tokens_a_neg:
+                tokens_neg.append(token)
+                text_type_ids_neg.append(0)
+            tokens_neg.append("[SEP]")
+            text_type_ids_neg.append(0)
+
+            if tokens_b_neg:
+                for token in tokens_b_neg:
+                    tokens_neg.append(token)
+                    text_type_ids_neg.append(1)
+                tokens_neg.append("[SEP]")
+                text_type_ids_neg.append(1)
+
+            token_ids_neg = tokenizer.convert_tokens_to_ids(tokens_neg)
+            position_ids_neg = list(range(len(token_ids_neg)))
+
+
         if self.is_inference:
             Record = namedtuple('Record',
                                 ['token_ids', 'text_type_ids', 'position_ids'])
@@ -197,6 +237,23 @@ class BaseReader(object):
                 token_ids=token_ids,
                 text_type_ids=text_type_ids,
                 position_ids=position_ids)
+        else:
+            qid = None
+            if "qid" in example._fields:
+                qid = example.qid
+            if self.learning_strategy == 'pairwise' and self.phase == 'train':
+                Record = namedtuple('Record',
+                                    ['token_ids', 'text_type_ids', 'position_ids', 'token_ids_neg', 'text_type_ids_neg', 'position_ids_neg', 'qid'])
+                
+                record = Record(
+                    token_ids=token_ids,
+                    text_type_ids=text_type_ids,
+                    position_ids=position_ids,
+                    token_ids_neg=token_ids_neg,
+                    text_type_ids_neg=text_type_ids_neg,
+                    position_ids_neg=position_ids_neg,
+                    qid=qid)
+ 
             else:
                 if self.label_map:
                     label_id = self.label_map[example.label]
@@ -207,10 +264,6 @@ class BaseReader(object):
                     'token_ids', 'text_type_ids', 'position_ids', 'label_id', 'qid'
                 ])
 
-            qid = None
-            if "qid" in example._fields:
-                qid = example.qid
-
                 record = Record(
                     token_ids=token_ids,
                     text_type_ids=text_type_ids,
@@ -219,7 +272,7 @@ class BaseReader(object):
                     qid=qid)
         return record
 
-    def _prepare_batch_data(self, examples, batch_size, phase=None):
+    def _prepare_batch_data(self, examples, batch_size, phase='train'):
         """generate batch records"""
         batch_records, max_len = [], 0
         if len(examples) < batch_size:
@@ -240,16 +293,14 @@ class BaseReader(object):
                 yield self._pad_batch_records(batch_records)
                 batch_records, max_len = [record], len(record.token_ids)
 
-        if phase == 'pred' and batch_records:
+        if phase == 'predict' and batch_records:
             yield self._pad_batch_records(batch_records)
 
-    def get_num_examples(self, input_file=None, phase=None):
-        if self.examples is not None:
-            if phase is None:
-                phase = 'all'
-            return len(self.examples[phase])
+    def get_num_examples(self, input_file=None, phase='train'):
+        if input_file is None:
+            return len(self.examples.get(phase, []))
         else:
-            assert input_file is not None, "Argument input_file should be given or the data_generator should be created when this func is called."
+            # assert input_file is not None, "Argument input_file should be given or the data_generator should be created when this func is called."
             examples = self._read_tsv(input_file)
             return len(examples)
 
@@ -285,6 +336,7 @@ class BaseReader(object):
                     if len(all_dev_batches) == dev_count:
                         for batch in all_dev_batches:
                             yield batch
+                        
                         all_dev_batches = []
         def f():
             for i in wrapper():
@@ -301,71 +353,7 @@ class BaseReader(object):
         return f
 
 
-class ClassifyReader(BaseReader):
-    def _read_tsv(self, input_file, quotechar=None):
-        """Reads a tab separated value file."""
-        with open(input_file, 'r', encoding='utf8') as f:
-            reader = csv_reader(f)
-            headers = next(reader)
-            text_indices = [
-                index for index, h in enumerate(headers) if h != "label"
-            ]
-            Example = namedtuple('Example', headers)
-
-            examples = []
-            for line in reader:
-                for index, text in enumerate(line):
-                    if index in text_indices:
-                        if self.for_cn:
-                            line[index] = text.replace(' ', '')
-                        else:
-                            line[index] = text
-                example = Example(*line)
-                examples.append(example)
-            return examples
-
-    def _pad_batch_records(self, batch_records):
-        batch_token_ids = [record.token_ids for record in batch_records]
-        batch_text_type_ids = [record.text_type_ids for record in batch_records]
-        batch_position_ids = [record.position_ids for record in batch_records]
-
-        if not self.is_inference:
-            batch_labels = [record.label_id for record in batch_records]
-            if self.is_classify:
-                batch_labels = np.array(batch_labels).astype("int64").reshape(
-                    [-1])
-            elif self.is_regression:
-                batch_labels = np.array(batch_labels).astype("float32").reshape(
-                    [-1])
-
-            if batch_records[0].qid:
-                batch_qids = [record.qid for record in batch_records]
-                batch_qids = np.array(batch_qids).astype("int64").reshape(
-                    [-1])
-            else:
-                batch_qids = np.array([]).astype("int64").reshape([-1])
-
-        # padding
-        padded_token_ids, input_mask = pad_batch_data(
-            batch_token_ids, pad_idx=self.pad_id, return_input_mask=True)
-        padded_text_type_ids = pad_batch_data(
-            batch_text_type_ids, pad_idx=self.pad_id)
-        padded_position_ids = pad_batch_data(
-            batch_position_ids, pad_idx=self.pad_id)
-        padded_task_ids = np.ones_like(
-            padded_token_ids, dtype="int64") * self.task_id
-
-        return_list = [
-            padded_token_ids, padded_text_type_ids, padded_position_ids,
-            padded_task_ids, input_mask
-        ]
-        if not self.is_inference:
-            return_list += [batch_labels, batch_qids]
-
-        return return_list
-
-
-class MaskLMReader(BaseReader):
+class MaskLMReader(Reader):
 
     def _convert_example_to_record(self, example, max_seq_length, tokenizer):
         """Converts a single `Example` into a single `Record`."""
@@ -432,13 +420,6 @@ class MaskLMReader(BaseReader):
         token_ids = tokenizer.convert_tokens_to_ids(tokens)
         position_ids = list(range(len(token_ids)))
 
-        # Record = namedtuple('Record',
-        #                     ['token_ids', 'text_type_ids', 'position_ids'])
-        # record = Record(
-        #     token_ids=token_ids,
-        #     text_type_ids=text_type_ids,
-        #     position_ids=position_ids)
-
         return [token_ids, text_type_ids, position_ids]
 
     def batch_reader(self, examples, batch_size, in_tokens, phase):
@@ -457,7 +438,7 @@ class MaskLMReader(BaseReader):
                 batch = [parsed_line]
                 total_token_num = len(parsed_line[0])
 
-        if len(batch) > 0 and phase == 'pred':
+        if len(batch) > 0 and phase == 'predict':
             yield batch, total_token_num
 
     def data_generator(self,
@@ -499,19 +480,102 @@ class MaskLMReader(BaseReader):
                         # max_len=self.max_seq_len, # 注意，如果padding到最大长度，会导致mask_pos与实际位置不对应。因为mask pos是基于batch内最大长度来计算的。
                         return_input_mask=True,
                         return_max_len=False,
-                        return_num_token=False)
+                        return_num_token=False,
+                        # dev_count=gpu_dev_count)
+                        dev_count=1)
 
-                    if len(all_dev_batches) < dev_count:
-                        all_dev_batches.append(batch_data)
-                    if len(all_dev_batches) == dev_count:
-                        for batch in all_dev_batches:
-                            yield batch
-                        all_dev_batches = []
+
+                    # yield batch
+                    for piece in palm.distribute.yield_pieces(batch_data, ['s', 's', 's', 's', 's', 'u', 'u'], batch_size):
+                        yield piece
 
         return wrapper
 
 
-class SequenceLabelReader(BaseReader):
+class ClassifyReader(Reader):
+    def _read_tsv(self, input_file, quotechar=None):
+        """Reads a tab separated value file."""
+        with open(input_file, 'r', encoding='utf8') as f:
+            reader = csv_reader(f)
+            headers = next(reader)
+            text_indices = [
+                index for index, h in enumerate(headers) if h != "label"
+            ]
+            Example = namedtuple('Example', headers)
+            examples = []
+            for line in reader:
+                for index, text in enumerate(line):
+                    if index in text_indices:
+                        if self.for_cn:
+                            line[index] = text.replace(' ', '')
+                        else:
+                            line[index] = text
+                example = Example(*line)
+                examples.append(example)
+            return examples
+
+    def _pad_batch_records(self, batch_records):
+        batch_token_ids = [record.token_ids for record in batch_records]
+        batch_text_type_ids = [record.text_type_ids for record in batch_records]
+        batch_position_ids = [record.position_ids for record in batch_records]
+        if self.phase=='train' and self.learning_strategy == 'pairwise':
+            batch_token_ids_neg = [record.token_ids_neg for record in batch_records]
+            batch_text_type_ids_neg = [record.text_type_ids_neg for record in batch_records]
+            batch_position_ids_neg = [record.position_ids_neg for record in batch_records]
+
+        if not self.is_inference:
+            if not self.learning_strategy == 'pairwise':
+                batch_labels = [record.label_id for record in batch_records]
+                if self.is_classify:
+                    batch_labels = np.array(batch_labels).astype("int64").reshape(
+                        [-1])
+                elif self.is_regression:
+                    batch_labels = np.array(batch_labels).astype("float32").reshape(
+                        [-1])
+
+            if batch_records[0].qid:
+                batch_qids = [record.qid for record in batch_records]
+                batch_qids = np.array(batch_qids).astype("int64").reshape(
+                    [-1])
+            else:
+                batch_qids = np.array([]).astype("int64").reshape([-1])
+
+        # padding
+        padded_token_ids, input_mask = pad_batch_data(
+            batch_token_ids, pad_idx=self.pad_id, return_input_mask=True)
+        padded_text_type_ids = pad_batch_data(
+            batch_text_type_ids, pad_idx=self.pad_id)
+        padded_position_ids = pad_batch_data(
+            batch_position_ids, pad_idx=self.pad_id)
+        padded_task_ids = np.ones_like(
+            padded_token_ids, dtype="int64") * self.task_id
+
+        return_list = [
+            padded_token_ids, padded_text_type_ids, padded_position_ids,
+            padded_task_ids, input_mask
+        ]
+
+        if self.phase=='train':
+            if self.learning_strategy == 'pairwise':
+                padded_token_ids_neg, input_mask_neg = pad_batch_data(
+                    batch_token_ids_neg, pad_idx=self.pad_id, return_input_mask=True)
+                padded_text_type_ids_neg = pad_batch_data(
+                    batch_text_type_ids_neg, pad_idx=self.pad_id)
+                padded_position_ids_neg = pad_batch_data(
+                    batch_position_ids_neg, pad_idx=self.pad_id)
+                padded_task_ids_neg = np.ones_like(
+                    padded_token_ids_neg, dtype="int64") * self.task_id
+
+                return_list += [padded_token_ids_neg, padded_text_type_ids_neg, \
+                                padded_position_ids_neg, padded_task_ids_neg, input_mask_neg]
+
+            elif self.learning_strategy == 'pointwise':
+                return_list += [batch_labels]
+
+        return return_list
+
+
+class SequenceLabelReader(Reader):
     def _pad_batch_records(self, batch_records):
         batch_token_ids = [record.token_ids for record in batch_records]
         batch_text_type_ids = [record.text_type_ids for record in batch_records]
@@ -552,19 +616,7 @@ class SequenceLabelReader(BaseReader):
                 ret_labels.append(label)
                 continue
 
-            if label == "O" or label.startswith("I-"):
             ret_labels.extend([label] * len(sub_token))
-            elif label.startswith("B-"):
-                i_label = "I-" + label[2:]
-                ret_labels.extend([label] + [i_label] * (len(sub_token) - 1))
-            elif label.startswith("S-"):
-                b_laebl = "B-" + label[2:]
-                e_label = "E-" + label[2:]
-                i_label = "I-" + label[2:]
-                ret_labels.extend([b_laebl] + [i_label] * (len(sub_token) - 2) + [e_label])
-            elif label.startswith("E-"):
-                i_label = "I-" + label[2:]
-                ret_labels.extend([i_label] * (len(sub_token) - 1) + [label])
 
         assert len(ret_tokens) == len(ret_labels)
         return ret_tokens, ret_labels
@@ -583,6 +635,9 @@ class SequenceLabelReader(BaseReader):
         position_ids = list(range(len(token_ids)))
         text_type_ids = [0] * len(token_ids)
         no_entity_id = len(self.label_map) - 1
+        labels = [
+            label if label in self.label_map else u"O" for label in labels
+        ]
         label_ids = [no_entity_id] + [
             self.label_map[label] for label in labels
         ] + [no_entity_id]
@@ -598,7 +653,7 @@ class SequenceLabelReader(BaseReader):
         return record
 
 
-class ExtractEmbeddingReader(BaseReader):
+class ExtractEmbeddingReader(Reader):
     def _pad_batch_records(self, batch_records):
         batch_token_ids = [record.token_ids for record in batch_records]
         batch_text_type_ids = [record.text_type_ids for record in batch_records]
@@ -625,7 +680,7 @@ class ExtractEmbeddingReader(BaseReader):
         return return_list
 
 
-class MRCReader(BaseReader):
+class MRCReader(Reader):
     def __init__(self,
                  vocab_path,
                  label_map_config=None,
@@ -890,11 +945,20 @@ class MRCReader(BaseReader):
             if to_append:
                 batch_records.append(record)
             else:
-                yield self._pad_batch_records(batch_records, phase == "train")
+                # yield self._pad_batch_records(batch_records, phase == "train")
+                ds = ['s'] * 8
+                for piece in palm.distribute.yield_pieces(\
+                        self._pad_batch_records(batch_records, phase == 'train'),
+                        ds, batch_size):
+                    yield piece
                 batch_records, max_len = [record], len(record.token_ids)
       
-        if phase == 'pred' and batch_records:
-            yield self._pad_batch_records(batch_records, phase == "train")
+        if phase == 'predict' and batch_records:
+            for piece in palm.distribute.yield_pieces(\
+                        self._pad_batch_records(batch_records, phase == 'train'),
+                        ds, batch_size):
+                yield piece
+
 
     def _pad_batch_records(self, batch_records, is_training):
         batch_token_ids = [record.token_ids for record in batch_records]
@@ -981,12 +1045,8 @@ class MRCReader(BaseReader):
   
                 for batch_data in self._prepare_batch_data(
                         features, batch_size, phase=phase):
-                    if len(all_dev_batches) < dev_count:
-                        all_dev_batches.append(batch_data)
-                    if len(all_dev_batches) == dev_count:
-                        for batch in all_dev_batches:
-                            yield batch
-                        all_dev_batches = []
+
+                    yield batch_data
 
         return wrapper
 

paddlepalm/task_instance.py

@@ -71,10 +71,10 @@ class TaskInstance(object):
         self._train_finish = False
 
         # 存放不同运行阶段（train，eval，pred）的数据集reader，key为phase，value为Reader实例
-        self._reader = {'train': None, 'eval': None, 'pred': None}
+        self._reader = {'train': None, 'eval': None, 'predict': None}
         self._input_layer = None
         self._inputname_to_varname = {}
-        self._task_layer = {'train': None, 'eval': None, 'pred': None}
+        self._task_layer = {'train': None, 'eval': None, 'predict': None}
         self._pred_input_name_list = []
         self._pred_input_varname_list = []
         self._pred_fetch_name_list = []
@@ -90,7 +90,7 @@ class TaskInstance(object):
 
     def build_task_layer(self, net_inputs, phase, scope=""):
         output_vars = self._task_layer[phase].build(net_inputs, scope_name=scope)
-        if phase == 'pred':
+        if phase == 'predict':
             if output_vars is not None:
                 self._pred_fetch_name_list, self._pred_fetch_var_list = zip(*output_vars.items())
             else:

paddlepalm/trainer.py

@@ -21,7 +21,7 @@ import time
 import numpy as np
 import paddlepalm.utils.basic_helper as helper
 from paddlepalm.utils import reader_helper, saver
-from paddlepalm.distribute import gpu_dev_count, data_feeder
+from paddlepalm.distribute import gpu_dev_count, data_feeder, decode_fake
 # from paddlepalm.default_settings import *
 
 DEBUG=False
@@ -29,16 +29,19 @@ DEBUG=False
 
 class Trainer(object):
 
-    def __init__(self, name, reader, task_head, \
-                 mix_ratio=1.0, reuse_head_with=None, \
+    def __init__(self, name, mix_ratio=1.0, reuse_head_with=None, \
                  silent=False):
 
         self._name = name
         self._verbose = not silent
-        self._reader = reader
         self._pred_reader = None
-        self._task_head = task_head
-        self._pred_head = pred_head
+        self._task_head = None
+        self._pred_head = None
+
+        self._train_init = False
+        self._predict_init = False
+
+        self._check_save = lambda: False
 
         # if save_predict_model:
         #     self._save_predict_model = True
@@ -58,20 +61,20 @@ class Trainer(object):
 
         self._num_examples = 0
 
+        self._multi_task = False
+        self._as_auxilary = False
+
         # training process management
         self._mix_ratio = mix_ratio
         self._expected_train_steps = None
         self._expected_train_epochs = None
         self._steps_pur_epoch = None
+        self._pred_steps_pur_epoch = None
         self._cur_train_epoch = 0
         self._cur_train_step = 0
         self._train_finish = False
 
-        # 存放不同运行阶段（train，eval，pred）的数据集reader，key为phase，value为Reader实例
-        # self._reader = {'train': reader, 'eval': None, 'pred': self._pred_reader}
-        # self._input_layer = None
         self._inputname_to_varname = {}
-        # self._task_layer = {'train': task_head, 'eval': None, 'pred': pred_head}
         self._pred_input_name_list = []
         self._pred_input_varname_list = []
         self._pred_fetch_name_list = []
@@ -89,24 +92,34 @@ class Trainer(object):
         self._lock = False
         self._build_forward = False
 
-    def build_predict_head(self, pred_backbone, pred_prog=None, pred_init_prog=None):
+    def build_predict_forward(self, pred_backbone, pred_head, pred_prog=None, pred_init_prog=None):
+        self._pred_head = pred_head
+        self._pred_backbone = pred_backbone
+        # self._pred_reader = self._reader.clone(phase='pred')
         pred_task_attr_from_reader = helper.encode_inputs(self._pred_head.inputs_attrs['reader'], self.name)
         # pred_task_attr_from_reader = self._pred_head.inputs_attrs['reader']
 
+        # _check_io(pred_backbone.inputs_attr, pred_reader.outputs_attr, in_name=bb_name+'_backbone', out_name='reader.pred')
+
         # _check_io(pred_backbone.inputs_attr, pred_reader.outputs_attr, in_name=bb_name+'_backbone', out_name='reader.pred')
         # _check_io(pred_parad.inputs_attrs['reader'], pred_reader.outputs_attr, in_name='task_paradigm.pred.reader', out_name='reader.pred')
         # _check_io(pred_parad.inputs_attrs['backbone'], pred_backbone.outputs_attr, in_name='task_paradigm.pred.backbone', out_name=bb_name+'_backbone')
-        pred_input_names, pred_shape_and_dtypes, _ = reader_helper.merge_input_attrs(backbone.inputs_attr, pred_task_attr_from_reader, insert_taskid=False, insert_batchsize=False, insert_seqlen=False, insert_batchsize_x_seqlen=False)
+        pred_input_names, pred_shape_and_dtypes, pred_name_to_position = reader_helper.merge_input_attrs(pred_backbone.inputs_attr, pred_task_attr_from_reader, insert_taskid=False)
         pred_input_attrs = [[i, j, k] for i, (j,k) in zip(pred_input_names, pred_shape_and_dtypes)]
+        self._pred_shape_and_dtypes = pred_shape_and_dtypes
+        self._pred_name_to_position = pred_name_to_position
         
         if pred_prog is None:
             pred_prog = fluid.Program()
+        self._pred_prog = pred_prog
         if pred_init_prog is None:
             pred_init_prog = fluid.Program()
+        self._pred_init_prog = pred_init_prog
         with fluid.program_guard(pred_prog, pred_init_prog):
             pred_net_inputs = reader_helper.create_net_inputs(pred_input_attrs)
             # pred_bb_output_vars = pred_backbone.build(pred_net_inputs, scope_name='__paddlepalm_')
             pred_bb_output_vars = pred_backbone.build(pred_net_inputs)
+            self._pred_net_inputs = pred_net_inputs
 
         # prepare predict vars for saving inference model
         with fluid.program_guard(pred_prog, pred_init_prog):
@@ -118,12 +131,23 @@ class Trainer(object):
             pred_task_inputs = {'backbone': pred_bb_output_vars, 'reader': cur_inputs}
             scope = self.name + '.'
             with fluid.unique_name.guard(scope):
-                self._build_head(pred_task_inputs, phase='pred', scope=scope)
+                output_vars = self._build_head(pred_task_inputs, phase='pred', scope=scope)
 
+        if output_vars is not None:
+            self._pred_fetch_name_list, self._pred_fetch_list = zip(*output_vars.items())
+        else:
+            self._pred_fetch_name_list = []
+            self._pred_fetch_var_list = []
 
+        return output_vars
 
+    def _set_multitask(self):
+        self._multi_task = True
 
-    def build_forward(self, backbone, pred_backbone=None, train_prog=None, train_init_prog=None, pred_prog=None, pred_init_prog=None):
+    def build_forward(self, backbone, task_head):
+        # assert not self._multi_task, "you cannot build_forward in trainer when a train is wrapper by MultiHeadTrainer."
+        self._task_head = task_head
+        self._backbone = backbone
 
         # assert self._backbone is not None, "backbone is required for Trainer to build net forward to run with single task mode"
         self._build_forward = True
@@ -142,10 +166,11 @@ class Trainer(object):
 
 
         # merge reader input attrs from backbone and task_instances
-        input_names, shape_and_dtypes, name_to_position = reader_helper.merge_input_attrs(backbone.inputs_attr, task_attr_from_reader, insert_taskid=False, insert_batchsize=False, insert_seqlen=False, insert_batchsize_x_seqlen=False)
+        input_names, shape_and_dtypes, name_to_position = reader_helper.merge_input_attrs(backbone.inputs_attr, task_attr_from_reader, insert_taskid=False)
         # shapes: [task_id, shapes_of_backbone, shapes_of_inst1, ..., shapes_of_instN]
         self._shape_and_dtypes = shape_and_dtypes
         self._name_to_position = name_to_position
+        self._input_names = input_names
 
         if DEBUG:
             print('----- for debug -----')
@@ -154,25 +179,24 @@ class Trainer(object):
             print('joint input shape and dtypes:')
             print(joint_shape_and_dtypes)
 
-
         input_attrs = [[i, j, k] for i, (j,k) in zip(input_names, shape_and_dtypes)]
 
-        if train_prog is None:
         train_prog = fluid.Program()
-        if train_init_prog is None:
         train_init_prog = fluid.Program()
-        self._prog = train_prog
+
         self._train_prog = train_prog
         self._train_init_prog = train_init_prog
+        if not self._multi_task:
             with fluid.program_guard(train_prog, train_init_prog):
                 net_inputs = reader_helper.create_net_inputs(input_attrs, async=False)
-            self._net_inputs = net_inputs
-
-            # build backbone and task layers
-            # bb_output_vars = self._backbone.build(net_inputs, scope_name='__paddlepalm_')
                 bb_output_vars = backbone.build(net_inputs)
+        else:
+            net_inputs = reader_helper.create_net_inputs(input_attrs, async=False)
+            bb_output_vars = backbone.build(net_inputs)
+        self._net_inputs = net_inputs
         assert sorted(bb_output_vars.keys()) == sorted(backbone.outputs_attr.keys())
 
+        # self._bb_output_vars.keys
 
         # fluid.framework.switch_main_program(train_prog)
         # fluid.framework.switch_startup_program(train_init_prog)
@@ -183,9 +207,14 @@ class Trainer(object):
         task_inputs['reader'] = task_inputs_from_reader
 
         scope = self.name+'.'
+        if not self._multi_task:
             with fluid.program_guard(train_prog, train_init_prog):
                 with fluid.unique_name.guard(scope):
                     output_vars = self._build_head(task_inputs, phase='train', scope=scope)
+        else:
+            with fluid.unique_name.guard(scope):
+                output_vars = self._build_head(task_inputs, phase='train', scope=scope)
+
         output_vars = {self.name+'.'+key: val for key, val in output_vars.items()}
         old = len(task_output_vars) # for debug
         task_output_vars.update(output_vars)
@@ -203,15 +232,23 @@ class Trainer(object):
         # task_id_vec = layers.one_hot(task_id_var, num_instances)
         # losses = fluid.layers.concat([task_output_vars[inst.name+'/loss'] for inst in instances], axis=0)
         # loss = layers.reduce_sum(task_id_vec * losses)
+        if not self._multi_task:
             with fluid.program_guard(train_prog, train_init_prog):
                 loss_var = fluid.layers.reduce_sum(task_output_vars[self.name+'.loss'])
+        else:
+            loss_var = fluid.layers.reduce_sum(task_output_vars[self.name+'.loss'])
 
-        self._distribute_train_prog = fluid.CompiledProgram(self._train_prog).with_data_parallel(loss_name=loss_var.name)
+        # for _id, block in enumerate(self._train_prog.blocks):
+        #   for var in block.vars:
+        #     print("[debug] : %d, %s" % (_id, var))
+        self._loss_var = loss_var
         return loss_var
 
-    def build_backward(self, optimizer, weight_decay=None, use_ema=False, ema_decay=0.9999):
+    def build_backward(self, optimizer, weight_decay=None, use_ema=False, ema_decay=None):
+        # assert not self._multi_task, "you cannot build_backward in trainer when a train is wrapper by MultiHeadTrainer."
         # build optimizer
-        optimizer._set_prog(self._train_prog)
+        assert self._train_init_prog is not None, "train graph not foung! You should build_forward first."
+        optimizer._set_prog(self._train_prog, self._train_init_prog)
         with fluid.program_guard(self._train_prog, self._train_init_prog):
             param_grads = optimizer.build()
 
@@ -219,7 +256,7 @@ class Trainer(object):
 
                 param_list = dict()
 
-                for param in self._prog.global_block().all_parameters():
+                for param in self._train_prog.global_block().all_parameters():
                     param_list[param.name] = param * 1.0
                     param_list[param.name].stop_gradient = True
 
@@ -247,73 +284,186 @@ class Trainer(object):
                 ema = fluid.optimizer.ExponentialMovingAverage(ema_decay)
                 ema.update()
 
-    def load_data(self, input_file, file_format, batch_size, num_epochs=None, shuffle_train=True):
+        # for bid, block in enumerate(self._train_prog.blocks):
+        #     print('block id: '+str(bid))
+        #     for var in block.vars:
+        #         print("%d : %s" % (bid, var))
+            
+        # print(self._train_prog)
+
+    def set_as_aux(self):
+        self._as_auxilary = True
+
+    def fit_reader(self, reader, phase='train', task_id=None):
+        # assert not self._multi_task, "you cannot fit_reader in trainer when a train is wrapper by MultiHeadTrainer."
         # load data
-        print("preparing data...", end='')
-        self._reader._load_data(input_file=input_file, batch_size=batch_size, \
-                                num_epochs=num_epochs, file_format=file_format, \
-                                shuffle_train=shuffle_train)
-        self._num_examples = self._reader.num_examples
+
+        assert self._shape_and_dtypes is not None or self._pred_shape_and_dtypes is not None, "You need to build_forward or build_predict_head first to prepare input features."
+
         # 这里不确定是否要向上取整，需确认
         # tail = self._num_examples % batch_size > 0
         # self._steps_pur_epoch = self._num_examples // batch_size + 1 if tail else 0
-        self._steps_pur_epoch = self._num_examples // batch_size
+        batch_size = reader._batch_size
+        self._num_epochs = reader.num_epochs
+        if phase == 'train':
+            self._steps_pur_epoch = reader.num_examples // batch_size
+            shape_and_dtypes = self._shape_and_dtypes
+            name_to_position = self._name_to_position
+            if task_id is not None:
+                self._net_inputs['__task_id'] = task_id
+            net_inputs = self._net_inputs
+            self._train_batch_size = batch_size
+            self._num_examples = reader.num_examples
+            reader_helper.check_io(self._backbone.inputs_attr, reader.outputs_attr, in_name='backbone', out_name='reader(train)')
+            reader_helper.check_io(self._task_head.inputs_attrs['reader'], reader.outputs_attr, in_name='task_head(reader)', out_name='reader(train)')
+            reader_helper.check_io(self._task_head.inputs_attrs['backbone'], self._backbone.outputs_attr, in_name='task_head(backbone, train)', out_name='backbone')
+        elif phase == 'predict':
+            tail = self._num_examples % batch_size > 0
+            self._pred_steps_pur_epoch = reader.num_examples // batch_size + 1 if tail else 0
+            shape_and_dtypes = self._pred_shape_and_dtypes
+            name_to_position = self._pred_name_to_position
+            net_inputs = self._pred_net_inputs
+            self._predict_batch_size = batch_size
+            self._pred_num_examples = reader.num_examples
+            reader_helper.check_io(self._pred_backbone.inputs_attr, reader.outputs_attr, in_name='backbone', out_name='reader(predict)')
+            reader_helper.check_io(self._pred_head.inputs_attrs['reader'], reader.outputs_attr, in_name='task_head(reader)', out_name='reader(predict)')
+            reader_helper.check_io(inst._pred_head.inputs_attrs['backbone'], self._pred_backbone.outputs_attr, in_name='task_head(backbone, predict)', out_name='backbone')
+        else:
+            raise NotImplementedError()
+            
         print('ok!')
 
         # merge dataset iterators and create net input vars
-        iterator = self._reader._iterator()
+        iterator = reader._iterator()
+        prefix = self.name
+
+
+        # merge dataset iterators and create net input vars
+        iterator = reader._iterator()
         prefix = self.name
 
         # 对yield出的数据进行runtime检查和适配
-        iterator_fn = reader_helper.create_iterator_fn(iterator, prefix, self._shape_and_dtypes, self._name_to_position, return_type='dict')
-        feed_batch_process_fn = reader_helper.create_feed_batch_process_fn(self._net_inputs)
-        self._feed_batch_process_fn = feed_batch_process_fn
+        iterator_fn = reader_helper.create_iterator_fn(iterator, prefix, shape_and_dtypes, name_to_position, return_type='dict')
+        self._raw_iterator_fn = iterator_fn
+        feed_batch_process_fn = reader_helper.create_feed_batch_process_fn(net_inputs)
         if gpu_dev_count > 1:
             distribute_feeder_fn = data_feeder(iterator_fn, feed_batch_process_fn)
         else:
             distribute_feeder_fn = iterator_fn
-        return distribute_feeder_fn()
 
-    def random_init_params(self):
+        if phase == 'train':
+            self._train_reader = distribute_feeder_fn()
+            self._feed_batch_process_fn = feed_batch_process_fn
+        elif phase == 'predict':
+            self._predict_reader = distribute_feeder_fn()
+            self._pred_feed_batch_process_fn = feed_batch_process_fn
+        # return distribute_feeder_fn()
+
+    def _init_exe_prog(self, for_train=True):
+        if not self._train_init and not self._predict_init:
             on_gpu = gpu_dev_count > 0
             self._exe = helper.build_executor(on_gpu)
+
+        if for_train:
+            assert self._train_prog is not None, "train graph not foung! You should build_forward first before you random init parameters."
+            self._train_init = True
+        else:
+            assert self._pred_prog is not None, "predict graph not foung! You should build_predict_head first before you random init parameters."
+            self._predict_init = True
+
+    def random_init_params(self):
+        
+        if not self._train_init:
+            self._init_exe_prog()
+        
         print('random init params...')
         self._exe.run(self._train_init_prog)
 
-    def load_pretrain(self, model_path):
+    def load_ckpt(self, model_path, phase='train'):
+        # load pretrain model (or ckpt)
+        # assert self._exe is not None, "You need to random_init_params before load checkpoints."
+        if phase == 'train' and not self._train_init:
+            self._init_exe_prog(for_train=True)
+            self._exe.run(self._train_init_prog)
+        if phase == 'predict' and not self._predict_init:
+            self._init_exe_prog(for_train=False)
+            self._exe.run(self._pred_init_prog)
+
+        if phase == 'train':
+            assert self._train_init_prog is not None, "train graph not found! You should build_forward first before load checkpoint."
+            saver.init_pretraining_params(
+                self._exe,
+                model_path,
+                main_program=self._train_init_prog,
+                strict=True)
+        elif phase == 'predict':
+            assert self._pred_init_prog is not None, "predict graph not found! You should build_predict_head first before load checkpoint."
+            saver.init_pretraining_params(
+                self._exe,
+                model_path,
+                main_program=self._pred_init_prog,
+                strict=True)
+        else:
+            raise NotImplementedError()
+            
+
+    def load_predict_model(self, model_path):
+        raise NotImplementedError()
+
+    def load_pretrain(self, model_path, convert=False):
         # load pretrain model (or ckpt)
         assert self._exe is not None, "You need to random_init_params before load pretrain models."
 
         saver.init_pretraining_params(
             self._exe,
             model_path,
+            convert=convert,
             main_program=self._train_init_prog)
 
-    def set_predict_head(self):
-        pass
-
-    def train(self, iterator, save_path=None, save_steps=None, save_type='ckpt', print_steps=5):
+    def set_saver(self, save_path, save_steps, save_type='ckpt'):
 
         save_type = save_type.split(',')
         if 'predict' in save_type:
-            assert self._pred_head is not None, "Predict head not found! You should call set_predict_head first if you want to save predict model."
+            assert self._pred_head is not None, "Predict head not found! You should build_predict_head first if you want to save predict model."
             assert save_path is not None and save_steps is not None, 'save_path and save_steps is required to save model.'
-            save_predict = True
+            self._save_predict = True
             if not os.path.exists(save_path):
                 os.makedirs(save_path)
         else:
-            save_predict = False
+            self._save_predict = False
 
         if 'ckpt' in save_type:
             if save_path is not None and save_steps is not None:
-                save_ckpt = True
+                self._save_ckpt = True
                 if not os.path.exists(save_path):
                     os.makedirs(save_path)
             else:
                 "WARNING: save_path or save_steps is not set, model will not be saved during training."
-                save_ckpt = False
+                self._save_ckpt = False
         else:
-            save_ckpt = False
+            self._save_ckpt = False
+
+        def temp_func():
+            if (self._save_predict or self._save_ckpt) and self._cur_train_step % save_steps == 0:
+                if self._save_predict:
+                    self.save(save_path, suffix='pred.step'+str(self._cur_train_step))
+                    print('predict model has been saved at '+os.path.join(save_path, 'pred.step'+str(self._cur_train_step)))
+                if self._save_ckpt:
+                    fluid.io.save_persistables(self._exe, os.path.join(save_path, 'ckpt.step'+str(self._cur_train_step)), self._train_prog)
+                    print('checkpoint has been saved at '+os.path.join(save_path, 'ckpt.step'+str(self._cur_train_step)))
+                return True
+            else:
+                return False
+
+        self._check_save = temp_func
+            
+    def train(self, save_path=None, save_steps=None, save_type='ckpt', print_steps=5):
+        """
+        Argument:
+            save_type: ckpt, predict, pretrain
+        """
+        iterator = self._train_reader
+        self._distribute_train_prog = fluid.CompiledProgram(self._train_prog).with_data_parallel(loss_name=self._loss_var.name)
 
         # if save_path is not None or save_steps is not None:
         #     assert self._save_predict_model, "If you want to save model, you need set save_predict_model=True when this trainer is built."
@@ -344,10 +494,7 @@ class Trainer(object):
             # rt_outputs = {k:v for k,v in zip(self._fetch_names, rt_outputs)}
 
             task_rt_outputs = {k[len(self.name+'.'):]: v for k,v in rt_outputs.items() if k.startswith(self.name+'.')}
-            self._task_head.postprocess(task_rt_outputs)
-
-            self._cur_train_step += 1
-            self._cur_train_epoch = (self._cur_train_step-1) // self._steps_pur_epoch
+            self._task_head.batch_postprocess(task_rt_outputs)
 
             # if self._save_predict_model and self._cur_train_step % save_steps == 0:
             #     self.save(save_path, suffix='.step'+str(self._cur_train_steps))
@@ -368,13 +515,8 @@ class Trainer(object):
             #     print(cur_task.name+': train finished!')
             #     cur_task.save()
 
-            if (save_predict or save_ckpt) and self._cur_train_step % save_steps == 0:
-                if save_predict_model:
-                    self.save(save_path, suffix='pred.step'+str(global_step))
-                if save_ckpt:
-                    fluid.io.save_persistables(self.exe, os.path.join(save_path, 'ckpt.step'+str(global_step)), self._train_prog)
-                    print('checkpoint has been saved at '+os.path.join(save_path, 'ckpt.step'+str(global_step)))
-
+            if self._num_epochs is None and not self._multi_task and self._cur_train_step == self._steps_pur_epoch:
+                break
         # save_path = os.path.join(main_conf['save_path'], 'ckpt',
         #                          "step_" + str(global_step))
         # fluid.io.save_persistables(self.exe, save_path, saver_program)
@@ -382,38 +524,112 @@ class Trainer(object):
 
         # print("ALL tasks train finished, exiting...")
 
-    def train_one_step(self, batch):
+    def get_one_batch(self, phase='train'):
+        if phase == 'train':
+            return next(self._train_reader)
+        elif phase == 'predict':
+            return next(self._predict_reader)
+        else:
+            raise NotImplementedError()
+        
+    def predict(self, output_dir=None, print_steps=1000):
+        """
+        Argument:
+            save_type: ckpt, predict, pretrain
+        """
+        iterator = self._predict_reader
+        self._distribute_pred_prog = fluid.CompiledProgram(self._pred_prog).with_data_parallel()
+
+        if output_dir is not None and not os.path.exists(output_dir):
+            os.makedirs(output_dir)
+
+        time_begin = time.time()
+        cur_predict_step = 0
+        for feed in iterator:
+            rt_outputs = self.predict_one_batch(feed)
+            # rt_outputs = {k[len(self.name+'.'):]: v for k,v in rt_outputs.items() if k.startswith(self.name+'.')}
+            # print(rt_outputs)
+            self._pred_head.batch_postprocess(rt_outputs)
+
+            cur_predict_step += 1
+
+            if print_steps > 0 and cur_predict_step % print_steps == 0:
+                time_end = time.time()
+                time_cost = time_end - time_begin
+
+                print("batch {}/{}, speed: {:.2f} steps/s".format(
+                       cur_predict_step, self._pred_steps_pur_epoch,
+                       print_steps / time_cost))
+                time_begin = time.time()
+
+        if self._pred_head.epoch_inputs_attrs:
+            reader_outputs = self._pred_reader.get_epoch_outputs()
+        else:
+            reader_outputs = None
+
+        results = self._pred_head.epoch_postprocess({'reader':reader_outputs}, output_dir=output_dir)
+        return results
+
+    def train_one_step(self, batch, executor=None, distribute_train_prog=None, fetch_list=None):
+        exe = self._exe if executor is None else executor
+        distribute_train_prog = self._distribute_train_prog if distribute_train_prog is None else distribute_train_prog
+        fetch_list = self._fetch_list if fetch_list is None else fetch_list
+
         if gpu_dev_count > 1:
             feed, mask = batch
-            rt_outputs = self.exe.run(self._distribute_train_prog, feed=feed, fetch_list=self._fetch_list)
-            while mask.pop() == False:
-                rt_outputs.pop()
+            rt_outputs = exe.run(distribute_train_prog, feed=feed, fetch_list=fetch_list)
+            num_fakes = decode_fake(len(rt_outputs[0]), mask, self._batch_size)
+            for _ in range(num_fakes):
+                for item in rt_outputs:
+                    item.pop()
         else:
             feed = self._feed_batch_process_fn(batch)
-            rt_outputs = self._exe.run(self._distribute_train_prog, feed=feed, fetch_list=self._fetch_list)
+            rt_outputs = exe.run(distribute_train_prog, feed=feed, fetch_list=fetch_list)
 
         rt_outputs = {k:v for k,v in zip(self._fetch_names, rt_outputs)}
+        self._cur_train_step += 1
+        self._cur_train_epoch = (self._cur_train_step-1) // self._steps_pur_epoch
+        self._check_save()
         return rt_outputs
 
+    @property
+    def num_epochs(self):
+        return self._num_epochs
+
+    @property
+    def cur_train_steps(self):
+        return self._cur_train_step
+
+    @property
+    def cur_train_epoch(self):
+        return self._cur_train_epoch
+
+    @property
+    def steps_pur_epoch(self):
+        return self._steps_pur_epoch
+
+    def predict_one_batch(self, batch):
+        if gpu_dev_count > 1:
+            feed, mask = batch
+            rt_outputs = self.exe.run(self._distribute_pred_prog, feed=feed, fetch_list=self._pred_fetch_list)
+            num_fakes = decode_fake(len(rt_outputs[0]), mask, self._batch_size)
+            for _ in range(num_fakes):
+                for item in rt_outputs:
+                    item.pop()
+        else:
+            feed = self._pred_feed_batch_process_fn(batch)
+            rt_outputs = self._exe.run(self._distribute_pred_prog, feed=feed, fetch_list=self._pred_fetch_list)
+
+        rt_outputs = {k:v for k,v in zip(self._pred_fetch_name_list, rt_outputs)}
+        return rt_outputs
 
     def _build_head(self, net_inputs, phase, scope=""):
         if phase == 'train':
             output_vars = self._task_head.build(net_inputs, scope_name=scope)
         if phase == 'pred':
             output_vars = self._pred_head.build(net_inputs, scope_name=scope)
-            if output_vars is not None:
-                self._pred_fetch_name_list, self._pred_fetch_var_list = zip(*output_vars.items())
-            else:
-                self._pred_fetch_name_list = []
-                self._pred_fetch_var_list = []
         return output_vars
     
-    def _postprocess(self, rt_outputs, phase):
-        return self._task_layer[phase].postprocess(rt_outputs)
-
-    def _epoch_postprocess(self, epoch_inputs, phase):
-        return self._task_layer[phase].epoch_postprocess(epoch_inputs)
-    
     def save(self, save_path, suffix=None):
         # dirpath = save_path.rstrip('/').rstrip('\\') + suffix
         if suffix is not None:
@@ -422,7 +638,7 @@ class Trainer(object):
             dirpath = save_path
         self._pred_input_varname_list = [str(i) for i in self._pred_input_varname_list]
 
-        prog = fluid.default_main_program().clone()
+        prog = self._pred_prog.clone()
         fluid.io.save_inference_model(dirpath, self._pred_input_varname_list, self._pred_fetch_var_list, self._exe, prog)
 
         conf = {}
@@ -435,6 +651,7 @@ class Trainer(object):
             writer.write(json.dumps(conf, indent=1))
         print(self._name + ': predict model saved at ' + dirpath)
 
+    
     def _load(self, infer_model_path=None):
         if infer_model_path is None:
             infer_model_path = self._save_infermodel_path
@@ -454,20 +671,6 @@ class Trainer(object):
     def num_examples(self):
         return self._num_examples
 
-    # @property
-    # def _pred_input(self):
-    #     return zip(*[self._pred_input_name_list, self._pred_input_varname_list])
-
-    # @_pred_input.setter
-    # def _pred_input(self, val):
-    #     assert isinstance(val, dict)
-    #     self._pred_input_name_list, self._pred_input_varname_list = \
-    #         zip(*[[k, v.name] for k,v in val.items()])
-
-    # @property
-    # def _pred_fetch_list(self):
-    #     return [self._pred_fetch_name_list, self._pred_fetch_var_list]
-
     @property
     def mix_ratio(self):
         if self._mix_ratio is not None:
@@ -477,66 +680,5 @@ class Trainer(object):
 
     @mix_ratio.setter
     def mix_ratio(self, value):
-        self._mix_ratio = float(value)
-        if self._verbose:
-            print('{}: mix_ratio is set to {}'.format(self._name, self._mix_ratio))
-
-    @property
-    def save_infermodel_every_n_steps(self):
-        return self._save_infermodel_every_n_steps
-
-    @save_infermodel_every_n_steps.setter
-    def save_infermodel_every_n_steps(self, val):
-        self._save_infermodel_every_n_steps = val
-
-    @property
-    def expected_train_steps(self):
-        return self._expected_train_steps
-
-    @expected_train_steps.setter
-    def expected_train_steps(self, value):
-        self._expected_train_steps = value
-        self._expected_train_epochs = value / float(self._steps_pur_epoch)
-
-    @property
-    def expected_train_epochs(self):
-        return self._expected_train_epochs
-
-    @property
-    def cur_train_epoch(self):
-        return self._cur_train_epoch
-
-    @property
-    def cur_train_step(self):
-        return self._cur_train_step
-
-    # @cur_train_step.setter
-    # def _cur_train_step(self, value):
-    #     self._cur_train_step = value
-    #     if self._cur_train_step > self._steps_pur_epoch:
-    #         self._cur_train_epoch += 1
-    #         self._cur_train_step = 1
-    #     if self._is_target and self._cur_train_step + self._cur_train_epoch * self._steps_pur_epoch >= self._expected_train_steps:
-    #         self._train_finish = True
-
-    @property
-    def steps_pur_epoch(self):
-        return self._steps_pur_epoch
-
-    @steps_pur_epoch.setter
-    def steps_pur_epoch(self, value):
-        self._steps_pur_epoch = value
-
-    @property
-    def train_finish(self):
-        return self._train_finish
-
-    def tasklayer_reuse_with(self, task):
-        assert isinstance(task, Task)
-        if self._lock:
-            raise Exception('you can only set tasklayer reuses BEFORE Controller created.')
-        self._task_reuse_scope = task.name
-    
-    def _set_lock(self):
         self._lock = True
 

paddlepalm/utils/basic_helper.py

@@ -3,6 +3,7 @@ import os
 import json
 import yaml
 from config_helper import PDConfig
+import logging
 from paddle import fluid
 
 def get_basename(f):

paddlepalm/utils/reader_helper.py

@@ -16,6 +16,7 @@
 import os
 import sys
 import random
+import logging
 import numpy as np
 import paddle
 from paddle import fluid
@@ -35,10 +36,43 @@ def create_feed_batch_process_fn(net_inputs):
 
     return feed_batch_process_fn
 
+
+# def create_multihead_feed_batch_process_fn(net_inputs):
+# 
+#     def feed_batch_process_fn(data, id=-1):
+#         # temps = {}
+#         # for i in range(len(net_inputs)):
+#         temp = {}
+#         inputs = net_inputs[id] if id != -1 else net_inputs
+#         
+#         for q, var in inputs.items():
+#             if isinstance(var, str) or isinstance(var, unicode):
+#                 temp[var] = data[q]
+#             else:
+#                 temp[var.name] = data[q]
+#             # temps[i] = temp
+#             
+#         return temp
+# 
+#     return feed_batch_process_fn
+
+
+def check_io(in_attr, out_attr, strict=False, in_name="left", out_name="right"):
+    for name, attr in in_attr.items():
+        assert name in out_attr, in_name+': '+name+' not found in '+out_name
+        if attr != out_attr[name]:
+            if strict:
+                raise ValueError(name+': shape or dtype not consistent!')
+            else:
+                logging.warning('{}: shape or dtype not consistent!\n{}:\n{}\n{}:\n{}'.format(name, in_name, attr, out_name, out_attr[name]))
+
+
 def _check_and_adapt_shape_dtype(rt_val, attr, message=""):
     if not isinstance(rt_val, np.ndarray):
+        if rt_val is None:
+            raise Exception(message+": get None value. ")
         rt_val = np.array(rt_val)
-        assert rt_val.dtype != np.dtype('O'), "yielded data is not a valid tensor(number of elements on some dimension may differ)."
+        assert rt_val.dtype != np.dtype('O'), message+"yielded data is not a valid tensor (number of elements on some dimension may not consistent): {}".format(rt_val)
         if rt_val.dtype == np.dtype('float64'):
             rt_val = rt_val.astype('float32')
     
@@ -126,6 +160,41 @@ def create_iterator_fn(iterator, iterator_prefix, shape_and_dtypes, outname_to_p
 
     return iterator_fn
 
+def create_multihead_iterator_fn(iterators, iterator_prefixes, joint_shape_and_dtypes, mrs, names, outname_to_pos, dev_count=1, keep_one_task=True):
+    task_ids = range(len(iterators))
+    weights = [mr / float(sum(mrs)) for mr in mrs]
+    if not keep_one_task:
+        dev_count = 1
+
+    def iterator():
+        while True:
+            id = np.random.choice(task_ids, p=weights)
+            task_id_tensor = np.array([id]).astype("int64")
+            
+            for i in range(dev_count):
+                
+                outputs = next(iterators[id]) # dict type
+
+                prefix = iterator_prefixes[id]
+                results = {}
+                results['__task_id'] = task_id_tensor
+                for outname, val in outputs.items():
+                    task_outname = prefix + '.' + outname
+
+                    if outname in names[id]:
+                        idx = outname_to_pos[id][outname]
+                        val = _check_and_adapt_shape_dtype(val, joint_shape_and_dtypes[id][idx], message=outname+': ')
+                        results[outname] = val
+
+                    if task_outname in names[id]:
+                        idx = outname_to_pos[id][task_outname]
+                        val = _check_and_adapt_shape_dtype(val, joint_shape_and_dtypes[id][idx], message=task_outname+': ')
+                        results[task_outname] = val
+
+                yield results
+
+    return iterator
+
 
 def create_joint_iterator_fn(iterators, iterator_prefixes, joint_shape_and_dtypes, mrs, outname_to_pos, dev_count=1, keep_one_task=True, verbose=0):
     """
@@ -229,7 +298,7 @@ def create_joint_iterator_fn(iterators, iterator_prefixes, joint_shape_and_dtype
     return iterator
 
 
-def merge_input_attrs(backbone_attr, task_attrs, insert_taskid=True, insert_batchsize=True, insert_seqlen=True, insert_batchsize_x_seqlen=True):
+def merge_input_attrs(backbone_attr, task_attrs, insert_taskid=True, insert_batchsize=False, insert_seqlen=False, insert_batchsize_x_seqlen=False):
     """
     Args:
         task_attrs(list[dict]|dict): task input attributes, key=attr_name, val=[shape, dtype], support single task and nested tasks
@@ -241,7 +310,7 @@ def merge_input_attrs(backbone_attr, task_attrs, insert_taskid=True, insert_batc
     names = []
     start = 0
     if insert_taskid:
-        ret.append(([1,1], 'int64'))
+        ret.append(([1, 1], 'int64'))
         names.append('__task_id')
         start += 1
     
@@ -273,5 +342,3 @@ def merge_input_attrs(backbone_attr, task_attrs, insert_taskid=True, insert_batc
         for pos, k in enumerate(task_names, start=len(name_to_position)):
             name_to_position[k] = pos
     return names, ret, name_to_position
-    
-

paddlepalm/utils/saver.py

@@ -46,14 +46,15 @@ def init_checkpoint(exe, init_checkpoint_path, main_program, skip_list = []):
 
 def init_pretraining_params(exe,
                             pretraining_params_path,
-                            main_program):
+                            convert,
+                            main_program,
+                            strict=False):
+                            
     assert os.path.exists(pretraining_params_path
                           ), "[%s] cann't be found." % pretraining_params_path
 
-
+    if convert:
         assert os.path.exists(os.path.join(pretraining_params_path, '__palmmodel__')), "__palmmodel__ not found."
-    print("Loading pretraining parameters from {}...".format(
-        pretraining_params_path))
 
         with tarfile.open(os.path.join(pretraining_params_path, '__palmmodel__'), 'r') as f:
             f.extractall(os.path.join(pretraining_params_path, '.temp'))
@@ -61,10 +62,18 @@ def init_pretraining_params(exe,
         log_path = os.path.join(pretraining_params_path, '__palmmodel__')
         pretraining_params_path = os.path.join(pretraining_params_path, '.temp')
 
+    else:
+        log_path = pretraining_params_path
+    
+    print("Loading pretraining parameters from {}...".format(pretraining_params_path))
+
     def existed_params(var):
         if not isinstance(var, fluid.framework.Parameter):
             return False
         if not os.path.exists(os.path.join(pretraining_params_path, var.name)):
+            if strict:
+                raise Exception('Error: {} not found in {}.'.format(var.name, log_path))
+            else:
                 print('Warning: {} not found in {}.'.format(var.name, log_path))
         return os.path.exists(os.path.join(pretraining_params_path, var.name))
 
@@ -73,7 +82,7 @@ def init_pretraining_params(exe,
         pretraining_params_path,
         main_program=main_program,
         predicate=existed_params)
-
+    if convert:
         shutil.rmtree(pretraining_params_path)
     print('')
 

reader/cls.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from paddlepalm.interface import reader
-from paddlepalm.reader.utils.reader4ernie import ClassifyReader
-
-class Reader(reader):
-    
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
-        """
-        Args:
-            phase: train, eval, pred
-            """
-
-        self._is_training = phase == 'train'
-
-        reader = ClassifyReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', False),
-            for_cn=config.get('for_cn', False),
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
-
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
-        self._num_classes = config['n_classes']
-
-        if phase == 'train':
-            self._input_file = config['train_file']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            # self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        elif phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-
-        self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 0)
-
-
-    @property
-    def outputs_attr(self):
-        if self._is_training:
-            return {"token_ids": [[-1, -1, 1], 'int64'],
-                    "position_ids": [[-1, -1, 1], 'int64'],
-                    "segment_ids": [[-1, -1, 1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32'],
-                    "label_ids": [[-1,1], 'int64'],
-                    "task_ids": [[-1, -1, 1], 'int64']
-                    }
-        else:
-            return {"token_ids": [[-1, -1, 1], 'int64'],
-                    "position_ids": [[-1, -1, 1], 'int64'],
-                    "segment_ids": [[-1, -1, 1], 'int64'],
-                    "task_ids": [[-1, -1, 1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32']
-                    }
-
-
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
-
-    def iterator(self): 
-
-        def list_to_dict(x):
-            names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 
-                'label_ids', 'unique_ids']
-            outputs = {n: i for n,i in zip(names, x)}
-            del outputs['unique_ids']
-            if not self._is_training:
-                del outputs['label_ids']
-            return outputs
-
-        for batch in self._data_generator():
-            yield list_to_dict(batch)
-
-    def get_epoch_outputs(self):
-        return {'examples': self._reader.get_examples(self._phase),
-                'features': self._reader.get_features(self._phase)}
-
-    @property
-    def num_examples(self):
-        return self._reader.get_num_examples(phase=self._phase)
-

reader/match.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from paddlepalm.interface import reader
-from paddlepalm.reader.utils.reader4ernie import ClassifyReader
-
-def match(vocab_path, max_seq_len, do_lower_case=True, phase, dev_count=1):
-    config={
-        xxx}
-
-    return Reader(config())
-
-class Reader(reader):
-    
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
-        """
-        Args:
-            phase: train, eval, pred
-            """
-
-        self._is_training = phase == 'train'
-
-        reader = ClassifyReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', True),
-            for_cn=config.get('for_cn', False),
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
-
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
-        if phase == 'train':
-            self._input_file = config['train_file']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        elif phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-
-        self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 1)
-
-
-    @property
-    def outputs_attr(self):
-        if self._is_training:
-            return {"token_ids": [[-1, -1, 1], 'int64'],
-                    "position_ids": [[-1, -1, 1], 'int64'],
-                    "segment_ids": [[-1, -1, 1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32'],
-                    "label_ids": [[-1,1], 'int64'],
-                    "task_ids": [[-1, -1, 1], 'int64']
-                    }
-        else:
-            return {"token_ids": [[-1, -1, 1], 'int64'],
-                    "position_ids": [[-1, -1, 1], 'int64'],
-                    "segment_ids": [[-1, -1, 1], 'int64'],
-                    "task_ids": [[-1, -1, 1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32']
-                    }
-
-
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
-
-    def iterator(self): 
-
-        def list_to_dict(x):
-            names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 
-                'label_ids', 'unique_ids']
-            outputs = {n: i for n,i in zip(names, x)}
-            del outputs['unique_ids']
-            if not self._is_training:
-                del outputs['label_ids']
-            return outputs
-
-        for batch in self._data_generator():
-            yield list_to_dict(batch)
-
-    @property
-    def num_examples(self):
-        return self._reader.get_num_examples(phase=self._phase)
-

reader/mrc.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from paddlepalm.interface import reader
-from paddlepalm.reader.utils.reader4ernie import MRCReader
-
-class Reader(reader):
-    
-    def __init__(self, config, phase='train', dev_count=1, print_prefix=''):
-        """
-        Args:
-            phase: train, eval, pred
-            """
-
-        self._is_training = phase == 'train'
-
-        reader = MRCReader(config['vocab_path'],
-            max_seq_len=config['max_seq_len'],
-            do_lower_case=config.get('do_lower_case', False),
-            tokenizer='FullTokenizer',
-            for_cn=config.get('for_cn', False),
-            doc_stride=config['doc_stride'],
-            max_query_length=config['max_query_len'],
-            random_seed=config.get('seed', None))
-        self._reader = reader
-        self._dev_count = dev_count
-
-        self._batch_size = config['batch_size']
-        self._max_seq_len = config['max_seq_len']
-        if phase == 'train':
-            self._input_file = config['train_file']
-            # self._num_epochs = config['num_epochs']
-            self._num_epochs = None # 防止iteartor终止
-            self._shuffle = config.get('shuffle', True)
-            self._shuffle_buffer = config.get('shuffle_buffer', 5000)
-        if phase == 'eval':
-            self._input_file = config['dev_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-        elif phase == 'pred':
-            self._input_file = config['pred_file']
-            self._num_epochs = 1
-            self._shuffle = False
-            self._batch_size = config.get('pred_batch_size', self._batch_size)
-
-        self._phase = phase
-        # self._batch_size = 
-        self._print_first_n = config.get('print_first_n', 1)
-
-        # TODO: without slide window version
-        self._with_slide_window = config.get('with_slide_window', False)
-
-
-    @property
-    def outputs_attr(self):
-        if self._is_training:
-            return {"token_ids": [[-1, -1, 1], 'int64'],
-                    "position_ids": [[-1, -1, 1], 'int64'],
-                    "segment_ids": [[-1, -1, 1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32'],
-                    "start_positions": [[-1, 1], 'int64'],
-                    "end_positions": [[-1, 1], 'int64'],
-                    "task_ids": [[-1, -1, 1], 'int64']
-                    }
-        else:
-            return {"token_ids": [[-1, -1, 1], 'int64'],
-                    "position_ids": [[-1, -1, 1], 'int64'],
-                    "segment_ids": [[-1, -1, 1], 'int64'],
-                    "task_ids": [[-1, -1, 1], 'int64'],
-                    "input_mask": [[-1, -1, 1], 'float32'],
-                    "unique_ids": [[-1, 1], 'int64']
-                    }
-
-    @property
-    def epoch_outputs_attr(self):
-        if not self._is_training:
-            return {"examples": None,
-                    "features": None}
-
-    def load_data(self):
-        self._data_generator = self._reader.data_generator(self._input_file, self._batch_size, self._num_epochs, dev_count=self._dev_count, shuffle=self._shuffle, phase=self._phase)
-
-    def iterator(self): 
-
-        def list_to_dict(x):
-            names = ['token_ids', 'segment_ids', 'position_ids', 'task_ids', 'input_mask', 
-                'start_positions', 'end_positions', 'unique_ids']
-            outputs = {n: i for n,i in zip(names, x)}
-            if self._is_training:
-                del outputs['unique_ids']
-            else:
-                del outputs['start_positions']
-                del outputs['end_positions']
-            return outputs
-
-        for batch in self._data_generator():
-            yield list_to_dict(batch)
-
-    def get_epoch_outputs(self):
-        return {'examples': self._reader.get_examples(self._phase),
-                'features': self._reader.get_features(self._phase)}
-
-    @property
-    def num_examples(self):
-        return self._reader.get_num_examples(phase=self._phase)
-

reader/utils/batching4bert.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Mask, padding and batching."""
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-import numpy as np
-
-
-def mask(batch_tokens, total_token_num, vocab_size, CLS=1, SEP=2, MASK=3):
-    """
-    Add mask for batch_tokens, return out, mask_label, mask_pos;
-    Note: mask_pos responding the batch_tokens after padded;
-    """
-    max_len = max([len(sent) for sent in batch_tokens])
-    mask_label = []
-    mask_pos = []
-    prob_mask = np.random.rand(total_token_num)
-    # Note: the first token is [CLS], so [low=1]
-    replace_ids = np.random.randint(1, high=vocab_size, size=total_token_num)
-    pre_sent_len = 0
-    prob_index = 0
-    for sent_index, sent in enumerate(batch_tokens):
-        mask_flag = False
-        prob_index += pre_sent_len
-        for token_index, token in enumerate(sent):
-            prob = prob_mask[prob_index + token_index]
-            if prob > 0.15:
-                continue
-            elif 0.03 < prob <= 0.15:
-                # mask
-                if token != SEP and token != CLS:
-                    mask_label.append(sent[token_index])
-                    sent[token_index] = MASK
-                    mask_flag = True
-                    mask_pos.append(sent_index * max_len + token_index)
-            elif 0.015 < prob <= 0.03:
-                # random replace
-                if token != SEP and token != CLS:
-                    mask_label.append(sent[token_index])
-                    sent[token_index] = replace_ids[prob_index + token_index]
-                    mask_flag = True
-                    mask_pos.append(sent_index * max_len + token_index)
-            else:
-                # keep the original token
-                if token != SEP and token != CLS:
-                    mask_label.append(sent[token_index])
-                    mask_pos.append(sent_index * max_len + token_index)
-        pre_sent_len = len(sent)
-        # ensure at least mask one word in a sentence
-        while not mask_flag:
-            token_index = int(np.random.randint(1, high=len(sent) - 1, size=1))
-            if sent[token_index] != SEP and sent[token_index] != CLS:
-                mask_label.append(sent[token_index])
-                sent[token_index] = MASK
-                mask_flag = True
-                mask_pos.append(sent_index * max_len + token_index)
-    mask_label = np.array(mask_label).astype("int64").reshape([-1, 1])
-    mask_pos = np.array(mask_pos).astype("int64").reshape([-1, 1])
-    return batch_tokens, mask_label, mask_pos
-
-
-def prepare_batch_data(insts,
-                       total_token_num,
-                       max_len=None,
-                       voc_size=0,
-                       pad_id=None,
-                       cls_id=None,
-                       sep_id=None,
-                       mask_id=None,
-                       return_input_mask=True,
-                       return_max_len=True,
-                       return_num_token=False):
-    """
-    1. generate Tensor of data
-    2. generate Tensor of position
-    3. generate self attention mask, [shape: batch_size *  max_len * max_len]
-    """
-    batch_src_ids = [inst[0] for inst in insts]
-    batch_sent_ids = [inst[1] for inst in insts]
-    batch_pos_ids = [inst[2] for inst in insts]
-    labels_list = []
-    # compatible with mrqa, whose example includes start/end positions, 
-    # or unique id
-    for i in range(3, len(insts[0]), 1):
-        labels = [inst[i] for inst in insts]
-        labels = np.array(labels).astype("int64").reshape([-1, 1])
-        labels_list.append(labels)
-    # First step: do mask without padding
-    if mask_id >= 0:
-        out, mask_label, mask_pos = mask(
-            batch_src_ids,
-            total_token_num,
-            vocab_size=voc_size,
-            CLS=cls_id,
-            SEP=sep_id,
-            MASK=mask_id)
-    else:
-        out = batch_src_ids
-    # Second step: padding
-    src_id, self_input_mask = pad_batch_data(
-        out, 
-        max_len=max_len,
-        pad_idx=pad_id, return_input_mask=True)
-    pos_id = pad_batch_data(
-        batch_pos_ids,
-        max_len=max_len,
-        pad_idx=pad_id,
-        return_pos=False,
-        return_input_mask=False)
-    sent_id = pad_batch_data(
-        batch_sent_ids,
-        max_len=max_len,
-        pad_idx=pad_id,
-        return_pos=False,
-        return_input_mask=False)
-    if mask_id >= 0:
-        return_list = [
-            src_id, pos_id, sent_id, self_input_mask, mask_label, mask_pos
-        ] + labels_list
-    else:
-        return_list = [src_id, pos_id, sent_id, self_input_mask] + labels_list
-    return return_list if len(return_list) > 1 else return_list[0]
-
-
-def pad_batch_data(insts,
-                   max_len=None,
-                   pad_idx=0,
-                   return_pos=False,
-                   return_input_mask=False,
-                   return_max_len=False,
-                   return_num_token=False):
-    """
-    Pad the instances to the max sequence length in batch, and generate the
-    corresponding position data and input mask.
-    """
-    return_list = []
-    if max_len is None:
-        max_len = max(len(inst) for inst in insts)
-    # Any token included in dict can be used to pad, since the paddings' loss
-    # will be masked out by weights and make no effect on parameter gradients.
-    inst_data = np.array([
-        list(inst) + list([pad_idx] * (max_len - len(inst))) for inst in insts
-    ])
-    return_list += [inst_data.astype("int64").reshape([-1, max_len, 1])]
-    # position data
-    if return_pos:
-        inst_pos = np.array([
-            list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))
-            for inst in insts
-        ])
-        return_list += [inst_pos.astype("int64").reshape([-1, max_len, 1])]
-    if return_input_mask:
-        # This is used to avoid attention on paddings.
-        input_mask_data = np.array([[1] * len(inst) + [0] *
-                                    (max_len - len(inst)) for inst in insts])
-        input_mask_data = np.expand_dims(input_mask_data, axis=-1)
-        return_list += [input_mask_data.astype("float32")]
-    if return_max_len:
-        return_list += [max_len]
-    if return_num_token:
-        num_token = 0
-        for inst in insts:
-            num_token += len(inst)
-        return_list += [num_token]
-    return return_list if len(return_list) > 1 else return_list[0]
-
-
-if __name__ == "__main__":
-    pass
-
-

reader/utils/batching4ernie.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Mask, padding and batching."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import numpy as np
-
-from six.moves import xrange
-
-
-def mask(batch_tokens,
-         seg_labels,
-         mask_word_tags,
-         total_token_num,
-         vocab_size,
-         CLS=1,
-         SEP=2,
-         MASK=3):
-    """
-    Add mask for batch_tokens, return out, mask_label, mask_pos;
-    Note: mask_pos responding the batch_tokens after padded;
-    """
-    max_len = max([len(sent) for sent in batch_tokens])
-    mask_label = []
-    mask_pos = []
-    prob_mask = np.random.rand(total_token_num)
-    # Note: the first token is [CLS], so [low=1]
-    replace_ids = np.random.randint(1, high=vocab_size, size=total_token_num)
-    pre_sent_len = 0
-    prob_index = 0
-    for sent_index, sent in enumerate(batch_tokens):
-        mask_flag = False
-        mask_word = mask_word_tags[sent_index]
-        prob_index += pre_sent_len
-        if mask_word:
-            beg = 0
-            for token_index, token in enumerate(sent):
-                seg_label = seg_labels[sent_index][token_index]
-                if seg_label == 1:
-                    continue
-                if beg == 0:
-                    if seg_label != -1:
-                        beg = token_index
-                    continue
-
-                prob = prob_mask[prob_index + beg]
-                if prob > 0.15:
-                    pass
-                else:
-                    for index in xrange(beg, token_index):
-                        prob = prob_mask[prob_index + index]
-                        base_prob = 1.0
-                        if index == beg:
-                            base_prob = 0.15
-                        if base_prob * 0.2 < prob <= base_prob:
-                            mask_label.append(sent[index])
-                            sent[index] = MASK
-                            mask_flag = True
-                            mask_pos.append(sent_index * max_len + index)
-                        elif base_prob * 0.1 < prob <= base_prob * 0.2:
-                            mask_label.append(sent[index])
-                            sent[index] = replace_ids[prob_index + index]
-                            mask_flag = True
-                            mask_pos.append(sent_index * max_len + index)
-                        else:
-                            mask_label.append(sent[index])
-                            mask_pos.append(sent_index * max_len + index)
-
-                if seg_label == -1:
-                    beg = 0
-                else:
-                    beg = token_index
-        else:
-            for token_index, token in enumerate(sent):
-                prob = prob_mask[prob_index + token_index]
-                if prob > 0.15:
-                    continue
-                elif 0.03 < prob <= 0.15:
-                    # mask
-                    if token != SEP and token != CLS:
-                        mask_label.append(sent[token_index])
-                        sent[token_index] = MASK
-                        mask_flag = True
-                        mask_pos.append(sent_index * max_len + token_index)
-                elif 0.015 < prob <= 0.03:
-                    # random replace
-                    if token != SEP and token != CLS:
-                        mask_label.append(sent[token_index])
-                        sent[token_index] = replace_ids[prob_index +
-                                                        token_index]
-                        mask_flag = True
-                        mask_pos.append(sent_index * max_len + token_index)
-                else:
-                    # keep the original token
-                    if token != SEP and token != CLS:
-                        mask_label.append(sent[token_index])
-                        mask_pos.append(sent_index * max_len + token_index)
-
-        pre_sent_len = len(sent)
-
-    mask_label = np.array(mask_label).astype("int64").reshape([-1, 1])
-    mask_pos = np.array(mask_pos).astype("int64").reshape([-1, 1])
-    return batch_tokens, mask_label, mask_pos
-
-
-def pad_batch_data(insts,
-                   pad_idx=0,
-                   return_pos=False,
-                   return_input_mask=False,
-                   return_max_len=False,
-                   return_num_token=False,
-                   return_seq_lens=False):
-    """
-    Pad the instances to the max sequence length in batch, and generate the
-    corresponding position data and attention bias.
-    """
-    return_list = []
-    max_len = max(len(inst) for inst in insts)
-    # Any token included in dict can be used to pad, since the paddings' loss
-    # will be masked out by weights and make no effect on parameter gradients.
-
-    inst_data = np.array(
-        [inst + list([pad_idx] * (max_len - len(inst))) for inst in insts])
-    return_list += [inst_data.astype("int64").reshape([-1, max_len, 1])]
-
-    # position data
-    if return_pos:
-        inst_pos = np.array([
-            list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))
-            for inst in insts
-        ])
-
-        return_list += [inst_pos.astype("int64").reshape([-1, max_len, 1])]
-
-    if return_input_mask:
-        # This is used to avoid attention on paddings.
-        input_mask_data = np.array([[1] * len(inst) + [0] *
-                                    (max_len - len(inst)) for inst in insts])
-        input_mask_data = np.expand_dims(input_mask_data, axis=-1)
-        return_list += [input_mask_data.astype("float32")]
-
-    if return_max_len:
-        return_list += [max_len]
-
-    if return_num_token:
-        num_token = 0
-        for inst in insts:
-            num_token += len(inst)
-        return_list += [num_token]
-
-    if return_seq_lens:
-        seq_lens = np.array([len(inst) for inst in insts])
-        return_list += [seq_lens.astype("int64").reshape([-1, 1])]
-
-    return return_list if len(return_list) > 1 else return_list[0]
-
-
-if __name__ == "__main__":
-
-    pass

reader/utils/mlm_batching.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Mask, padding and batching."""
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-import numpy as np
-
-
-def mask(batch_tokens, total_token_num, vocab_size, CLS=1, SEP=2, MASK=3):
-    """
-    Add mask for batch_tokens, return out, mask_label, mask_pos;
-    Note: mask_pos responding the batch_tokens after padded;
-    """
-    max_len = max([len(sent) for sent in batch_tokens])
-    mask_label = []
-    mask_pos = []
-    prob_mask = np.random.rand(total_token_num)
-    # Note: the first token is [CLS], so [low=1]
-    replace_ids = np.random.randint(1, high=vocab_size, size=total_token_num)
-    pre_sent_len = 0
-    prob_index = 0
-    for sent_index, sent in enumerate(batch_tokens):
-        mask_flag = False
-        prob_index += pre_sent_len
-        for token_index, token in enumerate(sent):
-            prob = prob_mask[prob_index + token_index]
-            if prob > 0.15:
-                continue
-            elif 0.03 < prob <= 0.15:
-                # mask
-                if token != SEP and token != CLS:
-                    mask_label.append(sent[token_index])
-                    sent[token_index] = MASK
-                    mask_flag = True
-                    mask_pos.append(sent_index * max_len + token_index)
-            elif 0.015 < prob <= 0.03:
-                # random replace
-                if token != SEP and token != CLS:
-                    mask_label.append(sent[token_index])
-                    sent[token_index] = replace_ids[prob_index + token_index]
-                    mask_flag = True
-                    mask_pos.append(sent_index * max_len + token_index)
-            else:
-                # keep the original token
-                if token != SEP and token != CLS:
-                    mask_label.append(sent[token_index])
-                    mask_pos.append(sent_index * max_len + token_index)
-        pre_sent_len = len(sent)
-        # ensure at least mask one word in a sentence
-        while not mask_flag:
-            token_index = int(np.random.randint(1, high=len(sent) - 1, size=1))
-            if sent[token_index] != SEP and sent[token_index] != CLS:
-                mask_label.append(sent[token_index])
-                sent[token_index] = MASK
-                mask_flag = True
-                mask_pos.append(sent_index * max_len + token_index)
-    mask_label = np.array(mask_label).astype("int64").reshape([-1, 1])
-    mask_pos = np.array(mask_pos).astype("int64").reshape([-1, 1])
-    return batch_tokens, mask_label, mask_pos
-
-
-def prepare_batch_data(insts,
-                       total_token_num,
-                       max_len=None,
-                       voc_size=0,
-                       pad_id=None,
-                       cls_id=None,
-                       sep_id=None,
-                       mask_id=None,
-                       task_id=0,
-                       return_input_mask=True,
-                       return_max_len=True,
-                       return_num_token=False):
-    """
-    1. generate Tensor of data
-    2. generate Tensor of position
-    3. generate self attention mask, [shape: batch_size *  max_len * max_len]
-    """
-    batch_src_ids = [inst[0] for inst in insts]
-    batch_sent_ids = [inst[1] for inst in insts]
-    batch_pos_ids = [inst[2] for inst in insts]
-
-    # 这里是否应该反过来？？？否则在task layer里展开后的word embedding是padding后的，这时候word的index是跟没有padding时的index对不上的？
-    # First step: do mask without padding
-    out, mask_label, mask_pos = mask(
-        batch_src_ids,
-        total_token_num,
-        vocab_size=voc_size,
-        CLS=cls_id,
-        SEP=sep_id,
-        MASK=mask_id)
-    # Second step: padding
-    src_id, self_input_mask = pad_batch_data(
-        out, 
-        max_len=max_len,
-        pad_idx=pad_id, return_input_mask=True)
-
-    pos_id = pad_batch_data(
-        batch_pos_ids,
-        max_len=max_len,
-        pad_idx=pad_id,
-        return_pos=False,
-        return_input_mask=False)
-    sent_id = pad_batch_data(
-        batch_sent_ids,
-        max_len=max_len,
-        pad_idx=pad_id,
-        return_pos=False,
-        return_input_mask=False)
-    task_ids = np.ones_like(
-        src_id, dtype="int64") * task_id
-    return_list = [
-        src_id, pos_id, sent_id, self_input_mask, task_ids, mask_label, mask_pos
-    ]
-    return return_list if len(return_list) > 1 else return_list[0]
-
-
-def pad_batch_data(insts,
-                   max_len=None,
-                   pad_idx=0,
-                   return_pos=False,
-                   return_input_mask=False,
-                   return_max_len=False,
-                   return_num_token=False):
-    """
-    Pad the instances to the max sequence length in batch, and generate the
-    corresponding position data and input mask.
-    """
-    return_list = []
-    if max_len is None:
-        max_len = max(len(inst) for inst in insts)
-    # Any token included in dict can be used to pad, since the paddings' loss
-    # will be masked out by weights and make no effect on parameter gradients.
-    inst_data = np.array([
-        list(inst) + list([pad_idx] * (max_len - len(inst))) for inst in insts
-    ])
-    return_list += [inst_data.astype("int64").reshape([-1, max_len, 1])]
-    # position data
-    if return_pos:
-        inst_pos = np.array([
-            list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))
-            for inst in insts
-        ])
-        return_list += [inst_pos.astype("int64").reshape([-1, max_len, 1])]
-    if return_input_mask:
-        # This is used to avoid attention on paddings.
-        input_mask_data = np.array([[1] * len(inst) + [0] *
-                                    (max_len - len(inst)) for inst in insts])
-        input_mask_data = np.expand_dims(input_mask_data, axis=-1)
-        return_list += [input_mask_data.astype("float32")]
-    if return_max_len:
-        return_list += [max_len]
-    if return_num_token:
-        num_token = 0
-        for inst in insts:
-            num_token += len(inst)
-        return_list += [num_token]
-    return return_list if len(return_list) > 1 else return_list[0]
-
-
-if __name__ == "__main__":
-    pass
-
-

reader/utils/mrqa_helper.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-class MRQAExample(object):
-    """A single training/test example for simple sequence classification.
-
-     For examples without an answer, the start and end position are -1.
-  """
-
-    def __init__(self,
-                 qas_id,
-                 question_text,
-                 doc_tokens,
-                 orig_answer_text=None,
-                 start_position=None,
-                 end_position=None,
-                 is_impossible=False):
-        self.qas_id = qas_id
-        self.question_text = question_text
-        self.doc_tokens = doc_tokens
-        self.orig_answer_text = orig_answer_text
-        self.start_position = start_position
-        self.end_position = end_position
-        self.is_impossible = is_impossible
-
-    def __str__(self):
-        return self.__repr__()
-
-    def __repr__(self):
-        s = ""
-        s += "qas_id: %s" % (tokenization.printable_text(self.qas_id))
-        s += ", question_text: %s" % (
-            tokenization.printable_text(self.question_text))
-        s += ", doc_tokens: [%s]" % (" ".join(self.doc_tokens))
-        if self.start_position:
-            s += ", start_position: %d" % (self.start_position)
-        if self.start_position:
-            s += ", end_position: %d" % (self.end_position)
-        if self.start_position:
-            s += ", is_impossible: %r" % (self.is_impossible)
-        return s
-
-
-class MRQAFeature(object):
-    """A single set of features of data."""
-
-    def __init__(self,
-                 unique_id,
-                 example_index,
-                 doc_span_index,
-                 tokens,
-                 token_to_orig_map,
-                 token_is_max_context,
-                 input_ids,
-                 input_mask,
-                 segment_ids,
-                 start_position=None,
-                 end_position=None,
-                 is_impossible=None):
-        self.unique_id = unique_id
-        self.example_index = example_index
-        self.doc_span_index = doc_span_index
-        self.tokens = tokens
-        self.token_to_orig_map = token_to_orig_map
-        self.token_is_max_context = token_is_max_context
-        self.input_ids = input_ids
-        self.input_mask = input_mask
-        self.segment_ids = segment_ids
-        self.start_position = start_position
-        self.end_position = end_position
-        self.is_impossible = is_impossible
-

reader/utils/reader4ernie.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-from __future__ import absolute_import
-
-import sys
-import os
-import json
-import random
-import logging
-import numpy as np
-import six
-from io import open
-from collections import namedtuple
-
-import paddlepalm.tokenizer.ernie_tokenizer as tokenization
-from paddlepalm.reader.utils.batching4ernie import pad_batch_data
-from paddlepalm.reader.utils.mlm_batching import prepare_batch_data
-
-
-log = logging.getLogger(__name__)
-
-if six.PY3:
-    import io
-    sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
-    sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8')
-
-
-def csv_reader(fd, delimiter='\t'):
-    def gen():
-        for i in fd:
-            yield i.rstrip('\n').split(delimiter)
-    return gen()
-
-
-class BaseReader(object):
-    def __init__(self,
-                 vocab_path,
-                 label_map_config=None,
-                 max_seq_len=512,
-                 do_lower_case=True,
-                 in_tokens=False,
-                 is_inference=False,
-                 random_seed=None,
-                 tokenizer="FullTokenizer",
-                 is_classify=True,
-                 is_regression=False,
-                 for_cn=True,
-                 task_id=0):
-        self.max_seq_len = max_seq_len
-        self.tokenizer = tokenization.FullTokenizer(
-            vocab_file=vocab_path, do_lower_case=do_lower_case)
-        self.vocab = self.tokenizer.vocab
-        self.pad_id = self.vocab["[PAD]"]
-        self.cls_id = self.vocab["[CLS]"]
-        self.sep_id = self.vocab["[SEP]"]
-        self.mask_id = self.vocab["[MASK]"]
-        self.in_tokens = in_tokens
-        self.is_inference = is_inference
-        self.for_cn = for_cn
-        self.task_id = task_id
-
-        np.random.seed(random_seed)
-
-        self.is_classify = is_classify
-        self.is_regression = is_regression
-        self.current_example = 0
-        self.current_epoch = 0
-        self.num_examples = 0
-
-        self.examples = {}
-
-        if label_map_config:
-            with open(label_map_config, encoding='utf8') as f: 
-                self.label_map = json.load(f)
-        else:
-            self.label_map = None
-
-    def get_train_progress(self):
-        """Gets progress for training phase."""
-        return self.current_example, self.current_epoch
-
-    def _read_tsv(self, input_file, quotechar=None):
-        """Reads a tab separated value file."""
-        with open(input_file, 'r', encoding='utf8') as f:
-            reader = csv_reader(f)
-            headers = next(reader)
-            Example = namedtuple('Example', headers)
-
-            examples = []
-            for line in reader:
-                example = Example(*line)
-                examples.append(example)
-            return examples
-
-    def _truncate_seq_pair(self, tokens_a, tokens_b, max_length):
-        """Truncates a sequence pair in place to the maximum length."""
-
-        # This is a simple heuristic which will always truncate the longer sequence
-        # one token at a time. This makes more sense than truncating an equal percent
-        # of tokens from each, since if one sequence is very short then each token
-        # that's truncated likely contains more information than a longer sequence.
-        while True:
-            total_length = len(tokens_a) + len(tokens_b)
-            if total_length <= max_length:
-                break
-            if len(tokens_a) > len(tokens_b):
-                tokens_a.pop()
-            else:
-                tokens_b.pop()
-
-    def _convert_example_to_record(self, example, max_seq_length, tokenizer):
-        """Converts a single `Example` into a single `Record`."""
-
-        text_a = tokenization.convert_to_unicode(example.text_a)
-        tokens_a = tokenizer.tokenize(text_a)
-        tokens_b = None
-
-        has_text_b = False
-        if isinstance(example, dict):
-            has_text_b = "text_b" in example.keys()
-        else:
-            has_text_b = "text_b" in example._fields
-
-        if has_text_b:
-            text_b = tokenization.convert_to_unicode(example.text_b)
-            tokens_b = tokenizer.tokenize(text_b)
-
-        if tokens_b:
-            # Modifies `tokens_a` and `tokens_b` in place so that the total
-            # length is less than the specified length.
-            # Account for [CLS], [SEP], [SEP] with "- 3"
-            self._truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
-        else:
-            # Account for [CLS] and [SEP] with "- 2"
-            if len(tokens_a) > max_seq_length - 2:
-                tokens_a = tokens_a[0:(max_seq_length - 2)]
-
-        # The convention in BERT/ERNIE is:
-        # (a) For sequence pairs:
-        #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
-        #  type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1
-        # (b) For single sequences:
-        #  tokens:   [CLS] the dog is hairy . [SEP]
-        #  type_ids: 0     0   0   0  0     0 0
-        #
-        # Where "type_ids" are used to indicate whether this is the first
-        # sequence or the second sequence. The embedding vectors for `type=0` and
-        # `type=1` were learned during pre-training and are added to the wordpiece
-        # embedding vector (and position vector). This is not *strictly* necessary
-        # since the [SEP] token unambiguously separates the sequences, but it makes
-        # it easier for the model to learn the concept of sequences.
-        #
-        # For classification tasks, the first vector (corresponding to [CLS]) is
-        # used as as the "sentence vector". Note that this only makes sense because
-        # the entire model is fine-tuned.
-        tokens = []
-        text_type_ids = []
-        tokens.append("[CLS]")
-        text_type_ids.append(0)
-        for token in tokens_a:
-            tokens.append(token)
-            text_type_ids.append(0)
-        tokens.append("[SEP]")
-        text_type_ids.append(0)
-
-        if tokens_b:
-            for token in tokens_b:
-                tokens.append(token)
-                text_type_ids.append(1)
-            tokens.append("[SEP]")
-            text_type_ids.append(1)
-
-        token_ids = tokenizer.convert_tokens_to_ids(tokens)
-        position_ids = list(range(len(token_ids)))
-
-        if self.is_inference:
-            Record = namedtuple('Record',
-                                ['token_ids', 'text_type_ids', 'position_ids'])
-            record = Record(
-                token_ids=token_ids,
-                text_type_ids=text_type_ids,
-                position_ids=position_ids)
-        else:
-            if self.label_map:
-                label_id = self.label_map[example.label]
-            else:
-                label_id = example.label
-
-            Record = namedtuple('Record', [
-                'token_ids', 'text_type_ids', 'position_ids', 'label_id', 'qid'
-            ])
-
-            qid = None
-            if "qid" in example._fields:
-                qid = example.qid
-
-            record = Record(
-                token_ids=token_ids,
-                text_type_ids=text_type_ids,
-                position_ids=position_ids,
-                label_id=label_id,
-                qid=qid)
-        return record
-
-    def _prepare_batch_data(self, examples, batch_size, phase=None):
-        """generate batch records"""
-        batch_records, max_len = [], 0
-        if len(examples) < batch_size:
-            raise Exception('CLS dataset contains too few samples. Expect more than '+str(batch_size))
-        for index, example in enumerate(examples):
-            if phase == "train":
-                self.current_example = index
-            record = self._convert_example_to_record(example, self.max_seq_len,
-                                                     self.tokenizer)
-            max_len = max(max_len, len(record.token_ids))
-            if self.in_tokens:
-                to_append = (len(batch_records) + 1) * max_len <= batch_size
-            else:
-                to_append = len(batch_records) < batch_size
-            if to_append:
-                batch_records.append(record)
-            else:
-                yield self._pad_batch_records(batch_records)
-                batch_records, max_len = [record], len(record.token_ids)
-
-        if phase == 'pred' and batch_records:
-            yield self._pad_batch_records(batch_records)
-
-    def get_num_examples(self, input_file=None, phase=None):
-        if self.examples is not None:
-            if phase is None:
-                phase = 'all'
-            return len(self.examples[phase])
-        else:
-            assert input_file is not None, "Argument input_file should be given or the data_generator should be created when this func is called."
-            examples = self._read_tsv(input_file)
-            return len(examples)
-
-    def data_generator(self,
-                       input_file,
-                       batch_size,
-                       epoch,
-                       dev_count=1,
-                       shuffle=True,
-                       phase=None):
-        examples = self._read_tsv(input_file)
-        if phase is None:
-            phase = 'all'
-        self.examples[phase] = examples
-
-        def wrapper():
-            all_dev_batches = []
-            if epoch is None:
-                num_epochs = 99999999
-            else:
-                num_epochs = epoch
-            for epoch_index in range(num_epochs):
-                if phase == "train":
-                    self.current_example = 0
-                    self.current_epoch = epoch_index
-                if shuffle:
-                    np.random.shuffle(examples)
-
-                for batch_data in self._prepare_batch_data(
-                        examples, batch_size, phase=phase):
-                    if len(all_dev_batches) < dev_count:
-                        all_dev_batches.append(batch_data)
-                    if len(all_dev_batches) == dev_count:
-                        for batch in all_dev_batches:
-                            yield batch
-                        all_dev_batches = []
-        def f():
-            for i in wrapper():
-                yield i
-
-        # def f():
-        #     try:
-        #         for i in wrapper():
-        #             yield i
-        #     except Exception as e:
-        #         import traceback
-        #         traceback.print_exc()
-
-        return f
-
-
-class MaskLMReader(BaseReader):
-
-    def _convert_example_to_record(self, example, max_seq_length, tokenizer):
-        """Converts a single `Example` into a single `Record`."""
-
-        text_a = tokenization.convert_to_unicode(example.text_a)
-        tokens_a = tokenizer.tokenize(text_a)
-        tokens_b = None 
-
-        has_text_b = False
-        if isinstance(example, dict):
-            has_text_b = "text_b" in example.keys()
-        else:
-            has_text_b = "text_b" in example._fields
-
-        if has_text_b:
-            text_b = tokenization.convert_to_unicode(example.text_b)
-            tokens_b = tokenizer.tokenize(text_b)
-
-        if tokens_b:
-            # Modifies `tokens_a` and `tokens_b` in place so that the total
-            # length is less than the specified length.
-            # Account for [CLS], [SEP], [SEP] with "- 3"
-            self._truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
-        else:
-            # Account for [CLS] and [SEP] with "- 2"
-            if len(tokens_a) > max_seq_length - 2:
-                tokens_a = tokens_a[0:(max_seq_length - 2)]
-
-        # The convention in BERT/ERNIE is:
-        # (a) For sequence pairs:
-        #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
-        #  type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1
-        # (b) For single sequences:
-        #  tokens:   [CLS] the dog is hairy . [SEP]
-        #  type_ids: 0     0   0   0  0     0 0
-        #
-        # Where "type_ids" are used to indicate whether this is the first
-        # sequence or the second sequence. The embedding vectors for `type=0` and
-        # `type=1` were learned during pre-training and are added to the wordpiece
-        # embedding vector (and position vector). This is not *strictly* necessary
-        # since the [SEP] token unambiguously separates the sequences, but it makes
-        # it easier for the model to learn the concept of sequences.
-        #
-        # For classification tasks, the first vector (corresponding to [CLS]) is
-        # used as as the "sentence vector". Note that this only makes sense because
-        # the entire model is fine-tuned.
-        tokens = []
-        text_type_ids = []
-        tokens.append("[CLS]")
-        text_type_ids.append(0)
-        for token in tokens_a:
-            tokens.append(token)
-            text_type_ids.append(0)
-        tokens.append("[SEP]")
-        text_type_ids.append(0)
-
-        if tokens_b:
-            for token in tokens_b:
-                tokens.append(token)
-                text_type_ids.append(1)
-            tokens.append("[SEP]")
-            text_type_ids.append(1)
-
-        token_ids = tokenizer.convert_tokens_to_ids(tokens)
-        position_ids = list(range(len(token_ids)))
-
-        # Record = namedtuple('Record',
-        #                     ['token_ids', 'text_type_ids', 'position_ids'])
-        # record = Record(
-        #     token_ids=token_ids,
-        #     text_type_ids=text_type_ids,
-        #     position_ids=position_ids)
-
-        return [token_ids, text_type_ids, position_ids]
-
-    def batch_reader(self, examples, batch_size, in_tokens, phase):
-        batch = []
-        total_token_num = 0
-        if len(examples) < batch_size:
-            raise Exception('MaskLM dataset contains too few samples. Expect more than '+str(batch_size))
-        for e in examples:
-            parsed_line = self._convert_example_to_record(e, self.max_seq_len, self.tokenizer)
-            to_append = len(batch) < batch_size
-            if to_append:
-                batch.append(parsed_line)
-                total_token_num += len(parsed_line[0])
-            else:
-                yield batch, total_token_num
-                batch = [parsed_line]
-                total_token_num = len(parsed_line[0])
-
-        if len(batch) > 0 and phase == 'pred':
-            yield batch, total_token_num
-
-    def data_generator(self,
-                       input_file,
-                       batch_size,
-                       epoch,
-                       dev_count=1,
-                       shuffle=True,
-                       phase=None):
-        examples = self._read_tsv(input_file)
-        if phase is None:
-            phase = 'all'
-        self.examples[phase] = examples
-
-        def wrapper():
-            all_dev_batches = []
-            if epoch is None:
-                num_epochs = 99999999
-            else:
-                num_epochs = epoch
-            for epoch_index in range(num_epochs):
-                if phase == "train":
-                    self.current_example = 0
-                    self.current_epoch = epoch_index
-                if shuffle:
-                    np.random.shuffle(examples)
-
-                all_dev_batches = []
-                for batch_data, num_tokens in self.batch_reader(examples, 
-                                                    batch_size, self.in_tokens, phase=phase):
-                    batch_data = prepare_batch_data(
-                        batch_data,
-                        num_tokens,
-                        voc_size=len(self.vocab),
-                        pad_id=self.pad_id,
-                        cls_id=self.cls_id,
-                        sep_id=self.sep_id,
-                        mask_id=self.mask_id,
-                        # max_len=self.max_seq_len, # 注意，如果padding到最大长度，会导致mask_pos与实际位置不对应。因为mask pos是基于batch内最大长度来计算的。
-                        return_input_mask=True,
-                        return_max_len=False,
-                        return_num_token=False)
-
-                    if len(all_dev_batches) < dev_count:
-                        all_dev_batches.append(batch_data)
-                    if len(all_dev_batches) == dev_count:
-                        for batch in all_dev_batches:
-                            yield batch
-                        all_dev_batches = []
-
-        return wrapper
-
-
-class ClassifyReader(BaseReader):
-    def _read_tsv(self, input_file, quotechar=None):
-        """Reads a tab separated value file."""
-        with open(input_file, 'r', encoding='utf8') as f:
-            reader = csv_reader(f)
-            headers = next(reader)
-            text_indices = [
-                index for index, h in enumerate(headers) if h != "label"
-            ]
-            Example = namedtuple('Example', headers)
-
-            examples = []
-            for line in reader:
-                for index, text in enumerate(line):
-                    if index in text_indices:
-                        if self.for_cn:
-                            line[index] = text.replace(' ', '')
-                        else:
-                            line[index] = text
-                example = Example(*line)
-                examples.append(example)
-            return examples
-
-    def _pad_batch_records(self, batch_records):
-        batch_token_ids = [record.token_ids for record in batch_records]
-        batch_text_type_ids = [record.text_type_ids for record in batch_records]
-        batch_position_ids = [record.position_ids for record in batch_records]
-
-        if not self.is_inference:
-            batch_labels = [record.label_id for record in batch_records]
-            if self.is_classify:
-                batch_labels = np.array(batch_labels).astype("int64").reshape(
-                    [-1, 1])
-            elif self.is_regression:
-                batch_labels = np.array(batch_labels).astype("float32").reshape(
-                    [-1, 1])
-
-            if batch_records[0].qid:
-                batch_qids = [record.qid for record in batch_records]
-                batch_qids = np.array(batch_qids).astype("int64").reshape(
-                    [-1, 1])
-            else:
-                batch_qids = np.array([]).astype("int64").reshape([-1, 1])
-
-        # padding
-        padded_token_ids, input_mask = pad_batch_data(
-            batch_token_ids, pad_idx=self.pad_id, return_input_mask=True)
-        padded_text_type_ids = pad_batch_data(
-            batch_text_type_ids, pad_idx=self.pad_id)
-        padded_position_ids = pad_batch_data(
-            batch_position_ids, pad_idx=self.pad_id)
-        padded_task_ids = np.ones_like(
-            padded_token_ids, dtype="int64") * self.task_id
-
-        return_list = [
-            padded_token_ids, padded_text_type_ids, padded_position_ids,
-            padded_task_ids, input_mask
-        ]
-        if not self.is_inference:
-            return_list += [batch_labels, batch_qids]
-
-        return return_list
-
-
-class SequenceLabelReader(BaseReader):
-    def _pad_batch_records(self, batch_records):
-        batch_token_ids = [record.token_ids for record in batch_records]
-        batch_text_type_ids = [record.text_type_ids for record in batch_records]
-        batch_position_ids = [record.position_ids for record in batch_records]
-        batch_label_ids = [record.label_ids for record in batch_records]
-
-        # padding
-        padded_token_ids, input_mask, batch_seq_lens = pad_batch_data(
-            batch_token_ids,
-            pad_idx=self.pad_id,
-            return_input_mask=True,
-            return_seq_lens=True)
-        padded_text_type_ids = pad_batch_data(
-            batch_text_type_ids, pad_idx=self.pad_id)
-        padded_position_ids = pad_batch_data(
-            batch_position_ids, pad_idx=self.pad_id)
-        padded_label_ids = pad_batch_data(
-            batch_label_ids, pad_idx=len(self.label_map) - 1)
-        padded_task_ids = np.ones_like(
-            padded_token_ids, dtype="int64") * self.task_id
-
-        return_list = [
-            padded_token_ids, padded_text_type_ids, padded_position_ids,
-            padded_task_ids, input_mask, padded_label_ids, batch_seq_lens
-        ]
-        return return_list
-
-    def _reseg_token_label(self, tokens, labels, tokenizer):
-        assert len(tokens) == len(labels)
-        ret_tokens = []
-        ret_labels = []
-        for token, label in zip(tokens, labels):
-            sub_token = tokenizer.tokenize(token)
-            if len(sub_token) == 0:
-                continue
-            ret_tokens.extend(sub_token)
-            if len(sub_token) == 1:
-                ret_labels.append(label)
-                continue
-
-            if label == "O" or label.startswith("I-"):
-                ret_labels.extend([label] * len(sub_token))
-            elif label.startswith("B-"):
-                i_label = "I-" + label[2:]
-                ret_labels.extend([label] + [i_label] * (len(sub_token) - 1))
-            elif label.startswith("S-"):
-                b_laebl = "B-" + label[2:]
-                e_label = "E-" + label[2:]
-                i_label = "I-" + label[2:]
-                ret_labels.extend([b_laebl] + [i_label] * (len(sub_token) - 2) + [e_label])
-            elif label.startswith("E-"):
-                i_label = "I-" + label[2:]
-                ret_labels.extend([i_label] * (len(sub_token) - 1) + [label])
-
-        assert len(ret_tokens) == len(ret_labels)
-        return ret_tokens, ret_labels
-
-    def _convert_example_to_record(self, example, max_seq_length, tokenizer):
-        tokens = tokenization.convert_to_unicode(example.text_a).split(u"")
-        labels = tokenization.convert_to_unicode(example.label).split(u"")
-        tokens, labels = self._reseg_token_label(tokens, labels, tokenizer)
-
-        if len(tokens) > max_seq_length - 2:
-            tokens = tokens[0:(max_seq_length - 2)]
-            labels = labels[0:(max_seq_length - 2)]
-
-        tokens = ["[CLS]"] + tokens + ["[SEP]"]
-        token_ids = tokenizer.convert_tokens_to_ids(tokens)
-        position_ids = list(range(len(token_ids)))
-        text_type_ids = [0] * len(token_ids)
-        no_entity_id = len(self.label_map) - 1
-        label_ids = [no_entity_id] + [
-            self.label_map[label] for label in labels
-        ] + [no_entity_id]
-
-        Record = namedtuple(
-            'Record',
-            ['token_ids', 'text_type_ids', 'position_ids', 'label_ids'])
-        record = Record(
-            token_ids=token_ids,
-            text_type_ids=text_type_ids,
-            position_ids=position_ids,
-            label_ids=label_ids)
-        return record
-
-
-class ExtractEmbeddingReader(BaseReader):
-    def _pad_batch_records(self, batch_records):
-        batch_token_ids = [record.token_ids for record in batch_records]
-        batch_text_type_ids = [record.text_type_ids for record in batch_records]
-        batch_position_ids = [record.position_ids for record in batch_records]
-
-        # padding
-        padded_token_ids, input_mask, seq_lens = pad_batch_data(
-            batch_token_ids,
-            pad_idx=self.pad_id,
-            return_input_mask=True,
-            return_seq_lens=True)
-        padded_text_type_ids = pad_batch_data(
-            batch_text_type_ids, pad_idx=self.pad_id)
-        padded_position_ids = pad_batch_data(
-            batch_position_ids, pad_idx=self.pad_id)
-        padded_task_ids = np.ones_like(
-            padded_token_ids, dtype="int64") * self.task_id
-
-        return_list = [
-            padded_token_ids, padded_text_type_ids, padded_position_ids,
-            padded_task_ids, input_mask, seq_lens
-        ]
-
-        return return_list
-
-
-class MRCReader(BaseReader):
-    def __init__(self,
-                 vocab_path,
-                 label_map_config=None,
-                 max_seq_len=512,
-                 do_lower_case=True,
-                 in_tokens=False,
-                 random_seed=None,
-                 tokenizer="FullTokenizer",
-                 is_classify=True,
-                 is_regression=False,
-                 for_cn=True,
-                 task_id=0,
-                 doc_stride=128,
-                 max_query_length=64,
-                 remove_noanswer=True):
-        self.max_seq_len = max_seq_len
-        self.tokenizer = tokenization.FullTokenizer(
-            vocab_file=vocab_path, do_lower_case=do_lower_case)
-        self.vocab = self.tokenizer.vocab
-        self.pad_id = self.vocab["[PAD]"]
-        self.cls_id = self.vocab["[CLS]"]
-        self.sep_id = self.vocab["[SEP]"]
-        self.in_tokens = in_tokens
-        self.for_cn = for_cn
-        self.task_id = task_id
-        self.doc_stride = doc_stride
-        self.max_query_length = max_query_length
-        self.examples = {}
-        self.features = {}
-        self.remove_noanswer = remove_noanswer
-
-        if random_seed is not None:
-            np.random.seed(random_seed)
-
-        self.current_example = 0
-        self.current_epoch = 0
-        self.num_examples = 0
-
-        self.Example = namedtuple('Example',
-                ['qas_id', 'question_text', 'doc_tokens', 'orig_answer_text',
-                'start_position', 'end_position'])
-        self.Feature = namedtuple("Feature", ["unique_id", "example_index", "doc_span_index",
-                "tokens", "token_to_orig_map", "token_is_max_context",
-                "token_ids", "position_ids", "text_type_ids",
-                "start_position", "end_position"])
-        self.DocSpan = namedtuple("DocSpan", ["start", "length"])
-
-    def _read_json(self, input_file, is_training):
-        examples = []
-        with open(input_file, "r", encoding='utf8') as f:
-            input_data = json.load(f)["data"]
-            for entry in input_data:
-                for paragraph in entry["paragraphs"]:
-                    paragraph_text = paragraph["context"]
-                    for qa in paragraph["qas"]:
-                        qas_id = qa["id"]
-                        question_text = qa["question"]
-                        start_pos = None
-                        end_pos = None
-                        orig_answer_text = None
-
-                        if is_training:
-                            if len(qa["answers"]) != 1:
-                                raise ValueError(
-                                    "For training, each question should have exactly 1 answer."
-                                )
-
-                            answer = qa["answers"][0]
-                            orig_answer_text = answer["text"]
-                            answer_offset = answer["answer_start"]
-                            answer_length = len(orig_answer_text)
-                            doc_tokens = [
-                                paragraph_text[:answer_offset],
-                                paragraph_text[answer_offset:answer_offset +
-                                               answer_length],
-                                paragraph_text[answer_offset + answer_length:]
-                            ]
-
-                            start_pos = 1
-                            end_pos = 1
-
-                            actual_text = " ".join(doc_tokens[start_pos:(end_pos
-                                                                         + 1)])
-                            if actual_text.find(orig_answer_text) == -1:
-                                log.info("Could not find answer: '%s' vs. '%s'",
-                                      actual_text, orig_answer_text)
-                                continue
-                        else:
-                            doc_tokens = tokenization.tokenize_chinese_chars(
-                                paragraph_text)
-
-                        example = self.Example(
-                            qas_id=qas_id,
-                            question_text=question_text,
-                            doc_tokens=doc_tokens,
-                            orig_answer_text=orig_answer_text,
-                            start_position=start_pos,
-                            end_position=end_pos)
-                        examples.append(example)
-
-        return examples
-
-    def _improve_answer_span(self, doc_tokens, input_start, input_end,
-                             tokenizer, orig_answer_text):
-        tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))
-
-        for new_start in range(input_start, input_end + 1):
-            for new_end in range(input_end, new_start - 1, -1):
-                text_span = " ".join(doc_tokens[new_start:(new_end + 1)])
-                if text_span == tok_answer_text:
-                    return (new_start, new_end)
-
-        return (input_start, input_end)
-
-    def _check_is_max_context(self, doc_spans, cur_span_index, position):
-        best_score = None
-        best_span_index = None
-        for (span_index, doc_span) in enumerate(doc_spans):
-            end = doc_span.start + doc_span.length - 1
-            if position < doc_span.start:
-                continue
-            if position > end:
-                continue
-            num_left_context = position - doc_span.start
-            num_right_context = end - position
-            score = min(num_left_context,
-                        num_right_context) + 0.01 * doc_span.length
-            if best_score is None or score > best_score:
-                best_score = score
-                best_span_index = span_index
-
-        return cur_span_index == best_span_index
-
-    def _convert_example_to_feature(self, examples, max_seq_length, tokenizer,
-                                    is_training, remove_noanswer=True):
-        features = []
-        unique_id = 1000000000
-
-        print('converting examples to features...')
-        for (example_index, example) in enumerate(examples):
-            if example_index % 1000 == 0:
-                print('processing {}th example...'.format(example_index))
-            query_tokens = tokenizer.tokenize(example.question_text)
-            if len(query_tokens) > self.max_query_length:
-                query_tokens = query_tokens[0:self.max_query_length]
-            tok_to_orig_index = []
-            orig_to_tok_index = []
-            all_doc_tokens = []
-            for (i, token) in enumerate(example.doc_tokens):
-                orig_to_tok_index.append(len(all_doc_tokens))
-                sub_tokens = tokenizer.tokenize(token)
-                for sub_token in sub_tokens:
-                    tok_to_orig_index.append(i)
-                    all_doc_tokens.append(sub_token)
-
-            tok_start_position = None
-            tok_end_position = None
-            if is_training:
-                tok_start_position = orig_to_tok_index[example.start_position]
-                if example.end_position < len(example.doc_tokens) - 1:
-                    tok_end_position = orig_to_tok_index[example.end_position +
-                                                         1] - 1
-                else:
-                    tok_end_position = len(all_doc_tokens) - 1
-                (tok_start_position,
-                 tok_end_position) = self._improve_answer_span(
-                     all_doc_tokens, tok_start_position, tok_end_position,
-                     tokenizer, example.orig_answer_text)
-
-            max_tokens_for_doc = max_seq_length - len(query_tokens) - 3
-            doc_spans = []
-            start_offset = 0
-            while start_offset < len(all_doc_tokens):
-                length = len(all_doc_tokens) - start_offset
-                if length > max_tokens_for_doc:
-                    length = max_tokens_for_doc
-                doc_spans.append(self.DocSpan(start=start_offset, length=length))
-                if start_offset + length == len(all_doc_tokens):
-                    break
-                start_offset += min(length, self.doc_stride)
-
-            for (doc_span_index, doc_span) in enumerate(doc_spans):
-                tokens = []
-                token_to_orig_map = {}
-                token_is_max_context = {}
-                text_type_ids = []
-                tokens.append("[CLS]")
-                text_type_ids.append(0)
-                for token in query_tokens:
-                    tokens.append(token)
-                    text_type_ids.append(0)
-                tokens.append("[SEP]")
-                text_type_ids.append(0)
-
-                for i in range(doc_span.length):
-                    split_token_index = doc_span.start + i
-                    token_to_orig_map[len(tokens)] = tok_to_orig_index[
-                        split_token_index]
-
-                    is_max_context = self._check_is_max_context(
-                        doc_spans, doc_span_index, split_token_index)
-                    token_is_max_context[len(tokens)] = is_max_context
-                    tokens.append(all_doc_tokens[split_token_index])
-                    text_type_ids.append(1)
-                tokens.append("[SEP]")
-                text_type_ids.append(1)
-
-                token_ids = tokenizer.convert_tokens_to_ids(tokens)
-                position_ids = list(range(len(token_ids)))
-                start_position = None
-                end_position = None
-                if is_training:
-                    doc_start = doc_span.start
-                    doc_end = doc_span.start + doc_span.length - 1
-                    out_of_span = False
-                    if not (tok_start_position >= doc_start and
-                            tok_end_position <= doc_end):
-                        out_of_span = True
-                    if out_of_span:
-                        start_position = 0
-                        end_position = 0
-                        if remove_noanswer:
-                            continue
-                    else:
-                        doc_offset = len(query_tokens) + 2
-                        start_position = tok_start_position - doc_start + doc_offset
-                        end_position = tok_end_position - doc_start + doc_offset
-
-                feature = self.Feature(
-                    unique_id=unique_id,
-                    example_index=example_index,
-                    doc_span_index=doc_span_index,
-                    tokens=tokens,
-                    token_to_orig_map=token_to_orig_map,
-                    token_is_max_context=token_is_max_context,
-                    token_ids=token_ids,
-                    position_ids=position_ids,
-                    text_type_ids=text_type_ids,
-                    start_position=start_position,
-                    end_position=end_position)
-                features.append(feature)
-
-                unique_id += 1
-
-        return features
-
-    def _prepare_batch_data(self, records, batch_size, phase=None):
-        """generate batch records"""
-        batch_records, max_len = [], 0
-
-        if len(records) < batch_size:
-            raise Exception('mrc dataset contains too few samples. Expect more than '+str(batch_size))
-
-        for index, record in enumerate(records):
-            if phase == "train":
-                self.current_example = index
-            max_len = max(max_len, len(record.token_ids))
-            if self.in_tokens:
-                to_append = (len(batch_records) + 1) * max_len <= batch_size
-            else:
-                to_append = len(batch_records) < batch_size
-            if to_append:
-                batch_records.append(record)
-            else:
-                yield self._pad_batch_records(batch_records, phase == "train")
-                batch_records, max_len = [record], len(record.token_ids)
-
-        if phase == 'pred' and batch_records:
-            yield self._pad_batch_records(batch_records, phase == "train")
-
-    def _pad_batch_records(self, batch_records, is_training):
-        batch_token_ids = [record.token_ids for record in batch_records]
-        batch_text_type_ids = [record.text_type_ids for record in batch_records]
-        batch_position_ids = [record.position_ids for record in batch_records]
-        if is_training:
-            batch_start_position = [
-                record.start_position for record in batch_records
-            ]
-            batch_end_position = [
-                record.end_position for record in batch_records
-            ]
-            batch_start_position = np.array(batch_start_position).astype(
-                "int64").reshape([-1, 1])
-            batch_end_position = np.array(batch_end_position).astype(
-                "int64").reshape([-1, 1])
-
-        else:
-            batch_size = len(batch_token_ids)
-            batch_start_position = np.zeros(
-                shape=[batch_size, 1], dtype="int64")
-            batch_end_position = np.zeros(shape=[batch_size, 1], dtype="int64")
-
-        batch_unique_ids = [record.unique_id for record in batch_records]
-        batch_unique_ids = np.array(batch_unique_ids).astype("int64").reshape(
-            [-1, 1])
-
-        # padding
-        padded_token_ids, input_mask = pad_batch_data(
-            batch_token_ids, pad_idx=self.pad_id, return_input_mask=True)
-        padded_text_type_ids = pad_batch_data(
-            batch_text_type_ids, pad_idx=self.pad_id)
-        padded_position_ids = pad_batch_data(
-            batch_position_ids, pad_idx=self.pad_id)
-        padded_task_ids = np.ones_like(
-            padded_token_ids, dtype="int64") * self.task_id
-
-        return_list = [
-            padded_token_ids, padded_text_type_ids, padded_position_ids,
-            padded_task_ids, input_mask, batch_start_position,
-            batch_end_position, batch_unique_ids
-        ]
-
-        return return_list
-
-    def get_num_examples(self, phase):
-        return len(self.features[phase])
-
-    def get_features(self, phase):
-        return self.features[phase]
-
-    def get_examples(self, phase):
-        return self.examples[phase]
-
-    def data_generator(self,
-                       input_file,
-                       batch_size,
-                       epoch,
-                       dev_count=1,
-                       shuffle=True,
-                       phase=None):
-
-        examples = self.examples.get(phase, None)
-        features = self.features.get(phase, None)
-        if not examples:
-            examples = self._read_json(input_file, phase == "train")
-            features = self._convert_example_to_feature(
-                examples, self.max_seq_len, self.tokenizer, phase == "train", remove_noanswer=self.remove_noanswer)
-            self.examples[phase] = examples
-            self.features[phase] = features
-
-        def wrapper():
-            all_dev_batches = []
-            if epoch is None:
-                num_epochs = 99999999
-            else:
-                num_epochs = epoch
-            for epoch_index in range(num_epochs):
-                if phase == "train":
-                    self.current_example = 0
-                    self.current_epoch = epoch_index
-                if phase == "train" and shuffle:
-                    np.random.shuffle(features)
-
-                for batch_data in self._prepare_batch_data(
-                        features, batch_size, phase=phase):
-                    if len(all_dev_batches) < dev_count:
-                        all_dev_batches.append(batch_data)
-                    if len(all_dev_batches) == dev_count:
-                        for batch in all_dev_batches:
-                            yield batch
-                        all_dev_batches = []
-
-        return wrapper
-
-
-if __name__ == '__main__':
-    pass

tasktype/match.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import paddle.fluid as fluid
-from paddle.fluid import layers
-from paddlepalm.interface import task_paradigm
-import numpy as np
-import os
-
-class TaskParadigm(task_paradigm):
-    '''
-    matching
-    '''
-    def __init__(self, config, phase, backbone_config=None):
-        self._is_training = phase == 'train'
-        self._hidden_size = backbone_config['hidden_size']
-
-        if 'initializer_range' in config:
-            self._param_initializer = config['initializer_range']
-        else:
-            self._param_initializer = fluid.initializer.TruncatedNormal(
-                scale=backbone_config.get('initializer_range', 0.02))
-        if 'dropout_prob' in config:
-            self._dropout_prob = config['dropout_prob']
-        else:
-            self._dropout_prob = backbone_config.get('hidden_dropout_prob', 0.0)
-
-        self._pred_output_path = config.get('pred_output_path', None)
-        self._preds = []
-
-    
-    @property
-    def inputs_attrs(self):
-        if self._is_training:
-            reader = {"label_ids": [[-1, 1], 'int64']}
-        else:
-            reader = {}
-        bb = {"sentence_pair_embedding": [[-1, self._hidden_size], 'float32']}
-        return {'reader': reader, 'backbone': bb}
-
-    @property
-    def outputs_attrs(self):
-        if self._is_training:
-            return {"loss": [[1], 'float32']}
-        else:
-            return {"logits": [[-1, 2], 'float32']}
-
-    def build(self, inputs, scope_name=""):
-        if self._is_training:
-            labels = inputs["reader"]["label_ids"] 
-        cls_feats = inputs["backbone"]["sentence_pair_embedding"]
-
-        if self._is_training:
-            cls_feats = fluid.layers.dropout(
-                x=cls_feats,
-                dropout_prob=self._dropout_prob,
-                dropout_implementation="upscale_in_train")
-
-        logits = fluid.layers.fc(
-            input=cls_feats,
-            size=2,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+"cls_out_w",
-                initializer=self._param_initializer),
-            bias_attr=fluid.ParamAttr(
-                name=scope_name+"cls_out_b",
-                initializer=fluid.initializer.Constant(0.)))
-
-        if self._is_training:
-            ce_loss, probs = fluid.layers.softmax_with_cross_entropy(
-                logits=logits, label=labels, return_softmax=True)
-            loss = fluid.layers.mean(x=ce_loss)
-            return {'loss': loss}
-        else:
-            return {'logits': logits}
-
-    def postprocess(self, rt_outputs):
-        if not self._is_training:
-            logits = rt_outputs['logits']
-            preds = np.argmax(logits, -1)
-            self._preds.extend(preds.tolist())
-
-    def epoch_postprocess(self, post_inputs):
-        # there is no post_inputs needed and not declared in epoch_inputs_attrs, hence no elements exist in post_inputs
-        if not self._is_training:
-            if self._pred_output_path is None:
-                raise ValueError('argument pred_output_path not found in config. Please add it into config dict/file.')
-            with open(os.path.join(self._pred_output_path, 'predictions.json'), 'w') as writer:
-                for p in self._preds:
-                    writer.write(str(p)+'\n')
-            print('Predictions saved at '+os.path.join(self._pred_output_path, 'predictions.json'))
-
-                

tasktype/mlm.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import paddle.fluid as fluid
-from paddlepalm.interface import task_paradigm
-from paddle.fluid import layers
-from paddlepalm.backbone.utils.transformer import pre_process_layer
-
-class TaskParadigm(task_paradigm):
-    '''
-    matching
-    '''
-    def __init__(self, config, phase, backbone_config=None):
-        self._is_training = phase == 'train'
-        self._emb_size = backbone_config['hidden_size']
-        self._hidden_size = backbone_config['hidden_size']
-        self._vocab_size = backbone_config['vocab_size']
-        self._hidden_act = backbone_config['hidden_act']
-        self._initializer_range = backbone_config['initializer_range']
-    
-    @property
-    def inputs_attrs(self):
-        reader = {
-            "mask_label": [[-1, 1], 'int64'],
-            "mask_pos": [[-1, 1], 'int64']}
-        if not self._is_training:
-            del reader['mask_label']
-            del reader['batchsize_x_seqlen']
-        bb = {
-            "encoder_outputs": [[-1, -1, self._hidden_size], 'float32'],
-            "embedding_table": [[-1, self._vocab_size, self._emb_size], 'float32']}
-        return {'reader': reader, 'backbone': bb}
-
-    @property
-    def outputs_attrs(self):
-        if self._is_training:
-            return {"loss": [[1], 'float32']}
-        else:
-            return {"logits": [[-1], 'float32']}
-
-    def build(self, inputs, scope_name=""):
-        mask_pos = inputs["reader"]["mask_pos"]
-        if self._is_training:
-            mask_label = inputs["reader"]["mask_label"] 
-            max_position = inputs["reader"]["batchsize_x_seqlen"] - 1
-            mask_pos = fluid.layers.elementwise_min(mask_pos, max_position)
-            mask_pos.stop_gradient = True
-
-        word_emb = inputs["backbone"]["embedding_table"]
-        enc_out = inputs["backbone"]["encoder_outputs"]
-
-        emb_size = word_emb.shape[-1]
-
-        _param_initializer = fluid.initializer.TruncatedNormal(
-            scale=self._initializer_range)
-
-        reshaped_emb_out = fluid.layers.reshape(
-            x=enc_out, shape=[-1, emb_size])
-
-        # extract masked tokens' feature
-        mask_feat = fluid.layers.gather(input=reshaped_emb_out, index=mask_pos)
-
-        # transform: fc
-        mask_trans_feat = fluid.layers.fc(
-            input=mask_feat,
-            size=emb_size,
-            act=self._hidden_act,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+'mask_lm_trans_fc.w_0',
-                initializer=_param_initializer),
-            bias_attr=fluid.ParamAttr(name=scope_name+'mask_lm_trans_fc.b_0'))
-        # transform: layer norm
-        mask_trans_feat = pre_process_layer(
-            mask_trans_feat, 'n', name=scope_name+'mask_lm_trans')
-
-        mask_lm_out_bias_attr = fluid.ParamAttr(
-            name=scope_name+"mask_lm_out_fc.b_0",
-            initializer=fluid.initializer.Constant(value=0.0))
-
-        fc_out = fluid.layers.matmul(
-            x=mask_trans_feat,
-            y=word_emb,
-            transpose_y=True)
-        fc_out += fluid.layers.create_parameter(
-            shape=[self._vocab_size],
-            dtype='float32',
-            attr=mask_lm_out_bias_attr,
-            is_bias=True)
-
-        if self._is_training:
-            mask_lm_loss = fluid.layers.softmax_with_cross_entropy(
-                logits=fc_out, label=mask_label)
-            loss = fluid.layers.mean(mask_lm_loss)
-            return {'loss': loss}
-        else:
-            return {'logits': fc_out}
-
-

tasktype/mrc.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import paddle.fluid as fluid
-from paddlepalm.interface import task_paradigm
-import collections
-import numpy as np
-import os
-import math
-import six
-import paddlepalm.tokenizer.ernie_tokenizer as tokenization
-import json
-
-RawResult = collections.namedtuple("RawResult",
-                                   ["unique_id", "start_logits", "end_logits"])
-
-class TaskParadigm(task_paradigm):
-    """"""
-
-    def __init__(self, config, phase, backbone_config=None):
-        
-        self._is_training = phase == 'train'
-        self._max_sequence_length = config['max_seq_len']
-        self._hidden_size = backbone_config['hidden_size']
-        self._pred_results = []
-        
-        if phase == 'pred':
-            self._max_answer_length = config.get('max_answer_len', None)
-            self._null_score_diff_threshold = config.get('null_score_diff_threshold', 0.0)
-            self._n_best_size = config.get('n_best_size', 20)
-            self._pred_output_path = config.get('pred_output_path', None)
-            self._verbose = config.get('verbose', False)
-            self._with_negative = config.get('with_negative', False)
-            self._do_lower_case = config.get('do_lower_case', False)
-
-
-    @property
-    def inputs_attrs(self):
-        if self._is_training:
-            reader = {"start_positions": [[-1, 1], 'int64'],
-                      "end_positions": [[-1, 1], 'int64'],
-                      }
-        else:
-            reader = {'unique_ids': [[-1, 1], 'int64']}
-        bb = {"encoder_outputs": [[-1, -1, self._hidden_size], 'float32']}
-        return {'reader': reader, 'backbone': bb}
-        
-    @property
-    def epoch_inputs_attrs(self):
-        if not self._is_training:
-            from_reader = {'examples': None, 'features': None}
-            return {'reader': from_reader}
-
-    @property
-    def outputs_attr(self):
-        if self._is_training:
-            return {'loss': [[1], 'float32']}
-        else:
-            return {'start_logits': [[-1, -1, 1], 'float32'],
-                    'end_logits': [[-1, -1, 1], 'float32'],
-                    'unique_ids': [[-1, 1], 'int64']}
-
-
-    def build(self, inputs, scope_name=""):
-        if self._is_training:
-            start_positions = inputs['reader']['start_positions']
-            end_positions = inputs['reader']['end_positions']
-            max_position = inputs["reader"]["seqlen"] - 1
-            start_positions = fluid.layers.elementwise_min(start_positions, max_position)
-            end_positions = fluid.layers.elementwise_min(end_positions, max_position)
-            start_positions.stop_gradient = True
-            end_positions.stop_gradient = True
-        else:
-            unique_id = inputs['reader']['unique_ids']
-
-        enc_out = inputs['backbone']['encoder_outputs']
-        logits = fluid.layers.fc(
-            input=enc_out,
-            size=2,
-            num_flatten_dims=2,
-            param_attr=fluid.ParamAttr(
-                name=scope_name+"cls_squad_out_w",
-                initializer=fluid.initializer.TruncatedNormal(scale=0.02)),
-            bias_attr=fluid.ParamAttr(
-                name=scope_name+"cls_squad_out_b", initializer=fluid.initializer.Constant(0.)))
-
-        logits = fluid.layers.transpose(x=logits, perm=[2, 0, 1])
-        start_logits, end_logits = fluid.layers.unstack(x=logits, axis=0)
-
-        def _compute_single_loss(logits, positions):
-            """Compute start/end loss for mrc model"""
-            loss = fluid.layers.softmax_with_cross_entropy(
-                logits=logits, label=positions)
-            loss = fluid.layers.mean(x=loss)
-            return loss
-
-        if self._is_training:
-            start_loss = _compute_single_loss(start_logits, start_positions)
-            end_loss = _compute_single_loss(end_logits, end_positions)
-            total_loss = (start_loss + end_loss) / 2.0
-            return {'loss': total_loss}
-        else:
-            return {'start_logits': start_logits,
-                    'end_logits': end_logits,
-                    'unique_ids': unique_id}
-
-
-    def postprocess(self, rt_outputs):
-        """this func will be called after each step(batch) of training/evaluating/predicting process."""
-        if not self._is_training:
-            unique_ids = np.squeeze(rt_outputs['unique_ids'], -1)
-            start_logits = rt_outputs['start_logits']
-            end_logits = rt_outputs['end_logits']
-            for idx in range(len(unique_ids)):
-                
-                if unique_ids[idx] < 0:
-                    continue
-                if len(self._pred_results) % 1000 == 0:
-                    print("Predicting example: {}".format(len(self._pred_results)))
-                uid = int(unique_ids[idx])
-
-                s = [float(x) for x in start_logits[idx].flat]
-                e = [float(x) for x in end_logits[idx].flat]
-                self._pred_results.append(
-                    RawResult(
-                        unique_id=uid,
-                        start_logits=s,
-                        end_logits=e))
-
-    def epoch_postprocess(self, post_inputs):
-        """(optional interface) this func will be called after evaluation/predicting process and each epoch during training process."""
-
-        if not self._is_training:
-            if self._pred_output_path is None:
-                raise ValueError('argument pred_output_path not found in config. Please add it into config dict/file.')
-            examples = post_inputs['reader']['examples']
-            features = post_inputs['reader']['features']
-            if not os.path.exists(self._pred_output_path):
-                os.makedirs(self._pred_output_path)
-            output_prediction_file = os.path.join(self._pred_output_path, "predictions.json")
-            output_nbest_file = os.path.join(self._pred_output_path, "nbest_predictions.json")
-            output_null_log_odds_file = os.path.join(self._pred_output_path, "null_odds.json")
-            _write_predictions(examples, features, self._pred_results,
-                              self._n_best_size, self._max_answer_length,
-                              self._do_lower_case, output_prediction_file,
-                              output_nbest_file, output_null_log_odds_file,
-                              self._with_negative,
-                              self._null_score_diff_threshold, self._verbose)
-
-
-def _write_predictions(all_examples, all_features, all_results, n_best_size,
-                      max_answer_length, do_lower_case, output_prediction_file,
-                      output_nbest_file, output_null_log_odds_file,
-                      with_negative, null_score_diff_threshold,
-                      verbose):
-    """Write final predictions to the json file and log-odds of null if needed."""
-    print("Writing predictions to: %s" % (output_prediction_file))
-    print("Writing nbest to: %s" % (output_nbest_file))
-
-    example_index_to_features = collections.defaultdict(list)
-    for feature in all_features:
-        example_index_to_features[feature.example_index].append(feature)
-
-    unique_id_to_result = {}
-    for result in all_results:
-        unique_id_to_result[result.unique_id] = result
-
-    _PrelimPrediction = collections.namedtuple(  # pylint: disable=invalid-name
-        "PrelimPrediction", [
-            "feature_index", "start_index", "end_index", "start_logit",
-            "end_logit"
-        ])
-
-    all_predictions = collections.OrderedDict()
-    all_nbest_json = collections.OrderedDict()
-    scores_diff_json = collections.OrderedDict()
-
-    for (example_index, example) in enumerate(all_examples):
-        features = example_index_to_features[example_index]
-
-        prelim_predictions = []
-        # keep track of the minimum score of null start+end of position 0
-        score_null = 1000000  # large and positive
-        min_null_feature_index = 0  # the paragraph slice with min mull score
-        null_start_logit = 0  # the start logit at the slice with min null score
-        null_end_logit = 0  # the end logit at the slice with min null score
-        for (feature_index, feature) in enumerate(features):
-            result = unique_id_to_result[feature.unique_id]
-            start_indexes = _get_best_indexes(result.start_logits, n_best_size)
-            end_indexes = _get_best_indexes(result.end_logits, n_best_size)
-            # if we could have irrelevant answers, get the min score of irrelevant
-            if with_negative:
-                feature_null_score = result.start_logits[0] + result.end_logits[
-                    0]
-                if feature_null_score < score_null:
-                    score_null = feature_null_score
-                    min_null_feature_index = feature_index
-                    null_start_logit = result.start_logits[0]
-                    null_end_logit = result.end_logits[0]
-            for start_index in start_indexes:
-                for end_index in end_indexes:
-                    # We could hypothetically create invalid predictions, e.g., predict
-                    # that the start of the span is in the question. We throw out all
-                    # invalid predictions.
-                    if start_index >= len(feature.tokens):
-                        continue
-                    if end_index >= len(feature.tokens):
-                        continue
-                    if start_index not in feature.token_to_orig_map:
-                        continue
-                    if end_index not in feature.token_to_orig_map:
-                        continue
-                    if not feature.token_is_max_context.get(start_index, False):
-                        continue
-                    if end_index < start_index:
-                        continue
-                    length = end_index - start_index + 1
-                    if length > max_answer_length:
-                        continue
-                    prelim_predictions.append(
-                        _PrelimPrediction(
-                            feature_index=feature_index,
-                            start_index=start_index,
-                            end_index=end_index,
-                            start_logit=result.start_logits[start_index],
-                            end_logit=result.end_logits[end_index]))
-
-        if with_negative:
-            prelim_predictions.append(
-                _PrelimPrediction(
-                    feature_index=min_null_feature_index,
-                    start_index=0,
-                    end_index=0,
-                    start_logit=null_start_logit,
-                    end_logit=null_end_logit))
-        prelim_predictions = sorted(
-            prelim_predictions,
-            key=lambda x: (x.start_logit + x.end_logit),
-            reverse=True)
-
-        _NbestPrediction = collections.namedtuple(  # pylint: disable=invalid-name
-            "NbestPrediction", ["text", "start_logit", "end_logit"])
-
-        seen_predictions = {}
-        nbest = []
-        for pred in prelim_predictions:
-            if len(nbest) >= n_best_size:
-                break
-            feature = features[pred.feature_index]
-            if pred.start_index > 0:  # this is a non-null prediction
-                tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1
-                                                              )]
-                orig_doc_start = feature.token_to_orig_map[pred.start_index]
-                orig_doc_end = feature.token_to_orig_map[pred.end_index]
-                orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end +
-                                                                 1)]
-                tok_text = " ".join(tok_tokens)
-
-                # De-tokenize WordPieces that have been split off.
-                tok_text = tok_text.replace(" ##", "")
-                tok_text = tok_text.replace("##", "")
-
-                # Clean whitespace
-                tok_text = tok_text.strip()
-                tok_text = " ".join(tok_text.split())
-                orig_text = " ".join(orig_tokens)
-
-                final_text = _get_final_text(tok_text, orig_text, do_lower_case,
-                                            verbose)
-                if final_text in seen_predictions:
-                    continue
-
-                seen_predictions[final_text] = True
-            else:
-                final_text = ""
-                seen_predictions[final_text] = True
-
-            nbest.append(
-                _NbestPrediction(
-                    text=final_text,
-                    start_logit=pred.start_logit,
-                    end_logit=pred.end_logit))
-
-        # if we didn't inlude the empty option in the n-best, inlcude it
-        if with_negative:
-            if "" not in seen_predictions:
-                nbest.append(
-                    _NbestPrediction(
-                        text="",
-                        start_logit=null_start_logit,
-                        end_logit=null_end_logit))
-        # In very rare edge cases we could have no valid predictions. So we
-        # just create a nonce prediction in this case to avoid failure.
-        if not nbest:
-            nbest.append(
-                _NbestPrediction(
-                    text="empty", start_logit=0.0, end_logit=0.0))
-
-        assert len(nbest) >= 1
-
-        total_scores = []
-        best_non_null_entry = None
-        for entry in nbest:
-            total_scores.append(entry.start_logit + entry.end_logit)
-            if not best_non_null_entry:
-                if entry.text:
-                    best_non_null_entry = entry
-        # debug
-        if best_non_null_entry is None:
-            print("Emmm..., sth wrong")
-
-        probs = _compute_softmax(total_scores)
-
-        nbest_json = []
-        for (i, entry) in enumerate(nbest):
-            output = collections.OrderedDict()
-            output["text"] = entry.text
-            output["probability"] = probs[i]
-            output["start_logit"] = entry.start_logit
-            output["end_logit"] = entry.end_logit
-            nbest_json.append(output)
-
-        assert len(nbest_json) >= 1
-
-        if not with_negative:
-            all_predictions[example.qas_id] = nbest_json[0]["text"]
-        else:
-            # predict "" iff the null score - the score of best non-null > threshold
-            score_diff = score_null - best_non_null_entry.start_logit - (
-                best_non_null_entry.end_logit)
-            scores_diff_json[example.qas_id] = score_diff
-            if score_diff > null_score_diff_threshold:
-                all_predictions[example.qas_id] = ""
-            else:
-                all_predictions[example.qas_id] = best_non_null_entry.text
-
-        all_nbest_json[example.qas_id] = nbest_json
-
-    with open(output_prediction_file, "w") as writer:
-        writer.write(json.dumps(all_predictions, indent=4) + "\n")
-
-    with open(output_nbest_file, "w") as writer:
-        writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
-
-    if with_negative:
-        with open(output_null_log_odds_file, "w") as writer:
-            writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
-
-
-def _get_final_text(pred_text, orig_text, do_lower_case, verbose):
-    """Project the tokenized prediction back to the original text."""
-
-    # When we created the data, we kept track of the alignment between original
-    # (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
-    # now `orig_text` contains the span of our original text corresponding to the
-    # span that we predicted.
-    #
-    # However, `orig_text` may contain extra characters that we don't want in
-    # our prediction.
-    #
-    # For example, let's say:
-    #   pred_text = steve smith
-    #   orig_text = Steve Smith's
-    #
-    # We don't want to return `orig_text` because it contains the extra "'s".
-    #
-    # We don't want to return `pred_text` because it's already been normalized
-    # (the MRQA eval script also does punctuation stripping/lower casing but
-    # our tokenizer does additional normalization like stripping accent
-    # characters).
-    #
-    # What we really want to return is "Steve Smith".
-    #
-    # Therefore, we have to apply a semi-complicated alignment heruistic between
-    # `pred_text` and `orig_text` to get a character-to-charcter alignment. This
-    # can fail in certain cases in which case we just return `orig_text`.
-
-    def _strip_spaces(text):
-        ns_chars = []
-        ns_to_s_map = collections.OrderedDict()
-        for (i, c) in enumerate(text):
-            if c == " ":
-                continue
-            ns_to_s_map[len(ns_chars)] = i
-            ns_chars.append(c)
-        ns_text = "".join(ns_chars)
-        return (ns_text, ns_to_s_map)
-
-    # We first tokenize `orig_text`, strip whitespace from the result
-    # and `pred_text`, and check if they are the same length. If they are
-    # NOT the same length, the heuristic has failed. If they are the same
-    # length, we assume the characters are one-to-one aligned.
-    tokenizer = tokenization.BasicTokenizer(do_lower_case=do_lower_case)
-
-    tok_text = " ".join(tokenizer.tokenize(orig_text))
-
-    start_position = tok_text.find(pred_text)
-    if start_position == -1:
-        if verbose:
-            print("Unable to find text: '%s' in '%s'" % (pred_text, orig_text))
-        return orig_text
-    end_position = start_position + len(pred_text) - 1
-
-    (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
-    (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)
-
-    if len(orig_ns_text) != len(tok_ns_text):
-        if verbose:
-            print("Length not equal after stripping spaces: '%s' vs '%s'",
-                  orig_ns_text, tok_ns_text)
-        return orig_text
-
-    # We then project the characters in `pred_text` back to `orig_text` using
-    # the character-to-character alignment.
-    tok_s_to_ns_map = {}
-    for (i, tok_index) in six.iteritems(tok_ns_to_s_map):
-        tok_s_to_ns_map[tok_index] = i
-
-    orig_start_position = None
-    if start_position in tok_s_to_ns_map:
-        ns_start_position = tok_s_to_ns_map[start_position]
-        if ns_start_position in orig_ns_to_s_map:
-            orig_start_position = orig_ns_to_s_map[ns_start_position]
-
-    if orig_start_position is None:
-        if verbose:
-            print("Couldn't map start position")
-        return orig_text
-
-    orig_end_position = None
-    if end_position in tok_s_to_ns_map:
-        ns_end_position = tok_s_to_ns_map[end_position]
-        if ns_end_position in orig_ns_to_s_map:
-            orig_end_position = orig_ns_to_s_map[ns_end_position]
-
-    if orig_end_position is None:
-        if verbose:
-            print("Couldn't map end position")
-        return orig_text
-
-    output_text = orig_text[orig_start_position:(orig_end_position + 1)]
-    return output_text
-
-
-def _get_best_indexes(logits, n_best_size):
-    """Get the n-best logits from a list."""
-    index_and_score = sorted(
-        enumerate(logits), key=lambda x: x[1], reverse=True)
-
-    best_indexes = []
-    for i in range(len(index_and_score)):
-        if i >= n_best_size:
-            break
-        best_indexes.append(index_and_score[i][0])
-    return best_indexes
-
-
-def _compute_softmax(scores):
-    """Compute softmax probability over raw logits."""
-    if not scores:
-        return []
-
-    max_score = None
-    for score in scores:
-        if max_score is None or score > max_score:
-            max_score = score
-
-    exp_scores = []
-    total_sum = 0.0
-    for score in scores:
-        x = math.exp(score - max_score)
-        exp_scores.append(x)
-        total_sum += x
-
-    probs = []
-    for score in exp_scores:
-        probs.append(score / total_sum)
-    return probs
-
-