update docs

README.md

 # PaddlePALM
 
-PaddlePALM (Paddle for Multi-task) 是一个灵活通用且易用的NLP大规模预训练与多任务学习框架。通过PaddlePALM，用户可以轻松完成复杂的多任务学习与参数复用，无缝集成「**单任务训练**」、「**多任务辅助训练**」和「**多目标任务联合训练**」这 *3* 种训练方式和灵活的保存与预测机制，且仅需书写极少量代码即可”一键启动”高性能单机单卡和分布式训练与推理。
+PaddlePALM (PArallel Learning from Multi-tasks) is a flexible, general and easy-to-use NLP large-scale pretraining and multi-task learning friendly framework. PALM is a high level framework aiming at **fastly** develop **high-performance** NLP models. With PALM, 8 steps to achieve a typical NLP task for supervised learning or pretraining. 6 steps to achieve multi-task learning for prepared tasks. Zero steps to adapt your code to large-scale training/inference (with multiple GPUs and multiple computation nodes).
 
-框架中内置了丰富的[主干网络](#附录b内置主干网络backbone)及其[预训练模型](#预训练模型)（BERT、ERNIE等）、常见的[任务范式](#附录c内置任务范式paradigm)（分类、匹配、机器阅读理解等）和相应的[数据集读取与处理工具](#附录a内置数据集载入与处理工具reader)。同时框架提供了用户自定义接口，若内置工具、主干网络和任务无法满足需求，开发者可以轻松完成相关组件的自定义。各个组件均为零耦合设计，用户仅需完成组件本身的特性开发即可完成与框架的融合。
-
-PaddlePALM (PArallel Learning from Multi-tasks) is a flexible, general and easy-to-use NLP large-scale pretraining and multi-task learning friendly framework. PALM is a high level framework aiming at **fastly** develop **high-performance** NLP models. With PALM, a typical NLP task can be achieved just in 8 steps. 
-s   
+PaddlePALM also provides state-of-the-art general purpose architectures (BERT,ERNIE,RoBERTa,...) as build-in model backbones. We have decoupled the model backbone, dataset reader and task output layers, so that you can easily replace any of the component to other candidates with quite minor changes of your code. In addition, PaddlePALM support customized development of any component, e.g, backbone, task head, reader and optimizer, which gives high flexibility for developers to adapt to complicated NLP scenes. 
 
 然后给出一些成功案例和一些公开数据集的各个backbone的实验结果（BERT、ERNIE、RoBERTa）和一些成功的多任务学习示例。
 
-## 目录
-
-- [安装](#安装)
-- [前期准备](#前期准备)
-    - [理论准备](#理论准备)
-    - [框架原理](#框架原理)
-    - [预训练模型](#预训练模型)
-- [X行代码实现文本分类](#三个demo入门paddlepalm)
-    - 
-- []
-    - [DEMO1：单任务训练](#demo1单任务训练)
-    - [DEMO2：多任务辅助训练与目标任务预测](#demo2多任务辅助训练与目标任务预测)
-    - [DEMO3：多目标任务联合训练与任务层参数复用](#demo3多目标任务联合训练与任务层参数复用)
-- [进阶篇](#进阶篇)
-    - [配置广播机制](#配置广播机制)
-    - [reader、backbone与paradigm的选择](#readerbackbone与paradigm的选择)
-    - [多目标任务下的训练终止条件与预期训练步数](#多目标任务下的训练终止条件与预期训练步数)
-        - [多个目标任务](#多个目标任务)
-        - [训练终止条件](#训练终止条件)
-        - [任务采样概率与预期训练步数](#任务采样概率与预期训练步数)
-        - [多个目标任务时预期训练步数的计算](#多个目标任务时预期训练步数的计算)
-    - [模型保存与预测机制](#模型保存与预测机制)
-    - [分布式训练](#分布式训练)
-- [附录A：内置数据集载入与处理工具（reader）](#附录a内置数据集载入与处理工具reader)
-- [附录B：内置主干网络（backbone）](#附录b内置主干网络backbone)
-- [附录C：内置任务范式（paradigm）](#附录c内置任务范式paradigm)
-- [附录D：可配置的全局参数列表](#附录d可配置的全局参数列表)
-
 
 ## Package Overview
 
 | **paddlepalm** | an open source NLP pretraining and multitask learning framework, built on paddlepaddle. |
 | **paddlepalm.reader** | a collection of elastic task-specific dataset readers. |
-| **paddlepalm.backbone** | a collection of classic NLP representation models, e.g., BERT. |
+| **paddlepalm.backbone** | a collection of classic NLP representation models, e.g., BERT, ERNIE, RoBERTa. |
 | **paddlepalm.head** | a collection of task-specific output layers. |
 | **paddlepalm.lr_sched** | a collection of learning rate schedualers. |
 | **paddlepalm.optimizer** | a collection of optimizers. |
@@ -67,7 +36,7 @@ cd PALM && python setup.py install
 ```
 
 ### Library Dependencies
-- Python >= 2.7 (即将支持python3)
+- Python >= 2.7
 - cuda >= 9.0
 - cudnn >= 7.0
 - PaddlePaddle >= 1.6.3 (请参考[安装指南](http://www.paddlepaddle.org/#quick-start)进行安装)
@@ -108,7 +77,7 @@ Available pretrain items:
 5. create a task *trainer* with `paddlepalm.Trainer`, then build forward graph with backbone and task head (created in step 2 and 4) through `trainer.build_forward`.
 6. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `trainer.build_backward`.
 7. fit prepared reader and data (achieved in step 1) to trainer with `trainer.fit_reader` method.
-8. randomly initialize model parameters (and `trainer.load_pretrain` if needed), then do training with `trainer.train`.
+8. load pretrain model with `trainer.load_pretrain`, or load checkpoint with `trainer.load_ckpt` or nothing to do for training from scratch, then do training with `trainer.train`.
 
 More implementation details see following demos: [Sentiment Classification](), [Quora Question Pairs matching](), [Tagging](), [SQuAD machine Reading Comprehension]().
 

paddlepalm/_downloader.py

@@ -38,9 +38,10 @@ _items = {
                  'roberta-cn-base': 'https://bert-models.bj.bcebos.com/chinese_roberta_wwm_ext_L-12_H-768_A-12.tar.gz',
                  'roberta-cn-large': 'https://bert-models.bj.bcebos.com/chinese_roberta_wwm_large_ext_L-24_H-1024_A-16.tar.gz',
                  'utils': None},
-    'reader': {'utils': None},
+    'vocab': {'utils': None},
     'backbone': {'utils': None},
-    'tasktype': {'utils': None},
+    'head': {'utils': None},
+    'reader': {'utils': None},
 }
 
 def _download(item, scope, path, silent=False, convert=False):
@@ -131,20 +132,27 @@ def _convert(path, silent=False):
             tar_info.close()
             os.removedirs(path + '/params1/') 
 
-def download(item, scope='all', path='.'):
-    item = item.lower()
+def download(scope, item='all', path='.'):
+    """download an item. The available scopes and contained items can be showed with `paddlepalm.downloader.ls`.
+
+    Args:
+        scope: the scope the item belongs to.
+        item: the item to download.
+        path: the target dir to download to. Default is `.`, means current dir.
+    """
     scope = scope.lower()
-    assert item in _items, '{} is not found. Support list: {}'.format(item, list(_items.keys()))
+    item = item.lower()
+    ascopeert scope in _scopes, '{} is not found. Support list: {}'.format(scope, list(_scopes.keys()))
    
-    if _items[item]['utils'] is not None:
-        _download(item, 'utils', path, silent=True)
+    if _scopes[scope]['utils'] is not None:
+        _download(scope, 'utils', path, silent=True)
 
-    if scope != 'all':
-        assert scope in _items[item], '{} is not found. Support scopes: {}'.format(scope, list(_items[item].keys()))
-        _download(item, scope, path)
+    if item != 'all':
+        ascopeert item in _scopes[scope], '{} is not found. Support items: {}'.format(item, list(_scopes[scope].keys()))
+        _download(scope, item, path)
     else:
-        for s in _items[item].keys():
-            _download(item, s, path)
+        for s in _scopes[scope].keys():
+            _download(scope, s, path)
 
 
 def _ls(item, scope, l = 10):
@@ -157,19 +165,22 @@ def _ls(item, scope, l = 10):
                 continue
             print ('  => '+s)
 
-def ls(item='all', scope='all'):
+def ls(scope='all'):
+    """show all the available download items of a scope.
+
+    Args:
+        scope: the scope to show items. Default is 'all', means to show all items in all scopes. Avaliable scopes: pretrain.
+    """
     
-    if scope == 'utils':
-        return
-    if item != 'all':
-        assert item in _items, '{} is not found. Support scopes: {}'.format(item, list(_items.keys()))
-        print ('Available {} items:'.format(item))
-        _ls(item, scope)
+    if scope != 'all':
+        assert scope in _items, '{} is not found. Support scopes: {}'.format(scope, list(_items.keys()))
+        print ('Available {} scopes:'.format(scope))
+        _ls(scope, 'all')
     else:
         l = max(map(len, _items.keys()))
         for i in _items.keys():
             print ('Available {} items: '.format(i))
-            _ls(i, scope, l)
+            _ls(i, 'all', l)
 
 
     

paddlepalm/controller/__init__.py

-
-from conf_controller import ConfigController
-from controller import Controller

paddlepalm/default_settings.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-BACKBONE_DIR='paddlepalm.backbone'
-TASK_INSTANCE_DIR='paddlepalm.task_instance'
-READER_DIR='paddlepalm.reader'
-PARADIGM_DIR='paddlepalm.task_paradigm'
-OPTIMIZER_DIR='paddlepalm.optimizer'
-OPTIMIZE_METHOD='optimize'
-
-REQUIRED_ARGS={
-    'task_instance': str,
-    'backbone': str,
-    'optimizer': str,
-    'learning_rate': float,
-    'batch_size': int
-    }
-
-OPTIONAL_ARGS={
-    'mix_ratio': str,
-    'target_tag': str,
-    'reuse_rag': str
-    }
-
-TASK_REQUIRED_ARGS={
-    'paradigm': str,
-    'reader': str,
-    'train_file': str
-    }
-

paddlepalm/interface.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""v1.1"""
-
-class reader(object):
-    """interface of data manager."""
-
-    def __init__(self, config):
-        assert isinstance(config, dict)
-
-    # @property
-    # def inputs_attr(self):
-    #     """描述reader输入对象的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1.
-    #     Return:
-    #         dict类型。对各个输入对象的属性描述。例如，
-    #         对于文本分类任务，可能需要包含输入文本和所属标签的id
-    #             {"text": ([], 'str'),
-    #              "label": ([], 'int')}
-    #         对于标注任务，可能需要输入词序列和对应的标签
-    #             {"tokens", ([-1], 'str'),
-    #              "tags", ([-1], 'str')}
-    #         对于机器阅读理解任务，可能需要包含上下文、问题、回答、答案区域的起止位置等
-    #             {"paragraph", ([], 'str'),
-    #              "question", ([], 'str'),
-    #              "start_position", ([], 'int')
-    #         """
-    #     raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述reader输出对象（被yield出的对象）的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        注意：当使用mini-batch梯度下降学习策略时，，应为常规的输入对象设置batch_size维度（一般为-1）
-        Return:
-            dict类型。对各个输入对象的属性描述。例如，
-            对于文本分类和匹配任务，yield的输出内容可能包含如下的对象（下游backbone和task可按需访问其中的对象）
-                {"token_ids": ([-1, max_len], 'int64'),
-                 "input_ids": ([-1, max_len], 'int64'),
-                 "segment_ids": ([-1, max_len], 'int64'),
-                 "input_mask": ([-1, max_len], 'float32'),
-                 "label": ([-1], 'int')}
-        """
-        raise NotImplementedError()
-
-    # def parse_line(self):
-    #     """框架内部使用字典描述每个样本，字典的key为inputs_attr，value为每个input对应的符合attr描述的值。
-    #         该函数负责将文本行解析成符合inputs_attr描述的字典类型的样本。默认的parse_line方法会读取json格式的数据集文件，数据集的每一行为json格式描述的样本。
-    #         用户可通过对该方法的继承改写来适配不同格式的数据集，例如csv格式甚至tfrecord文件。
-    #         """
-    #     raise NotImplementedError()
-    # 
-    # def tokenize(self, line):
-    #     """框架中内置了word piece tokenizer等分词器，用户可通过修改tokenizer超参数来制定使用的分词器，若内置的分词器均无法满足需求，用户可通过对该方法的继承改写来自定义分词器。
-    #         Args:
-    #             - line: a unicode string. 
-    #         Return:
-    #             a list of tokens
-    #         """
-    #     raise NotImplementedError()
-    
-    def iterator(self):
-        """数据集遍历接口，注意，当数据集遍历到尾部时该接口应自动完成指针重置，即重新从数据集头部开始新的遍历。
-        Yield:
-            (dict) elements that meet the requirements in output_templete
-        """
-        raise NotImplementedError()
-
-    @property
-    def num_examples(self):
-        """数据集中的样本数量，即每个epoch中iterator所生成的样本数。注意，使用滑动窗口等可能导致数据集样本数发生变化的策略时，该接口应返回runtime阶段的实际样本数。"""
-        raise NotImplementedError()
-
-
-
-class backbone(object):
-    """interface of backbone model."""
-
-    def __init__(self, config, phase):
-        """
-        Args:
-            config: dict类型。描述了 多任务配置文件+预训练模型配置文件 中定义超参数
-            phase: str类型。运行阶段，目前支持train和predict
-            """
-        assert isinstance(config, dict)
-
-    @property
-    def inputs_attr(self):
-        """描述backbone从reader处需要得到的输入对象的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输入对象的属性描述。例如，
-            对于文本分类和匹配任务，bert backbone依赖的reader对象主要包含如下的对象
-                {"token_ids": ([-1, max_len], 'int64'),
-                 "input_ids": ([-1, max_len], 'int64'),
-                 "segment_ids": ([-1, max_len], 'int64'),
-                 "input_mask": ([-1, max_len], 'float32')}"""
-        raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述backbone输出对象的属性，包含各个对象的名字、shape以及数据类型。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输出对象的属性描述。例如，
-            对于文本分类和匹配任务，bert backbone的输出内容可能包含如下的对象
-                {"word_emb": ([-1, max_seqlen, word_emb_size], 'float32'),
-                 "sentence_emb": ([-1, hidden_size], 'float32'),
-                 "sim_vec": ([-1, hidden_size], 'float32')}""" 
-        raise NotImplementedError()
-
-    def build(self, inputs):
-        """建立backbone的计算图。将符合inputs_attr描述的静态图Variable输入映射成符合outputs_attr描述的静态图Variable输出。
-        Args:
-            inputs: dict类型。字典中包含inputs_attr中的对象名到计算图Variable的映射，inputs中至少会包含inputs_attr中定义的对象
-        Return:
-           需要输出的计算图变量，输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-            """
-        raise NotImplementedError()
-
-
-
-
-class task_paradigm(object):
-
-    def __init__(self, config, phase, backbone_config):
-        """
-            config: dict类型。描述了 任务实例(task instance)+多任务配置文件 中定义超参数
-            phase: str类型。运行阶段，目前支持train和predict
-            """
-
-    @property
-    def inputs_attrs(self):
-        """描述task_layer需要从reader, backbone等输入对象集合所读取到的输入对象的属性，第一级key为对象集和的名字，如backbone，reader等（后续会支持更灵活的输入），第二级key为对象集和中各对象的属性，包括对象的名字，shape和dtype。当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个对象集及其输入对象的属性描述。"""
-        raise NotImplementedError()
-
-    @property
-    def outputs_attr(self):
-        """描述task输出对象的属性，包括对象的名字，shape和dtype。输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-        当某个对象为标量数据类型（如str, int, float等）时，shape设置为空列表[]，当某个对象的某个维度长度可变时，shape中的相应维度设置为-1。
-        Return:
-            dict类型。对各个输入对象的属性描述。注意，训练阶段必须包含名为loss的输出对象。
-            """
-
-        raise NotImplementedError()
-
-    @property
-    def epoch_inputs_attrs(self):
-        return {}
-
-    def build(self, inputs, scope_name=""):
-        """建立task_layer的计算图。将符合inputs_attrs描述的来自各个对象集的静态图Variables映射成符合outputs_attr描述的静态图Variable输出。
-        Args:
-            inputs: dict类型。字典中包含inputs_attrs中的对象名到计算图Variable的映射，inputs中至少会包含inputs_attr中定义的对象
-        Return:
-           需要输出的计算图变量，输出对象会被加入到fetch_list中，从而在每个训练/推理step时得到runtime的计算结果，该计算结果会被传入postprocess方法中供用户处理。
-
-        """
-        raise NotImplementedError()
-
-    def postprocess(self, rt_outputs):
-        """每个训练或推理step后针对当前batch的task_layer的runtime计算结果进行相关后处理。注意，rt_outputs除了包含build方法，还自动包含了loss的计算结果。"""
-        pass
-        
-    def epoch_postprocess(self, post_inputs):
-        pass
-

paddlepalm/reader/cls.py

@@ -24,15 +24,15 @@ class ClassifyReader(Reader):
     For tsv format, training dataset file should have two header areas, i.e., `label` and `text`, and test set only requires `text` area. For example,
 
     ```
-    label	text
-		1	Today is a good day.
-		0	Such a terriable day!
-		1	I feel lucky to meet you, dear.
-		1	He likes sunshine and I like him :).
-		0	JUST! GO! OUT!
-		```
-
-		CAUTIOUS: The first line of the file must be header! And areas are splited by tab (\\t).
+    label [TAB] text
+    1 [TAB] Today is a good day.
+    0 [TAB] Such a terriable day!
+    1 [TAB] I feel lucky to meet you, dear.
+    1 [TAB] He likes sunshine and I like him :).
+    0 [TAB] JUST! GO! OUT!
+    ```
+
+    CAUTIOUS: The first line of the file must be header! And areas are splited by tab (\\t).
 
     """
     

paddlepalm/reader/match.py

@@ -18,6 +18,31 @@ from paddlepalm.reader.utils.reader4ernie import ClassifyReader as CLSReader
 
 
 class MatchReader(Reader):
+    """
+    The reader completes the loading and processing of matching-like task (e.g, query-query, question-answer, text similarity, natural language inference) dataset. Supported file format: tsv. 
+    
+    For pointwise learning strategy, there should be two fields in training dataset file, i.e., `text_a`, `text_b` and `label`. For pairwise learning, there should exist three fields, i.e., `text_a`, `text_b` and `text_b_neg`. For predicting, only `text_a` and `text_b` are required.
+    
+    A pointwise learning case shows as follows:
+    ```
+    label [TAB] text_a [TAB] text_b
+    1 [TAB] Today is a good day. [TAB] what a nice day!
+    0 [TAB] Such a terriable day! [TAB] There is a dog.
+    1 [TAB] I feel lucky to meet you, dear. [TAB] You are my lucky, darling.
+    1 [TAB] He likes sunshine and I like him :). [TAB] I like him. He like sunshine.
+    0 [TAB] JUST! GO! OUT! [TAB] Come in please.
+    ```
+    A pairwise learning case shows as follows:
+    text_a [TAB] text_b [TAB] text_b_neg
+    Today is a good day. [TAB] what a nice day! [TAB] terriable day!
+    Such a terriable day! [TAB] So terriable today! [TAB] There is a dog.
+    I feel lucky to meet you, dear. [TAB] You are my lucky, darling. [TAB] 
+    He likes sunshine and I like him :). [TAB] I like him. He like sunshine.
+    JUST! GO! OUT! [TAB] Come in please.
+
+    CAUTIOUS: The first line of the file must be header! And areas are splited by tab (\\t).
+
+    """
     
     def __init__(self, vocab_path, max_len, tokenizer='wordpiece', lang='en', seed=None, \
         do_lower_case=False, learning_strategy='pointwise', phase='train', dev_count=1, print_prefix=''):  # 需要什么加什么

paddlepalm/task_instance.py

-# -*- coding: UTF-8 -*-
-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from paddlepalm.interface import reader as base_reader
-from paddlepalm.interface import task_paradigm as base_paradigm
-import os
-import json
-from paddle import fluid
-import importlib
-from paddlepalm.default_settings import *
-
-
-def check_req_args(conf, name):
-    assert 'reader' in conf, name+': reader is required to build TaskInstance.'
-    assert 'paradigm' in conf, name+': paradigm is required to build TaskInstance.'
-    assert 'train_file' in conf or 'pred_file' in conf, name+': at least train_file or pred_file should be provided to build TaskInstance.'
-
-
-class TaskInstance(object):
-    
-    def __init__(self, name, id, config, verbose=True):
-        self._name = name
-        self._config = config
-        self._verbose = verbose
-
-        check_req_args(config, name)
-
-        # parse Reader and Paradigm
-        reader_name = config['reader']
-        reader_mod = importlib.import_module(READER_DIR + '.' + reader_name)
-        Reader = getattr(reader_mod, 'Reader')
-
-        parad_name = config['paradigm']
-        parad_mod = importlib.import_module(PARADIGM_DIR + '.' + parad_name)
-        Paradigm = getattr(parad_mod, 'TaskParadigm')
-
-        self._Reader = Reader
-        self._Paradigm = Paradigm
-
-        self._save_infermodel_path = os.path.join(self._config['save_path'], self._name, 'infer_model')
-        self._save_ckpt_path = os.path.join(self._config['save_path'], 'ckpt')
-        self._save_infermodel_every_n_steps = config.get('save_infermodel_every_n_steps', -1)
-
-        # following flags can be fetch from instance config file
-        self._is_target = config.get('is_target', True)
-        self._first_target = config.get('is_first_target', False)
-        self._task_reuse_scope = config.get('task_reuse_scope', name)
-
-        self._feeded_var_names = None
-        self._target_vars = None
-
-        # training process management
-        self._mix_ratio = None
-        self._expected_train_steps = None
-        self._expected_train_epochs = None
-        self._steps_pur_epoch = None
-        self._cur_train_epoch = 0
-        self._cur_train_step = 0
-        self._train_finish = False
-
-        # 存放不同运行阶段（train，eval，pred）的数据集reader，key为phase，value为Reader实例
-        self._reader = {'train': None, 'eval': None, 'predict': None}
-        self._input_layer = None
-        self._inputname_to_varname = {}
-        self._task_layer = {'train': None, 'eval': None, 'predict': None}
-        self._pred_input_name_list = []
-        self._pred_input_varname_list = []
-        self._pred_fetch_name_list = []
-        self._pred_fetch_var_list = []
-
-        self._exe = fluid.Executor(fluid.CPUPlace())
-
-        self._save_protocol = {
-            'input_names': 'self._pred_input_name_list',
-            'input_varnames': 'self._pred_input_varname_list',
-            'fetch_list': 'self._pred_fetch_name_list'}
-
-
-    def build_task_layer(self, net_inputs, phase, scope=""):
-        output_vars = self._task_layer[phase].build(net_inputs, scope_name=scope)
-        if phase == 'predict':
-            if output_vars is not None:
-                self._pred_fetch_name_list, self._pred_fetch_var_list = zip(*output_vars.items())
-            else:
-                self._pred_fetch_name_list = []
-                self._pred_fetch_var_list = []
-        return output_vars
-
-    def postprocess(self, rt_outputs, phase):
-        return self._task_layer[phase].postprocess(rt_outputs)
-
-    def epoch_postprocess(self, epoch_inputs, phase):
-        return self._task_layer[phase].epoch_postprocess(epoch_inputs)
-    
-    def save(self, suffix=''):
-        dirpath = self._save_infermodel_path + suffix
-        self._pred_input_varname_list = [str(i) for i in self._pred_input_varname_list]
-
-        # fluid.io.save_inference_model(dirpath, self._pred_input_varname_list, self._pred_fetch_var_list, self._exe, export_for_deployment = True)
-        prog = fluid.default_main_program().clone()
-        fluid.io.save_inference_model(dirpath, self._pred_input_varname_list, self._pred_fetch_var_list, self._exe, prog)
-
-        conf = {}
-        for k, strv in self._save_protocol.items(): 
-            d = None
-            v = locals()
-            exec('d={}'.format(strv), globals(), v)
-            conf[k] = v['d']
-        with open(os.path.join(dirpath, '__conf__'), 'w') as writer:
-            writer.write(json.dumps(conf, indent=1))
-        print(self._name + ': inference model saved at ' + dirpath)
-
-    def load(self, infer_model_path=None):
-        if infer_model_path is None:
-            infer_model_path = self._save_infermodel_path
-        for k,v in json.load(open(os.path.join(infer_model_path, '__conf__'))).items(): 
-            strv = self._save_protocol[k]
-            exec('{}=v'.format(strv))
-        pred_prog, self._pred_input_varname_list, self._pred_fetch_var_list = \
-            fluid.io.load_inference_model(infer_model_path, self._exe)
-        print(self._name+': inference model loaded from ' + infer_model_path)
-        return pred_prog
-
-    @property
-    def name(self):
-        return self._name
-
-    @property
-    def Reader(self):
-        return self._Reader
-
-    # @Reader.setter
-    # def Reader(self, cls):
-    #     assert base_reader.__name__ == cls.__bases__[-1].__name__, \
-    #         "expect: {}, receive: {}.".format(base_reader.__name__, \
-    #                                           cls.__bases__[-1].__name__)
-    #     self._Reader = cls
-
-    @property
-    def Paradigm(self):
-        return self._Paradigm
-
-    # @Paradigm.setter
-    # def Paradigm(self, cls):
-    #     assert base_paradigm.__name__ == cls.__bases__[-1].__name__, \
-    #         "expect: {}, receive: {}.".format(base_paradigm.__name__, \
-    #                                           cls.__bases__[-1].__name__)
-    #     self._Paradigm = cls
-
-    @property
-    def config(self):
-        return self._config
-
-    @property
-    def reader(self):
-        return self._reader
-
-    @property
-    def pred_input(self):
-        return zip(*[self._pred_input_name_list, self._pred_input_varname_list])
-
-    @pred_input.setter
-    def pred_input(self, val):
-        assert isinstance(val, dict)
-        self._pred_input_name_list, self._pred_input_varname_list = \
-            zip(*[[k, v.name] for k,v in val.items()])
-
-    @property
-    def pred_fetch_list(self):
-        return [self._pred_fetch_name_list, self._pred_fetch_var_list]
-
-    @property
-    def task_layer(self):
-        return self._task_layer
-
-    @property
-    def is_first_target(self):
-        return self._is_first_target
-
-    @is_first_target.setter
-    def is_first_target(self, value):
-        self._is_first_target = bool(value)
-        if self._is_first_target:
-            assert self._is_target, "ERROR: only target task could be set as main task."
-        if self._verbose and self._is_first_target:
-            print("{}: set as main task".format(self._name))
-
-    @property
-    def is_target(self):
-        if self._is_target is not None:
-            return self._is_target
-        else:
-            raise ValueError("{}: is_target is None".format(self._name))
-
-    @is_target.setter
-    def is_target(self, value):
-        self._is_target = bool(value)
-        if self._verbose:
-            if self._is_target:
-                print('{}: set as target task.'.format(self._name))
-            else:
-                print('{}: set as aux task.'.format(self._name))
-
-    @property
-    def mix_ratio(self):
-        if self._mix_ratio is not None:
-            return self._mix_ratio
-        else:
-            raise ValueError("{}: mix_ratio is None".format(self._name))
-
-    @mix_ratio.setter
-    def mix_ratio(self, value):
-        self._mix_ratio = float(value)
-        if self._verbose:
-            print('{}: mix_ratio is set to {}'.format(self._name, self._mix_ratio))
-
-    @property
-    def save_infermodel_every_n_steps(self):
-        return self._save_infermodel_every_n_steps
-
-    @property
-    def expected_train_steps(self):
-        return self._expected_train_steps
-
-    @expected_train_steps.setter
-    def expected_train_steps(self, value):
-        self._expected_train_steps = value
-        self._expected_train_epochs = value / float(self._steps_pur_epoch)
-
-    @property
-    def expected_train_epochs(self):
-        return self._expected_train_epochs
-
-    @property
-    def cur_train_epoch(self):
-        return self._cur_train_epoch
-
-    @cur_train_epoch.setter
-    def cur_train_epoch(self, value):
-        self._cur_train_epoch = value
-
-    @property
-    def cur_train_step(self):
-        return self._cur_train_step
-
-    @cur_train_step.setter
-    def cur_train_step(self, value):
-        self._cur_train_step = value
-        if self._cur_train_step > self._steps_pur_epoch:
-            self._cur_train_epoch += 1
-            self._cur_train_step = 1
-        if self._is_target and self._cur_train_step + self._cur_train_epoch * self._steps_pur_epoch >= self._expected_train_steps:
-            self._train_finish = True
-
-    @property
-    def steps_pur_epoch(self):
-        return self._steps_pur_epoch
-
-    @steps_pur_epoch.setter
-    def steps_pur_epoch(self, value):
-        self._steps_pur_epoch = value
-
-    @property
-    def train_finish(self):
-        return self._train_finish
-
-    @property
-    def task_reuse_scope(self):
-        if self._task_reuse_scope is not None:
-            return self._task_reuse_scope
-        else:
-            raise ValueError("{}: task_reuse_scope is None".format(self._name))
-
-    @task_reuse_scope.setter
-    def task_reuse_scope(self, scope_name):
-        self._task_reuse_scope = str(scope_name)
-        if self._verbose:
-            print('{}: task_reuse_scope is set to {}'.format(self._name, self._task_reuse_scope))
-
-
-
-
-
-        
-
-def check_instances(insts):
-    """to check ids, first_target"""
-    pass
-
-def _check_ids():
-    pass
-
-def _check_targets():
-    pass
-
-def _check_reuse_scopes():
-    pass

setup.cfg

 [metadata]
 
-name = paddle-palm
+name = paddlepalm
 
 author = zhangyiming
 author_email = zhangyiming04@baidu.com
 
-version = 1.2
+version = 1.0.0
 
-description = Paddle-PALM
+description = PaddlePALM
 long_description = file: README.md
 long_description_content_type = text/markdown
 
@@ -27,6 +27,8 @@ classifier =
 keywords =
     paddlepaddle
     paddle
+    nlp
+    pretrain
     multi-task-learning
 
 [options]

setup.py

@@ -18,7 +18,7 @@
 """
 Setup script.
 Authors: zhouxiangyang(zhouxiangyang@baidu.com)
-Date:    2019/09/29 21:00:01
+Date:    2020/1/22 12:00:01
 """
 import setuptools
 with open("README.md", "r") as fh:
@@ -28,10 +28,10 @@ setuptools.setup(
     version="1.0.0",
     author="PaddlePaddle",
     author_email="zhangyiming04@baidu.com",
-    description="A Multi-task Learning Lib for PaddlePaddle Users.",
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    url="https://github.com/PaddlePadd",
+    description="a flexible, general and easy-to-use NLP large-scale pretraining and multi-task learning framework.",
+    # long_description=long_description,
+    # long_description_content_type="text/markdown",
+    url="https://github.com/PaddlePaddle/PALM",
     # packages=setuptools.find_packages(),
     packages = ['paddlepalm', 
         'paddlepalm.backbone', 
@@ -39,16 +39,20 @@ setuptools.setup(
         'paddlepalm.optimizer',
         'paddlepalm.reader', 
         'paddlepalm.reader.utils', 
-        'paddlepalm.task_paradigm', 
+        'paddlepalm.head', 
+        'paddlepalm.distribute', 
+        'paddlepalm.lr_sched', 
         'paddlepalm.tokenizer', 
         'paddlepalm.utils'],
     package_dir={'paddlepalm':'./paddlepalm',
                  'paddlepalm.backbone':'./paddlepalm/backbone',
                  'paddlepalm.backbone.utils':'./paddlepalm/backbone/utils',
                  'paddlepalm.optimizer':'./paddlepalm/optimizer',
+                 'paddlepalm.lr_sched': './paddlepalm/lr_sched',
+                 'paddlepalm.distribute': './paddlepalm/distribute',
                  'paddlepalm.reader':'./paddlepalm/reader',
                  'paddlepalm.reader.utils':'./paddlepalm/reader/utils',
-                 'paddlepalm.task_paradigm':'./paddlepalm/task_paradigm',
+                 'paddlepalm.head':'./paddlepalm/head',
                  'paddlepalm.tokenizer':'./paddlepalm/tokenizer',
                  'paddlepalm.utils':'./paddlepalm/utils'},
     platforms = "any",
@@ -64,7 +68,7 @@ setuptools.setup(
             'Programming Language :: Python :: 3.7',
           ],
     install_requires = [
-        'paddlepaddle-gpu>=1.6.1'
+        'paddlepaddle-gpu>=1.6.3'
     ]
 )