Merge branch 'develop' into develop

README.md

@@ -19,7 +19,6 @@ English | [简体中文](./README.zh.md)
  * [Results](#results)
    * [Results on English Datasets](#results-on-english-datasets)
    * [Results on Chinese Datasets](#results-on-chinese-datasets)
- * [Release Notes](#release-notes)
  * [Communication](#communication)
  * [Usage](#usage)
 
@@ -615,14 +614,6 @@ LCQMC is a Chinese question semantic matching corpus published in COLING2018. [u
 BQ Corpus (Bank Question corpus) is a Chinese corpus for sentence semantic equivalence identification. This dataset was published in EMNLP 2018. [url: https://www.aclweb.org/anthology/D18-1536]
 ```
 
-## Release Notes
-
-- Aug 21, 2019: featuers update: fp16 finetuning, multiprocess finetining.
-- July 30, 2019: release ERNIE 2.0
-- Apr 10, 2019: update ERNIE_stable-1.0.1.tar.gz, update config and vocab
-- Mar 18, 2019: update ERNIE_stable.tgz
-- Mar 15, 2019: release ERNIE 1.0
-
 
 ## Communication
 
@@ -657,11 +648,12 @@ BQ Corpus (Bank Question corpus) is a Chinese corpus for sentence semantic equiv
      * [FAQ3: Is the  argument batch_size for one GPU card or for all GPU cards?](#faq3-is-the--argument-batch_size-for-one-gpu-card-or-for-all-gpu-cards)
      * [FAQ4: Can not find library: libcudnn.so. Please try to add the lib path to LD_LIBRARY_PATH.](#faq4-can-not-find-library-libcudnnso-please-try-to-add-the-lib-path-to-ld_library_path)
      * [FAQ5: Can not find library: libnccl.so. Please try to add the lib path to LD_LIBRARY_PATH.](#faq5-can-not-find-library-libncclso-please-try-to-add-the-lib-path-to-ld_library_path)
+     * [FQA6: Runtime error: `ModuleNotFoundError No module named propeller`](#faq6)
 
 
 ### Install PaddlePaddle
 
-This code base has been tested with Paddle Fluid 1.5.1 under Python2.
+This code base has been tested with Paddle Fluid 1.6 with Python 2/3.5+, since Paddle 1.6 has changed some of APIs, using version before 1.6 might have bug on NER tasks.
 
 **\*Important\*** When finished installing Paddle Fluid, remember to update LD_LIBRARY_PATH about CUDA, cuDNN, NCCL2, for more information on paddlepaddle setup, you can click [here](http://en.paddlepaddle.org/documentation/docs/en/1.5/beginners_guide/index_en.html) and [here](http://en.paddlepaddle.org/documentation/docs/en/1.5/beginners_guide/install/install_Ubuntu_en.html). Also, you can read FAQ at the end of this document when you encounter errors.
 
@@ -1010,6 +1002,12 @@ Export the path of cuda to LD_LIBRARY_PATH, e.g.: `export LD_LIBRARY_PATH=/home/
 
 Download [NCCL2](https://developer.nvidia.com/nccl/nccl-download), and export the library path to LD_LIBRARY_PATH, e.g.:`export LD_LIBRARY_PATH=/home/work/nccl/lib`
 
-#### FAQ6: Cannot malloc XXX MB GPU memory.
+### FAQ6: Runtime error: `ModuleNotFoundError No module named propeller`<a name="faq6"></a>
+
+you can import propeller to your PYTHONPATH by `export PYTHONPATH:./:$PYTHONPATH`
+
+
+#### FAQ7: Cannot malloc XXX MB GPU memory.
 
 Try to reduce the batch_size, reduce the max_seq_len and set FLAGS_eager_delete_tensor_gb=0.0
+

README.zh.md

@@ -19,7 +19,7 @@
  * [效果验证](#效果验证)
     * [中文效果验证](#中文效果验证)
     * [英文效果验证](#英文效果验证)
- * [开源记录](#开源记录)
+ * [ERNIE tiny](#ernie-tiny)
  * [技术交流](#技术交流)
  * [使用](#使用)
 
@@ -589,7 +589,6 @@ ERNIE 2.0 的英文效果验证在 GLUE 上进行。GLUE 评测的官方地址
 
 
 
-
 #### GLUE - 验证集结果
 
 | <strong>数据集</strong> | <strong>CoLA</strong> | <strong>SST-2</strong> | <strong>MRPC</strong> | <strong>STS-B</strong> | <strong>QQP</strong>  | <strong>MNLI-m</strong> | <strong>QNLI</strong> | <strong>RTE</strong>  |
@@ -617,11 +616,34 @@ ERNIE 2.0 的英文效果验证在 GLUE 上进行。GLUE 评测的官方地址
 由于 XLNet 暂未公布 GLUE 测试集上的单模型结果，所以我们只与 BERT 进行单模型比较。上表为ERNIE 2.0 单模型在 GLUE 测试集的表现结果。
 
 
-## 开源记录
-- 2019-07-30 发布 ERNIE 2.0
-- 2019-04-10 更新: update ERNIE_stable-1.0.1.tar.gz, 将模型参数、配置 ernie_config.json、vocab.txt 打包发布
-- 2019-03-18 更新: update ERNIE_stable.tgz
-- 2019-03-15 发布 ERNIE 1.0
+### ERNIE tiny
+
+为了提升ERNIE模型在实际工业应用中的落地能力，我们推出ERNIE-tiny模型。 
+
+![ernie_tiny](.metas/ernie_tiny.png)
+
+ERNIE-tiny作为小型化ERNIE，采用了以下4点技术，保证了在实际真实数据中将近4.3倍的预测提速。
+
+1. 浅：12层的ERNIE Base模型直接压缩为3层，线性提速4倍，但效果也会有较大幅度的下降；
+
+1. 胖：模型变浅带来的损失可通过hidden size的增大来弥补。由于fluid inference框架对于通用矩阵运算（gemm）的最后一维（hidden size）参数的不同取值会有深度的优化，因为将hidden size从768提升至1024并不会带来速度线性的增加；
+
+1. 短：ERNIE Tiny是首个开源的中文subword粒度的预训练模型。这里的短是指通过subword粒度替换字（char）粒度，能够明显地缩短输入文本的长度，而输入文本长度是和预测速度有线性相关。统计表明，在XNLI dev集上采用subword字典切分出来的序列长度比字表平均缩短40%；
+
+1. 萃：为了进一步提升模型的效果，ERNIE Tiny扮演学生角色，利用模型蒸馏的方式在Transformer层和Prediction层去学习教师模型ERNIE模型对应层的分布或输出，这种方式能够缩近ERNIE Tiny和ERNIE的效果差异。
+
+
+#### Benchmark
+
+ERNIE Tiny轻量级模型在公开数据集的效果如下所示，任务均值相对于ERNIE Base只下降了2.37%，但相对于“SOTA Before BERT”提升了8%。在延迟测试中，ERNIE Tiny能够带来4.3倍的速度提升
+（测试环境为：GPU P4，Paddle Inference C++ API，XNLI Dev集，最大maxlen=128，测试结果10次均值）
+
+|model|XNLI(acc)|LCQCM(acc)|CHNSENTICORP(acc)|NLPCC-DBQA(mrr/f1)|Average|Latency
+|--|--|--|--|--|--|--|
+|SOTA-before-ERNIE|68.3|83.4|92.2|72.01/-|78.98|-|
+|ERNIE2.0-base|79.7|87.9|95.5|95.7/85.3|89.70|633ms(1x)|
+|ERNIE-tiny-subword|75.1|86.1|95.2|92.9/78.6|87.33|146ms(4.3x)|
+
 
 ## 技术交流
 
@@ -646,6 +668,7 @@ ERNIE 2.0 的英文效果验证在 GLUE 上进行。GLUE 评测的官方地址
      * [序列标注任务](#序列标注任务)
         * [实体识别](#实体识别)
      * [阅读理解任务](#阅读理解任务-1)
+     * [ERNIE tiny](#tune-ernie-tiny)
   * [利用Propeller进行二次开发](#利用propeller进行二次开发)
   * [预训练 (ERNIE 1.0)](#预训练-ernie-10)
      * [数据预处理](#数据预处理)
@@ -666,7 +689,7 @@ ERNIE 2.0 的英文效果验证在 GLUE 上进行。GLUE 评测的官方地址
 
 ## PaddlePaddle安装
 
-本项目依赖于 Paddle Fluid 1.5，* 暂时不支持Paddle 1.6的使用 *，请参考[安装指南](http://www.paddlepaddle.org/#quick-start)进行安装。
+本项目依赖于 Paddle 1.6，* 由于Paddle 1.6版本相比之前版本有较大API改动，使用Paddle 1.6以前版本运行本代码库会导致序列标注等任务报错 *，请参考[安装指南](http://www.paddlepaddle.org/#quick-start)进行安装。
 
 **【重要】安装后，需要及时的将 CUDA、cuDNN、NCCL2 等动态库路径加入到环境变量 LD_LIBRARY_PATH 之中，否则训练过程中会报相关的库错误。具体的paddlepaddle配置细节请查阅[这里](http://en.paddlepaddle.org/documentation/docs/zh/1.5/beginners_guide/quick_start_cn.html)**
 
@@ -695,6 +718,7 @@ pip install -r requirements.txt
 | [ERNIE 1.0 中文 Base 模型(max_len=512)](https://ernie.bj.bcebos.com/ERNIE_1.0_max-len-512.tar.gz) | 包含预训练模型参数、词典 vocab.txt、模型配置 ernie_config.json|
 | [ERNIE 2.0 英文 Base 模型](https://ernie.bj.bcebos.com/ERNIE_Base_en_stable-2.0.0.tar.gz) | 包含预训练模型参数、词典 vocab.txt、模型配置 ernie_config.json|
 | [ERNIE 2.0 英文 Large 模型](https://ernie.bj.bcebos.com/ERNIE_Large_en_stable-2.0.0.tar.gz) | 包含预训练模型参数、词典 vocab.txt、模型配置 ernie_config.json|
+| [ERNIE tiny 中文模型](https://ernie.bj.bcebos.com/ernie_tiny.tar.gz)|包含预训练模型参数、词典 vocab.txt、模型配置 ernie_config.json 以及切词词表|
 
 
 
@@ -894,6 +918,16 @@ text_a  label
 [test evaluation] em: 88.061838, f1: 93.520152, avg: 90.790995, question_num: 3493
  ```
 
+
+### ERNIE tiny <a name="tune-ernie-tiny"></a>
+
+ERNIE tiny 模型采用了subword粒度输入，需要在数据前处理中加入切词(segmentation)并使用[sentence piece](https://github.com/google/sentencepiece)进行tokenization. 
+segmentation 以及 tokenization 需要使用的模型包含在了 ERNIE tiny 的[预训练模型文件](#预训练模型下载)中，分别是 `./subword/dict.wordseg.pickle` 和 `./subword/spm_cased_simp_sampled.model`.
+
+目前`./example/`下的代码针对 ERNIE tiny 的前处理进行了适配只需在脚本中通过 `--sentence_piece_model` 引入tokenization 模型，再通过 `--word_dict` 引入 segmentation 模型之后即可进行 ERNIE tiny 的 Fine-tune。
+对于命名实体识别类型的任务，为了跟输入标注对齐，ERNIE tiny 仍然采用中文单字粒度进行作为输入。因此使用 `./example/finetune_ner.py` 时只需要打开 `--use_sentence_piece_vocab` 即可。
+具体的使用方法可以参考[下节](#利用propeller进行二次开发).
+
 ## 利用Propeller进行二次开发
 
 [Propeller](./propeller/README.md) 是基于PaddlePaddle构建的一键式训练API，对于具备一定机器学习应用经验的开发者可以使用Propeller获得定制化开发体验。
@@ -947,7 +981,7 @@ cat input_file | python3 ./example/finetune_classifier.py --do_predict ... > out
 
 ### 数据预处理
 
-基于百科类、资讯类、论坛对话类数据构造具有上下文关系的句子对数据，利用百度内部词法分析工具对句对数据进行字、词、实体等不同粒度的切分，然后基于 [`tokenization.py`](tokenization.py) 中的 CharTokenizer 对切分后的数据进行 token 化处理，得到明文的 token 序列及切分边界，然后将明文数据根据词典 [`config/vocab.txt`](config/vocab.txt) 映射为 id 数据，在训练过程中，根据切分边界对连续的 token 进行随机 mask 操作；
+基于百科类、资讯类、论坛对话类数据构造具有上下文关系的句子对数据，利用百度内部词法分析工具对句对数据进行字、词、实体等不同粒度的切分，然后基于 [`tokenization.py`](./ernie/tokenization.py) 中的 CharTokenizer 对切分后的数据进行 token 化处理，得到明文的 token 序列及切分边界，然后将明文数据根据词典 [`config/vocab.txt`](config/vocab.txt) 映射为 id 数据，在训练过程中，根据切分边界对连续的 token 进行随机 mask 操作；
 
 我们给出了 id 化后的部分训练数据：[`data/demo_train_set.gz`](./data/demo_train_set.gz)、和测试数据：[`data/demo_valid_set.gz`](./data/demo_valid_set.gz)，每行数据为1个训练样本，示例如下:
 
@@ -1022,7 +1056,7 @@ ERNIE提供了通过数据蒸馏从而达到模型压缩、加速的开发套件
 如果您采用 `propeller` 完成finetune，则 `BestInferenceExporter` 会在finetune过程中根据预测指标，挑最好的模型生成 inference_model .
 
 ### 在线预测
-随后您可以使用[PaddleInference C++ API](https://www.paddlepaddle.org.cn/documentation/docs/zh/advanced_usage/deploy/inference/native_infer.html)将模型的前向预测代码联编到您的生产环境中。或者您可以使用我们为您构建好的python预测引擎来完成一个简单的服务。执行如下指令，便可以开启一个propeller server：
+随后您可以使用[ERNIE fast inference C++ API](./inference/README.md)将模型的前向预测代码联编到您的生产环境中。或者您可以使用我们为您构建好的python预测引擎来完成一个简单的服务。执行如下指令，便可以开启一个propeller server：
 
 ```script
 python -m propeller.tools.start_server -m /path/to/saved/model -p 8888
@@ -1099,7 +1133,7 @@ python -u infer_classifyer.py \
 
 需要先下载 [NCCL](https://developer.nvidia.com/nccl/nccl-download)，然后在 LD_LIBRARY_PATH 中添加 NCCL 库的路径，如`export LD_LIBRARY_PATH=/home/work/nccl/lib`
 
-### FQA6: 运行报错`ModuleNotFoundError: No module named 'propeller'`<a name="faq6"></a>
+### FAQ6: 运行报错`ModuleNotFoundError: No module named 'propeller'`<a name="faq6"></a>
 
 您可以通过`export PYTHONPATH=./:$PYTHONPATH`的方式引入Propeller.
 

ernie/batching.py

@@ -208,7 +208,7 @@ def pad_batch_data(insts,
 
     if return_seq_lens:
         seq_lens = np.array([len(inst) for inst in insts])
-        return_list += [seq_lens.astype("int64").reshape([-1, 1])]
+        return_list += [seq_lens.astype("int64").reshape([-1])]
 
     return return_list if len(return_list) > 1 else return_list[0]
 

ernie/ernie_encoder.py

@@ -22,13 +22,16 @@ import argparse
 import numpy as np
 import multiprocessing
 
+import logging
 import paddle.fluid as fluid
 
 import reader.task_reader as task_reader
-from model.ernie import ErnieConfig, ErnieModel
-from utils.args import ArgumentGroup, print_arguments
+from model.ernie_v1 import ErnieConfig, ErnieModel
+from utils.args import ArgumentGroup, print_arguments, prepare_logger
 from utils.init import init_pretraining_params
 
+log = logging.getLogger()
+
 # yapf: disable
 parser = argparse.ArgumentParser(__doc__)
 model_g = ArgumentGroup(parser, "model", "model configuration and paths.")
@@ -52,24 +55,21 @@ run_type_g.add_arg("use_cuda",                     bool,   True,  "If set, use G
 
 
 def create_model(args, pyreader_name, ernie_config):
-    pyreader = fluid.layers.py_reader(
-        capacity=50,
-        shapes=[[-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, 1]],
-        dtypes=['int64', 'int64', 'int64', 'int64', 'float', 'int64'],
-        lod_levels=[0, 0, 0, 0, 0, 0],
-        name=pyreader_name,
-        use_double_buffer=True)
-
-    (src_ids, sent_ids, pos_ids, task_ids, input_mask,
-     seq_lens) = fluid.layers.read_file(pyreader)
+    src_ids = fluid.layers.data(name='1', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    sent_ids = fluid.layers.data(name='2', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    pos_ids = fluid.layers.data(name='3', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    task_ids = fluid.layers.data(name='4', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    input_mask = fluid.layers.data(name='5', shape=[-1, args.max_seq_len, 1], dtype='float32')
+    seq_lens = fluid.layers.data(name='8', shape=[-1], dtype='int64')
+
+    pyreader = fluid.io.DataLoader.from_generator(feed_list=[src_ids, sent_ids, pos_ids, task_ids, input_mask, seq_lens], 
+            capacity=70,
+            iterable=False)
 
     ernie = ErnieModel(
         src_ids=src_ids,
         position_ids=pos_ids,
         sentence_ids=sent_ids,
-        task_ids=task_ids,
         input_mask=input_mask,
         config=ernie_config)
 
@@ -143,7 +143,7 @@ def main(args):
     exec_strategy = fluid.ExecutionStrategy()
     exec_strategy.num_threads = dev_count
 
-    pyreader.decorate_tensor_provider(data_generator)
+    pyreader.set_batch_generator(data_generator)
     pyreader.start()
 
     total_cls_emb = []
@@ -167,6 +167,11 @@ def main(args):
     total_cls_emb = np.concatenate(total_cls_emb)
     total_top_layer_emb = np.concatenate(total_top_layer_emb)
 
+    if not os.path.exists(args.output_dir):
+        os.mkdir(args.output_dir)
+    else:
+        raise RuntimeError('output dir exists: %s' % args.output_dir)
+
     with open(os.path.join(args.output_dir, "cls_emb.npy"),
               "wb") as cls_emb_file:
         np.save(cls_emb_file, total_cls_emb)
@@ -176,6 +181,7 @@ def main(args):
 
 
 if __name__ == '__main__':
+    prepare_logger(log)
     args = parser.parse_args()
     print_arguments(args)
 

ernie/finetune/classifier.py

@@ -39,34 +39,22 @@ def create_model(args,
                  is_classify=False,
                  is_regression=False,
                  ernie_version="1.0"):
+
+    src_ids = fluid.layers.data(name='eval_placeholder_0', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    sent_ids = fluid.layers.data(name='eval_placeholder_1', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    pos_ids = fluid.layers.data(name='eval_placeholder_2', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    input_mask = fluid.layers.data(name='eval_placeholder_3', shape=[-1, args.max_seq_len, 1], dtype='float32')
+    task_ids = fluid.layers.data(name='eval_placeholder_4', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    qids = fluid.layers.data(name='eval_placeholder_5', shape=[-1, 1], dtype='int64')
+
     if is_classify:
-        pyreader = fluid.layers.py_reader(
-            capacity=50,
-            shapes=[[-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                    [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                    [-1, args.max_seq_len, 1], [-1, 1], [-1, 1]],
-            dtypes=[
-                'int64', 'int64', 'int64', 'int64', 'float32', 'int64', 'int64'
-            ],
-            lod_levels=[0, 0, 0, 0, 0, 0, 0],
-            name=task_name + "_" + pyreader_name,
-            use_double_buffer=True)
+        labels = fluid.layers.data(name='6', shape=[-1, 1], dtype='int64')
     elif is_regression:
-        pyreader = fluid.layers.py_reader(
-            capacity=50,
-            shapes=[[-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                    [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                    [-1, args.max_seq_len, 1], [-1, 1], [-1, 1]],
-            dtypes=[
-                'int64', 'int64', 'int64', 'int64', 'float32', 'float32',
-                'int64'
-            ],
-            lod_levels=[0, 0, 0, 0, 0, 0, 0],
-            name=task_name + "_" + pyreader_name,
-            use_double_buffer=True)
-
-    (src_ids, sent_ids, pos_ids, task_ids, input_mask, labels,
-     qids) = fluid.layers.read_file(pyreader)
+        labels = fluid.layers.data(name='6', shape=[-1, 1], dtype='float32')
+
+    pyreader = fluid.io.DataLoader.from_generator(feed_list=[src_ids, sent_ids, pos_ids, task_ids, input_mask, labels, qids], 
+            capacity=70, 
+            iterable=False)
 
     ernie = ErnieModel(
         src_ids=src_ids,
@@ -99,7 +87,7 @@ def create_model(args,
         else:
             probs = logits
         feed_targets_name = [
-            src_ids.name, sent_ids.name, pos_ids.name, input_mask.name
+            src_ids.name, sent_ids.name,  pos_ids.name, input_mask.name
         ]
         if ernie_version == "2.0":
             feed_targets_name += [task_ids.name]

ernie/finetune/mrc.py

@@ -40,20 +40,18 @@ import tokenization
 log = logging.getLogger(__name__)
 
 def create_model(args, pyreader_name, ernie_config, is_training):
-    pyreader = fluid.layers.py_reader(
-        capacity=50,
-        shapes=[[-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, 1], [-1, 1], [-1, 1]],
-        dtypes=[
-            'int64', 'int64', 'int64', 'int64', 'float32', 'int64', 'int64',
-            'int64'
-        ],
-        lod_levels=[0, 0, 0, 0, 0, 0, 0, 0],
-        name=pyreader_name,
-        use_double_buffer=True)
-    (src_ids, sent_ids, pos_ids, task_ids, input_mask, start_positions,
-     end_positions, unique_id) = fluid.layers.read_file(pyreader)
+    src_ids = fluid.layers.data(name='1', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    pos_ids = fluid.layers.data(name='2', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    sent_ids= fluid.layers.data(name='3', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    task_ids= fluid.layers.data(name='4', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    input_mask = fluid.layers.data(name='5', shape=[-1, args.max_seq_len, 1], dtype='float32')
+    start_positions = fluid.layers.data(name='6', shape=[-1, 1], dtype='int64')
+    end_positions = fluid.layers.data(name='7', shape=[-1, 1], dtype='int64')
+    unique_id = fluid.layers.data(name='8', shape=[-1, 1], dtype='int64')
+
+    pyreader = fluid.io.DataLoader.from_generator(feed_list=[
+        src_ids, sent_ids, pos_ids, task_ids, input_mask, start_positions,
+        end_positions, unique_id], capacity=50, iterable=False)
 
     ernie = ErnieModel(
         src_ids=src_ids,

ernie/finetune/sequence_label.py

@@ -36,20 +36,17 @@ from model.ernie import ErnieModel
 log = logging.getLogger(__name__)
 
 def create_model(args, pyreader_name, ernie_config, is_prediction=False):
-    pyreader = fluid.layers.py_reader(
-        capacity=50,
-        shapes=[[-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1], [-1, 1]],
-        dtypes=[
-            'int64', 'int64', 'int64', 'int64', 'float32', 'int64', 'int64'
-        ],
-        lod_levels=[0, 0, 0, 0, 0, 0, 0],
-        name=pyreader_name,
-        use_double_buffer=True)
-
-    (src_ids, sent_ids, pos_ids, task_ids, input_mask, labels,
-     seq_lens) = fluid.layers.read_file(pyreader)
+    src_ids = fluid.layers.data(name='1', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    sent_ids = fluid.layers.data(name='2', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    pos_ids = fluid.layers.data(name='3', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    task_ids = fluid.layers.data(name='4', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    input_mask = fluid.layers.data(name='5', shape=[-1, args.max_seq_len, 1], dtype='float32')
+    labels = fluid.layers.data(name='7', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    seq_lens = fluid.layers.data(name='8', shape=[-1], dtype='int64')
+
+    pyreader = fluid.io.DataLoader.from_generator(feed_list=[src_ids, sent_ids, pos_ids, task_ids, input_mask, labels, seq_lens], 
+            capacity=70,
+            iterable=False)
 
     ernie = ErnieModel(
         src_ids=src_ids,

ernie/infer_classifyer.py

@@ -130,6 +130,7 @@ def main(args):
     if not args.use_cuda:
         log.info("disable gpu")
         config.disable_gpu()
+        config.switch_ir_optim(True) 
     else:
         log.info("using gpu")
         config.enable_use_gpu(1024)
@@ -162,8 +163,7 @@ def main(args):
 
         # parse outputs
         output = outputs[0]
-        output_data = output.data.float_data()
-        batch_result  = np.array(output_data).reshape(output.shape)
+        batch_result  = output.as_ndarray()
         for single_example_probs in batch_result:
             print('\t'.join(map(str, single_example_probs.tolist())))
             index += 1
@@ -173,17 +173,7 @@ def main(args):
 def array2tensor(ndarray):
     """ convert numpy array to PaddleTensor"""
     assert isinstance(ndarray, np.ndarray), "input type must be np.ndarray"
-    tensor = PaddleTensor()
-    tensor.name = "data"
-    tensor.shape = ndarray.shape
-    if "float" in str(ndarray.dtype):
-        tensor.dtype = PaddleDType.FLOAT32
-    elif "int" in str(ndarray.dtype):
-        tensor.dtype = PaddleDType.INT64
-    else:
-        raise ValueError("{} type ndarray is unsupported".format(tensor.dtype))
-
-    tensor.data = PaddleBuf(ndarray.flatten().tolist())
+    tensor = PaddleTensor(data=ndarray)
     return tensor
 
 if __name__ == '__main__':

ernie/model/ernie.py

@@ -24,6 +24,7 @@ import six
 import logging
 import paddle.fluid as fluid
 from io import open
+from paddle.fluid.layers import core
 
 from model.transformer_encoder import encoder, pre_process_layer
 
@@ -85,8 +86,8 @@ class ErnieModel(object):
         self._pos_emb_name = "pos_embedding"
         self._sent_emb_name = "sent_embedding"
         self._task_emb_name = "task_embedding"
-        self._dtype = "float16" if use_fp16 else "float32"
-        self._emb_dtype = "float32"
+        self._dtype = core.VarDesc.VarType.FP16 if use_fp16 else core.VarDesc.VarType.FP32
+        self._emb_dtype = core.VarDesc.VarType.FP32
 
         # Initialize all weigths by truncated normal initializer, and all biases
         # will be initialized by constant zero by default.
@@ -138,7 +139,7 @@ class ErnieModel(object):
         emb_out = pre_process_layer(
             emb_out, 'nd', self._prepostprocess_dropout, name='pre_encoder')
 
-        if self._dtype == "float16":
+        if self._dtype == core.VarDesc.VarType.FP16:
             emb_out = fluid.layers.cast(x=emb_out, dtype=self._dtype)
             input_mask = fluid.layers.cast(x=input_mask, dtype=self._dtype)
         self_attn_mask = fluid.layers.matmul(
@@ -167,7 +168,7 @@ class ErnieModel(object):
             postprocess_cmd="dan",
             param_initializer=self._param_initializer,
             name='encoder')
-        if self._dtype == "float16":        
+        if self._dtype == core.VarDesc.VarType.FP16:
             self._enc_out = fluid.layers.cast(
                 x=self._enc_out, dtype=self._emb_dtype)
 

ernie/model/ernie_v1.py

@@ -24,6 +24,7 @@ import logging
 import six
 import paddle.fluid as fluid
 from io import open
+from paddle.fluid.layers import core
 
 from model.transformer_encoder import encoder, pre_process_layer
 
@@ -76,7 +77,7 @@ class ErnieModel(object):
         self._word_emb_name = "word_embedding"
         self._pos_emb_name = "pos_embedding"
         self._sent_emb_name = "sent_embedding"
-        self._dtype = "float16" if use_fp16 else "float32"
+        self._dtype = core.VarDesc.VarType.FP16 if use_fp16 else core.VarDesc.VarType.FP32
 
         # Initialize all weigths by truncated normal initializer, and all biases
         # will be initialized by constant zero by default.
@@ -114,7 +115,7 @@ class ErnieModel(object):
         emb_out = pre_process_layer(
             emb_out, 'nd', self._prepostprocess_dropout, name='pre_encoder')
 
-        if self._dtype == "float16":
+        if self._dtype == core.VarDesc.VarType.FP16:
             input_mask = fluid.layers.cast(x=input_mask, dtype=self._dtype)
         self_attn_mask = fluid.layers.matmul(
             x=input_mask, y=input_mask, transpose_y=True)

ernie/run_classifier.py

@@ -228,7 +228,7 @@ def main(args):
             num_trainers=nccl2_num_trainers,
             trainer_id=nccl2_trainer_id)
 
-        train_pyreader.decorate_tensor_provider(train_data_generator)
+        train_pyreader.set_batch_generator(train_data_generator)
     else:
         train_exe = None
 
@@ -349,7 +349,7 @@ def main(args):
 
     # final eval on dianostic, hack for glue-ax
     if args.diagnostic:
-        test_pyreader.decorate_tensor_provider(
+        test_pyreader.set_batch_generator(
             reader.data_generator(
                 args.diagnostic,
                 batch_size=args.batch_size,
@@ -380,7 +380,7 @@ def evaluate_wrapper(args, reader, exe, test_prog, test_pyreader, graph_vars,
     # evaluate dev set
     batch_size = args.batch_size if args.predict_batch_size is None else args.predict_batch_size
     for ds in args.dev_set.split(','):
-        test_pyreader.decorate_tensor_provider(
+        test_pyreader.set_batch_generator(
             reader.data_generator(
                 ds,
                 batch_size=batch_size,
@@ -409,7 +409,7 @@ def predict_wrapper(args, reader, exe, test_prog, test_pyreader, graph_vars,
     batch_size = args.batch_size if args.predict_batch_size is None else args.predict_batch_size
 
     for test_f, save_f in zip(test_sets, save_dirs):
-        test_pyreader.decorate_tensor_provider(
+        test_pyreader.set_batch_generator(
             reader.data_generator(
                 test_f,
                 batch_size=batch_size,

ernie/run_mrc.py

@@ -228,7 +228,7 @@ def main(args):
             num_trainers=nccl2_num_trainers,
             trainer_id=nccl2_trainer_id)
 
-        train_pyreader.decorate_tensor_provider(train_data_generator)
+        train_pyreader.set_batch_generator(train_data_generator)
     else:
         train_exe = None
 
@@ -272,7 +272,7 @@ def main(args):
 
                 if steps % args.validation_steps == 0:
                     if args.do_val:
-                        test_pyreader.decorate_tensor_provider(
+                        test_pyreader.set_batch_generator(
                             reader.data_generator(
                                 args.dev_set,
                                 batch_size=args.batch_size,
@@ -291,7 +291,7 @@ def main(args):
                             args=args)
 
                     if args.do_test:
-                        test_pyreader.decorate_tensor_provider(
+                        test_pyreader.set_batch_generator(
                             reader.data_generator(
                                 args.test_set,
                                 batch_size=args.batch_size,
@@ -318,7 +318,7 @@ def main(args):
     # final eval on dev set
     if args.do_val:
         log.info("Final validation result:")
-        test_pyreader.decorate_tensor_provider(
+        test_pyreader.set_batch_generator(
             reader.data_generator(
                 args.dev_set,
                 batch_size=args.batch_size,
@@ -339,7 +339,7 @@ def main(args):
     # final eval on test set
     if args.do_test:
         log.info("Final test result:")
-        test_pyreader.decorate_tensor_provider(
+        test_pyreader.set_batch_generator(
             reader.data_generator(
                 args.test_set,
                 batch_size=args.batch_size,
@@ -361,7 +361,6 @@ def main(args):
 if __name__ == '__main__':
     prepare_logger(log)
     print_arguments(args)
-    while True:
-        scope = fluid.core.Scope()
-        with fluid.scope_guard(scope):
-            main(args)
+    scope = fluid.core.Scope()
+    with fluid.scope_guard(scope):
+        main(args)

ernie/run_sequence_labeling.py

@@ -217,7 +217,7 @@ def main(args):
             num_trainers=nccl2_num_trainers,
             trainer_id=nccl2_trainer_id)
 
-        train_pyreader.decorate_tensor_provider(train_data_generator)
+        train_pyreader.set_batch_generator(train_data_generator)
     else:
         train_exe = None
 
@@ -302,7 +302,7 @@ def evaluate_wrapper(reader, exe, test_prog, test_pyreader, graph_vars,
     # evaluate dev set
     batch_size = args.batch_size if args.predict_batch_size is None else args.predict_batch_size
     for ds in args.dev_set.split(','): #single card eval
-        test_pyreader.decorate_tensor_provider(
+        test_pyreader.set_batch_generator(
             reader.data_generator(
                 ds,
                 batch_size=batch_size,
@@ -324,7 +324,7 @@ def predict_wrapper(reader, exe, test_prog, test_pyreader, graph_vars,
 
     batch_size = args.batch_size if args.predict_batch_size is None else args.predict_batch_size
     for test_f, save_f in zip(test_sets, save_dirs):
-        test_pyreader.decorate_tensor_provider(reader.data_generator(
+        test_pyreader.set_batch_generator(reader.data_generator(
                     test_f,
                     batch_size=batch_size,
                     epoch=1,

ernie/service/encoder_server.py

@@ -52,6 +52,10 @@ if __name__ == "__main__":
     cuda_env = os.getenv("CUDA_VISIBLE_DEVICES")
     if cuda_env is None:
         raise RuntimeError('CUDA_VISIBLE_DEVICES not set')
+    if not os.path.exists(args.model_dir):
+        raise ValueError('model_dir not found: %s' % args.model_dir)
+    if not os.path.exists(args.model_dir):
+        raise ValueError('model_dir not found: %s' % args.model_dir)
     n_devices = len(cuda_env.split(","))
     if args.encode_layer.lower() == 'pooler':
         model_dir = os.path.join(args.model_dir, 'pooler')

ernie/train.py

@@ -41,20 +41,17 @@ args = parser.parse_args()
 
 
 def create_model(pyreader_name, ernie_config):
-    pyreader = fluid.layers.py_reader(
-        capacity=70,
-        shapes=[[-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1],
-                [-1, args.max_seq_len, 1], [-1, args.max_seq_len, 1], [-1, 1],
-                [-1, 1], [-1, 1]],
-        dtypes=[
-            'int64', 'int64', 'int64', 'float32', 'int64', 'int64', 'int64'
-        ],
-        lod_levels=[0, 0, 0, 0, 0, 0, 0],
-        name=pyreader_name,
-        use_double_buffer=True)
-
-    (src_ids, pos_ids, sent_ids, input_mask, mask_label, mask_pos,
-     labels) = fluid.layers.read_file(pyreader)
+    src_ids = fluid.layers.data(name='1', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    pos_ids = fluid.layers.data(name='2', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    sent_ids= fluid.layers.data(name='3', shape=[-1, args.max_seq_len, 1], dtype='int64')
+    input_mask = fluid.layers.data(name='4', shape=[-1, args.max_seq_len, 1], dtype='float32')
+    mask_label = fluid.layers.data(name='5', shape=[-1, 1], dtype='int64')
+    mask_pos = fluid.layers.data(name='6', shape=[-1, 1], dtype='int64')
+    labels = fluid.layers.data(name='r', shape=[-1, 1], dtype='int64')
+
+    pyreader = fluid.io.DataLoader.from_generator(feed_list=[
+        src_ids, pos_ids, sent_ids, input_mask, mask_label, mask_pos, labels
+        ], capacity=70, iterable=False)
 
     ernie = ErnieModel(
         src_ids=src_ids,
@@ -97,7 +94,7 @@ def predict_wrapper(args,
 
     def predict(exe=exe, pyreader=pyreader):
 
-        pyreader.decorate_tensor_provider(data_reader.data_generator())
+        pyreader.set_batch_generator(data_reader.data_generator())
         pyreader.start()
 
         cost = 0
@@ -285,7 +282,7 @@ def train(args):
                 next_sent_acc.name, mask_lm_loss.name, total_loss.name
             ])
 
-    train_pyreader.decorate_tensor_provider(data_reader.data_generator())
+    train_pyreader.set_batch_generator(data_reader.data_generator())
     train_pyreader.start()
     steps = 0
     cost = []

ernie/utils/args.py

@@ -20,6 +20,8 @@ from __future__ import absolute_import
 
 
 import six
+import os
+import sys
 import argparse
 import logging
 
@@ -33,7 +35,7 @@ def prepare_logger(logger, debug=False, save_to_file=None):
     console_hdl = logging.StreamHandler()
     console_hdl.setFormatter(formatter)
     logger.addHandler(console_hdl)
-    if save_to_file is not None and not os.path.exits(save_to_file):
+    if save_to_file is not None and not os.path.exists(save_to_file):
         file_hdl = logging.FileHandler(save_to_file)
         file_hdl.setFormatter(formatter)
         logger.addHandler(file_hdl)

ernie/utils/data.py

@@ -4,6 +4,7 @@ import re
 from propeller import log
 import itertools
 from propeller.paddle.data import Dataset
+import pickle
 
 import six
 
@@ -101,7 +102,7 @@ class SpaceTokenizer(object):
 
 
 class CharTokenizer(object):
-    def __init__(self, vocab, lower=True):
+    def __init__(self, vocab, lower=True, sentencepiece_style_vocab=False):
         """
         char tokenizer (wordpiece english)
         normed txt(space seperated or not) => list of word-piece
@@ -110,6 +111,7 @@ class CharTokenizer(object):
         #self.pat = re.compile(r'([,.!?\u3002\uff1b\uff0c\uff1a\u201c\u201d\uff08\uff09\u3001\uff1f\u300a\u300b]|[\u4e00-\u9fa5]|[a-zA-Z0-9]+)')
         self.pat =  re.compile(r'([a-zA-Z0-9]+|\S)')
         self.lower = lower
+        self.sentencepiece_style_vocab = sentencepiece_style_vocab
 
     def __call__(self, sen):
         if len(sen) == 0:
@@ -119,11 +121,51 @@ class CharTokenizer(object):
             sen = sen.lower()
         res = []
         for match in self.pat.finditer(sen):
-            words, _ = wordpiece(match.group(0), vocab=self.vocab, unk_token='[UNK]')
+            words, _ = wordpiece(match.group(0), vocab=self.vocab, unk_token='[UNK]', sentencepiece_style_vocab=self.sentencepiece_style_vocab)
             res.extend(words)
         return res
 
 
+class WSSPTokenizer(object):
+    def __init__(self, sp_model_dir, word_dict, ws=True, lower=True):
+        self.ws = ws
+        self.lower = lower
+        self.dict = pickle.load(open(word_dict, 'rb'), encoding='utf8')
+        import sentencepiece as spm
+        self.sp_model = spm.SentencePieceProcessor()
+        self.window_size = 5
+        self.sp_model.Load(sp_model_dir)
+
+    def cut(self, chars):
+        words = []
+        idx = 0
+        while idx < len(chars):
+            matched = False
+            for i in range(self.window_size, 0, -1):
+                cand = chars[idx: idx+i]
+                if cand in self.dict:
+                    words.append(cand)
+                    matched = True
+                    break
+            if not matched: 
+                i = 1
+                words.append(chars[idx])
+            idx += i
+        return words
+ 
+    def __call__(self, sen):
+        sen = sen.decode('utf8')
+        if self.ws:
+            sen = [s for s in self.cut(sen) if s != ' ']
+        else:
+            sen = sen.split(' ')
+        if self.lower:
+            sen = [s.lower() for s in sen]
+        sen = ' '.join(sen)
+        ret = self.sp_model.EncodeAsPieces(sen)
+        return ret
+
+
 def build_2_pair(seg_a, seg_b, max_seqlen, cls_id, sep_id):
     token_type_a = np.ones_like(seg_a, dtype=np.int64) * 0
     token_type_b = np.ones_like(seg_b, dtype=np.int64) * 1

example/finetune_classifier.py

@@ -55,7 +55,7 @@ class ClassificationErnieModel(propeller.train.Model):
         pos_ids = L.cast(pos_ids, 'int64')
         pos_ids.stop_gradient = True
         input_mask.stop_gradient = True
-        task_ids = L.zeros_like(src_ids) + self.hparam.task_id #this shit wont use at the moment
+        task_ids = L.zeros_like(src_ids) + self.hparam.task_id
         task_ids.stop_gradient = True
 
         ernie = ErnieModel(
@@ -128,6 +128,8 @@ if __name__ == '__main__':
     parser.add_argument('--vocab_file', type=str, required=True)
     parser.add_argument('--do_predict', action='store_true')
     parser.add_argument('--warm_start_from', type=str)
+    parser.add_argument('--sentence_piece_model', type=str, default=None)
+    parser.add_argument('--word_dict', type=str, default=None)
     args = parser.parse_args()
     run_config = propeller.parse_runconfig(args)
     hparams = propeller.parse_hparam(args)
@@ -138,7 +140,12 @@ if __name__ == '__main__':
     cls_id = vocab['[CLS]']
     unk_id = vocab['[UNK]']
 
-    tokenizer = utils.data.CharTokenizer(vocab.keys())
+    if args.sentence_piece_model is not None:
+        if args.word_dict is None:
+            raise ValueError('--word_dict no specified in subword Model')
+        tokenizer = utils.data.WSSPTokenizer(args.sentence_piece_model, args.word_dict, ws=True, lower=True)
+    else:
+        tokenizer = utils.data.CharTokenizer(vocab.keys())
 
     def tokenizer_func(inputs):
         '''avoid pickle error'''
@@ -179,7 +186,7 @@ if __name__ == '__main__':
         dev_ds.data_shapes = shapes
         dev_ds.data_types = types
 
-        varname_to_warmstart = re.compile('encoder.*|pooled.*|.*embedding|pre_encoder_.*')
+        varname_to_warmstart = re.compile(r'^encoder.*[wb]_0$|^.*embedding$|^.*bias$|^.*scale$|^pooled_fc.[wb]_0$')
         warm_start_dir = args.warm_start_from
         ws = propeller.WarmStartSetting(
                 predicate_fn=lambda v: varname_to_warmstart.match(v.name) and os.path.exists(os.path.join(warm_start_dir, v.name)),

example/finetune_ner.py

@@ -32,7 +32,6 @@ import paddle.fluid.layers as L
 
 from model.ernie import ErnieModel
 from optimization import optimization
-import tokenization
 import utils.data
 
 from propeller import log
@@ -121,7 +120,7 @@ class SequenceLabelErnieModel(propeller.train.Model):
 def make_sequence_label_dataset(name, input_files, label_list, tokenizer, batch_size, max_seqlen, is_train):
     label_map = {v: i for i, v in enumerate(label_list)}
     no_entity_id = label_map['O']
-    delimiter = ''
+    delimiter = b''
 
     def read_bio_data(filename):
         ds = propeller.data.Dataset.from_file(filename)
@@ -132,10 +131,10 @@ def make_sequence_label_dataset(name, input_files, label_list, tokenizer, batch_
             while 1:
                 line = next(iterator)
                 cols = line.rstrip(b'\n').split(b'\t')
+                tokens = cols[0].split(delimiter)
+                labels = cols[1].split(delimiter)
                 if len(cols) != 2:
                     continue
-                tokens = tokenization.convert_to_unicode(cols[0]).split(delimiter)
-                labels = tokenization.convert_to_unicode(cols[1]).split(delimiter)
                 if len(tokens) != len(labels) or len(tokens) == 0:
                     continue
                 yield [tokens, labels]
@@ -151,7 +150,8 @@ def make_sequence_label_dataset(name, input_files, label_list, tokenizer, batch_
                 ret_tokens = []
                 ret_labels = []
                 for token, label in zip(tokens, labels):
-                    sub_token = tokenizer.tokenize(token)
+                    sub_token = tokenizer(token)
+                    label = label.decode('utf8')
                     if len(sub_token) == 0:
                         continue
                     ret_tokens.extend(sub_token)
@@ -179,7 +179,7 @@ def make_sequence_label_dataset(name, input_files, label_list, tokenizer, batch_
                     labels = labels[: max_seqlen - 2]
 
                 tokens = ['[CLS]'] + tokens + ['[SEP]']
-                token_ids = tokenizer.convert_tokens_to_ids(tokens)
+                token_ids = [vocab[t] for t in tokens]
                 label_ids = [no_entity_id] + [label_map[x] for x in labels] + [no_entity_id]
                 token_type_ids = [0] * len(token_ids)
                 input_seqlen = len(token_ids)
@@ -211,7 +211,7 @@ def make_sequence_label_dataset(name, input_files, label_list, tokenizer, batch_
 
 
 def make_sequence_label_dataset_from_stdin(name, tokenizer, batch_size, max_seqlen):
-    delimiter = ''
+    delimiter = b''
 
     def stdin_gen():
         if six.PY3:
@@ -232,9 +232,9 @@ def make_sequence_label_dataset_from_stdin(name, tokenizer, batch_size, max_seql
             while 1:
                 line, = next(iterator)
                 cols = line.rstrip(b'\n').split(b'\t')
+                tokens = cols[0].split(delimiter)
                 if len(cols) != 1:
                     continue
-                tokens = tokenization.convert_to_unicode(cols[0]).split(delimiter)
                 if len(tokens) == 0:
                     continue
                 yield tokens, 
@@ -247,7 +247,7 @@ def make_sequence_label_dataset_from_stdin(name, tokenizer, batch_size, max_seql
                 tokens, = next(iterator)
                 ret_tokens = []
                 for token in tokens:
-                    sub_token = tokenizer.tokenize(token)
+                    sub_token = tokenizer(token)
                     if len(sub_token) == 0:
                         continue
                     ret_tokens.extend(sub_token)
@@ -266,7 +266,7 @@ def make_sequence_label_dataset_from_stdin(name, tokenizer, batch_size, max_seql
                     tokens = tokens[: max_seqlen - 2]
 
                 tokens = ['[CLS]'] + tokens + ['[SEP]']
-                token_ids = tokenizer.convert_tokens_to_ids(tokens)
+                token_ids = [vocab[t] for t in tokens]
                 token_type_ids = [0] * len(token_ids)
                 input_seqlen = len(token_ids)
 
@@ -296,13 +296,15 @@ if __name__ == '__main__':
     parser.add_argument('--data_dir', type=str, required=True)
     parser.add_argument('--vocab_file', type=str, required=True)
     parser.add_argument('--do_predict', action='store_true')
+    parser.add_argument('--use_sentence_piece_vocab', action='store_true')
     parser.add_argument('--warm_start_from', type=str)
     args = parser.parse_args()
     run_config = propeller.parse_runconfig(args)
     hparams = propeller.parse_hparam(args)
 
-    tokenizer = tokenization.FullTokenizer(args.vocab_file)
-    vocab = tokenizer.vocab
+
+    vocab = {j.strip().split('\t')[0]: i for i, j in enumerate(open(args.vocab_file, 'r', encoding='utf8'))}
+    tokenizer = utils.data.CharTokenizer(vocab, sentencepiece_style_vocab=args.use_sentence_piece_vocab)
     sep_id = vocab['[SEP]']
     cls_id = vocab['[CLS]']
     unk_id = vocab['[UNK]']
@@ -358,7 +360,7 @@ if __name__ == '__main__':
                 from_dir=warm_start_dir
             )
 
-        best_exporter = propeller.train.exporter.BestExporter(os.path.join(run_config.model_dir, 'best'), cmp_fn=lambda old, new: new['dev']['f1'] > old['dev']['f1'])
+        best_exporter = propeller.train.exporter.BestInferenceModelExporter(os.path.join(run_config.model_dir, 'best'), cmp_fn=lambda old, new: new['dev']['f1'] > old['dev']['f1'])
         propeller.train.train_and_eval(
                 model_class_or_model_fn=SequenceLabelErnieModel,
                 params=hparams, 
@@ -387,7 +389,6 @@ if __name__ == '__main__':
         predict_ds.data_types = types
 
         rev_label_map = {i: v for i, v in enumerate(label_list)}
-        best_exporter = propeller.train.exporter.BestExporter(os.path.join(run_config.model_dir, 'best'), cmp_fn=lambda old, new: new['dev']['f1'] > old['dev']['f1'])
         learner = propeller.Learner(SequenceLabelErnieModel, run_config, hparams)
         for pred, _  in learner.predict(predict_ds, ckpt=-1):
             pred_str = ' '.join([rev_label_map[idx] for idx in np.argmax(pred, 1).tolist()])

example/finetune_ranker.py

@@ -146,6 +146,7 @@ if __name__ == '__main__':
     parser.add_argument('--data_dir', type=str, required=True)
     parser.add_argument('--warm_start_from', type=str)
     parser.add_argument('--sentence_piece_model', type=str, default=None)
+    parser.add_argument('--word_dict', type=str, default=None)
     args = parser.parse_args()
     run_config = propeller.parse_runconfig(args)
     hparams = propeller.parse_hparam(args)
@@ -157,7 +158,9 @@ if __name__ == '__main__':
     unk_id = vocab['[UNK]']
 
     if args.sentence_piece_model is not None:
-        tokenizer = utils.data.JBSPTokenizer(args.sentence_piece_model, jb=True, lower=True)
+        if args.word_dict is None:
+            raise ValueError('--word_dict no specified in subword Model')
+        tokenizer = utils.data.WSSPTokenizer(args.sentence_piece_model, args.word_dict, ws=True, lower=True)
     else:
         tokenizer = utils.data.CharTokenizer(vocab.keys())
 
@@ -218,7 +221,7 @@ if __name__ == '__main__':
                 from_dir=warm_start_dir
             )
 
-        best_exporter = propeller.train.exporter.BestExporter(os.path.join(run_config.model_dir, 'best'), cmp_fn=lambda old, new: new['dev']['f1'] > old['dev']['f1'])
+        best_exporter = propeller.train.exporter.BestInferenceModelExporter(os.path.join(run_config.model_dir, 'best'), cmp_fn=lambda old, new: new['dev']['f1'] > old['dev']['f1'])
         propeller.train_and_eval(
                 model_class_or_model_fn=RankingErnieModel, 
                 params=hparams, 
@@ -258,6 +261,7 @@ if __name__ == '__main__':
         est = propeller.Learner(RankingErnieModel, run_config, hparams)
         for qid, res in est.predict(predict_ds, ckpt=-1):
             print('%d\t%d\t%.5f\t%.5f' % (qid[0], np.argmax(res), res[0], res[1]))
+
         #for i in predict_ds:
         #    sen = i[0]
         #    for ss in np.squeeze(sen):

inference/README.md

+# ERNIE fast inference (C++)
+
+ERNIE C++ fast inference API提供了一种更为高效的在线预测方案，可以直接联编译至生产环境以获取更好的性能。
+其实现基于[fluid inference](https://www.paddlepaddle.org.cn/documentation/docs/zh/advanced_usage/deploy/inference/native_infer.html).
+**请确保您的 fluid inference 版本高于 1.6.3 以获得正确的预测结果。**
+本页面提供了一个ERNIE C++ fast inference 的 demo benchmark.
+
+## 准备工作
+
+demo 数据取自XNLI数据集test集合，位于./data 中。采用明文id格式，一行代表一个 batch, 包含四个字段：
+```text
+src_ids, pos_ids, sent_ids, self_attn_mask
+```
+字段之间按照分号(;)分隔；各字段内部包含 `shape` 和 `data` 两部分，按照冒号(:)分隔； `shape` 和 `data` 内部按空格分隔；`self_attn_mask` 为 FLOAT32 类型，其余字段为 INT64 类型。
+
+ERNIE fast inference 需要输入 inference\_model 格式的模型，可以参考[这里](../README.zh.md#生成inference_model)生成 inference\_model .
+
+**使用propeller产出的 inference\_model 只需要`src_ids`，`sent_ids` 两个字段，因此需要适当修改数据文件**
+
+
+## 编译和运行
+
+为了编译本 demo，c++ 编译器需要支持 C++11 标准。
+
+下载对应的 [fluid_inference库](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/advanced_usage/deploy/inference/build_and_install_lib_cn.html) , 根据使用的 paddle 的版本和配置状况 (是否使用 avx, mkl, 以及 cuda, cudnn 版本) 选择下载对应的版本并解压，会得到 `fluid_inference` 文件夹，将其放在与`inference.cc`同一级目录。
+
+用以下命令编译：
+``` bash
+cd ./gpu # cd ./cpu
+mkdir build
+cd build
+cmake ..
+make
+```
+
+用以下命令运行：
+```
+./run.sh ../data/sample /path/to/inference_mode_dir
+```
+
+## 性能测试
+
+测试样本：XNLI test集合，输入BatchSize=1, SequenceLength=128.
+重复5遍取平均值。
+
+| 测试环境 | 延迟(ms) |
+| ----- | -----    |
+| CPU（Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz (20 线程)） | 29.8818|
+| GPU （P4）  | 8.5 |
+

inference/cpu/CMakeLists.txt

+CMAKE_MINIMUM_REQUIRED(VERSION 3.2)
+PROJECT(inference_demo)
+SET(CMAKE_C_COMPILER gcc)
+SET(CMAKE_CXX_COMPILER g++)
+ADD_COMPILE_OPTIONS(-std=c++11 -g)
+
+SET(FLUID_INFER_LIB fluid_inference)
+SET(FLUID_INC_PATH ${FLUID_INFER_LIB}/paddle/include)
+SET(FLUID_LIB_PATH ${FLUID_INFER_LIB}/paddle/lib)
+
+SET(GLOG_INC_PATH ${FLUID_INFER_LIB}/third_party/install/glog/include)
+SET(GLOG_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/glog/lib)
+
+SET(GFLAGS_INC_PATH ${FLUID_INFER_LIB}/third_party/install/gflags/include)
+SET(GFLAGS_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/gflags/lib)
+SET(MKLDNN_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/mkldnn/lib)
+SET(MKLML_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/mklml/lib)
+
+INCLUDE_DIRECTORIES(${FLUID_INC_PATH})
+INCLUDE_DIRECTORIES(${GLOG_INC_PATH})
+INCLUDE_DIRECTORIES(${GFLAGS_INC_PATH})
+
+LINK_DIRECTORIES(${FLUID_LIB_PATH})
+LINK_DIRECTORIES(${GLOG_LIB_PATH})
+LINK_DIRECTORIES(${GFLAGS_LIB_PATH})
+LINK_DIRECTORIES(${MKLML_LIB_PATH})
+LINK_DIRECTORIES(${MKLDNN_LIB_PATH})
+
+ADD_EXECUTABLE(inference inference.cc)
+TARGET_LINK_LIBRARIES(inference dl paddle_fluid glog gflags pthread)
+

inference/cpu/inference.cc

+// Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+#include <paddle_inference_api.h>
+#include <chrono>
+#include <fstream>
+#include <iostream>
+#include <numeric>
+#include <sstream>
+#include <string>
+#include <vector>
+
+DEFINE_string(model_dir, "", "model directory");
+DEFINE_string(data, "", "input data path");
+DEFINE_int32(repeat, 1, "repeat");
+DEFINE_bool(output_prediction, false, "Whether to output the prediction results.");
+DEFINE_bool(use_gpu, false, "Whether to use GPU for prediction.");
+DEFINE_int32(device, 0, "device.");
+
+
+template <typename T>
+void GetValueFromStream(std::stringstream *ss, T *t) {
+  (*ss) >> (*t);
+}
+
+template <>
+void GetValueFromStream<std::string>(std::stringstream *ss, std::string *t) {
+  *t = ss->str();
+}
+
+// Split string to vector
+template <typename T>
+void Split(const std::string &line, char sep, std::vector<T> *v) {
+  std::stringstream ss;
+  T t;
+  for (auto c : line) {
+    if (c != sep) {
+      ss << c;
+    } else {
+      GetValueFromStream<T>(&ss, &t);
+      v->push_back(std::move(t));
+      ss.str({});
+      ss.clear();
+    }
+  }
+
+  if (!ss.str().empty()) {
+    GetValueFromStream<T>(&ss, &t);
+    v->push_back(std::move(t));
+    ss.str({});
+    ss.clear();
+  }
+}
+
+template <typename T>
+constexpr paddle::PaddleDType GetPaddleDType();
+
+template <>
+constexpr paddle::PaddleDType GetPaddleDType<int64_t>() {
+  return paddle::PaddleDType::INT64;
+}
+
+template <>
+constexpr paddle::PaddleDType GetPaddleDType<float>() {
+  return paddle::PaddleDType::FLOAT32;
+}
+
+// Parse tensor from string
+template <typename T>
+bool ParseTensor(const std::string &field, paddle::PaddleTensor *tensor) {
+  std::vector<std::string> data;
+  Split(field, ':', &data);
+  if (data.size() < 2) return false;
+
+  std::string shape_str = data[0];
+
+  std::vector<int> shape;
+  Split(shape_str, ' ', &shape);
+
+  std::string mat_str = data[1];
+
+  std::vector<T> mat;
+  Split(mat_str, ' ', &mat);
+
+  tensor->shape = shape;
+  auto size =
+      std::accumulate(shape.begin(), shape.end(), 1, std::multiplies<int>()) *
+      sizeof(T);
+  tensor->data.Resize(size);
+  std::copy(mat.begin(), mat.end(), static_cast<T *>(tensor->data.data()));
+  tensor->dtype = GetPaddleDType<T>();
+
+  return true;
+}
+
+// Parse input tensors from string
+bool ParseLine(const std::string &line,
+               std::vector<paddle::PaddleTensor> *tensors) {
+  std::vector<std::string> fields;
+  Split(line, ';', &fields);
+
+  if (fields.size() <= 2) return false;
+
+  tensors->clear();
+  tensors->reserve(4);
+
+  int i = 0;
+  // src_ids
+  paddle::PaddleTensor src_ids;
+  ParseTensor<int64_t>(fields[i++], &src_ids);
+  src_ids.name = "eval_placeholder_0";
+  tensors->push_back(src_ids);
+
+  // sent_ids
+  paddle::PaddleTensor sent_ids;
+  ParseTensor<int64_t>(fields[i++], &sent_ids);
+  sent_ids.name = "eval_placeholder_1";
+  tensors->push_back(sent_ids);
+
+  // pos_ids
+  paddle::PaddleTensor pos_ids;
+  ParseTensor<int64_t>(fields[i++], &pos_ids);
+  pos_ids.name = "eval_placeholder_2";
+  tensors->push_back(pos_ids);
+
+
+  // input_mask
+  paddle::PaddleTensor input_mask;
+  ParseTensor<float>(fields[i++], &input_mask);
+  input_mask.name = "eval_placeholder_3";
+  tensors->push_back(input_mask);
+
+  return true;
+}
+
+// Print outputs to log
+void PrintOutputs(const std::vector<paddle::PaddleTensor> &outputs) {
+  //LOG(INFO) << "example_id\tcontradiction\tentailment\tneutral";
+  for (size_t i = 0; i < outputs.front().data.length() / sizeof(float) / 3; i += 1) {
+    std::cout << static_cast<float *>(outputs[0].data.data())[3 * i] << "\t"
+         << static_cast<float *>(outputs[0].data.data())[3 * i + 1] << "\t"
+         << static_cast<float *>(outputs[0].data.data())[3 * i + 2] << std::endl;
+  }
+}
+
+bool LoadInputData(std::vector<std::vector<paddle::PaddleTensor>> *inputs) {
+  if (FLAGS_data.empty()) {
+    LOG(ERROR) << "please set input data path";
+    return false;
+  }
+
+  std::ifstream fin(FLAGS_data);
+  std::string line;
+
+  int lineno = 0;
+  while (std::getline(fin, line)) {
+    std::vector<paddle::PaddleTensor> feed_data;
+    if (!ParseLine(line, &feed_data)) {
+      LOG(ERROR) << "Parse line[" << lineno << "] error!";
+    } else {
+      inputs->push_back(std::move(feed_data));
+    }
+  }
+
+  return true;
+}
+
+// ernie inference demo
+// Options:
+//     --model_dir: ernie model file directory
+//     --data: data path
+//     --repeat: repeat num
+//     --use_gpu: use gpu
+int main(int argc, char *argv[]) {
+  google::InitGoogleLogging(*argv);
+  gflags::ParseCommandLineFlags(&argc, &argv, true);
+
+  if (FLAGS_model_dir.empty()) {
+    LOG(ERROR) << "please set model dir";
+    return -1;
+  }
+
+  paddle::AnalysisConfig config;
+  config.SetModel(FLAGS_model_dir);
+  config.DisableGpu();
+  config.SwitchIrOptim();
+  config.EnableMKLDNN();
+  config.SetCpuMathLibraryNumThreads(20);
+  //config.EnableMemoryOptim();
+
+  auto predictor = CreatePaddlePredictor(config);
+
+  std::vector<std::vector<paddle::PaddleTensor>> inputs;
+  if (!LoadInputData(&inputs)) {
+    LOG(ERROR) << "load input data error!";
+    return -1;
+  }
+
+  std::vector<paddle::PaddleTensor> fetch;
+  int total_time{0};
+  // auto predict_timer = []()
+  int num_samples{0};
+  int count{0};
+  for (int i = 0; i < FLAGS_repeat; i++) {
+    for (auto feed : inputs) {
+      fetch.clear();
+      auto start = std::chrono::system_clock::now();
+      predictor->Run(feed, &fetch);
+      if (FLAGS_output_prediction && i == 0) {
+        PrintOutputs(fetch);
+      }
+      auto end = std::chrono::system_clock::now();
+      count += 1;
+      if (!fetch.empty()) {
+        total_time +=
+            std::chrono::duration_cast<std::chrono::milliseconds>(end - start)
+                .count();
+        //num_samples += fetch.front().data.length() / 2 / sizeof(float);
+        num_samples += fetch.front().data.length() / (sizeof(float) * 2);
+      }
+    }
+  }
+
+  auto per_sample_ms =
+      static_cast<float>(total_time) / num_samples;
+  LOG(INFO) << "Run " << num_samples
+            << " samples, average latency: " << per_sample_ms
+            << "ms per sample.";
+  LOG(INFO) << count;
+
+  return 0;
+}

inference/cpu/run.sh

+set -x
+(($# != 2)) && echo "${0} data model" && exit -1
+
+export LD_LIBRARY_PATH=fluid_inference/third_party/install/mkldnn/lib:fluid_inference/third_party/install/mklml/lib:fluid_inference/paddle/lib/:/home/work/cuda-9.0/lib64/:/home/work/cudnn/cudnn_v7_3_1_cuda9.0/lib64/:$LD_LIBRARY_PATH \
+
+./build/inference --logtostderr \
+    --model_dir $2 \
+    --data $1 \
+    --repeat 5 \
+    --output_prediction true \
+    --use_gpu true \
+    --device 0 \

inference/gpu/CMakeLists.txt

+CMAKE_MINIMUM_REQUIRED(VERSION 3.2)
+PROJECT(inference_demo)
+SET(CMAKE_C_COMPILER gcc)
+SET(CMAKE_CXX_COMPILER g++)
+ADD_COMPILE_OPTIONS(-std=c++11 -g)
+
+SET(FLUID_INFER_LIB fluid_inference)
+SET(FLUID_INC_PATH ${FLUID_INFER_LIB}/paddle/include)
+SET(FLUID_LIB_PATH ${FLUID_INFER_LIB}/paddle/lib)
+
+SET(GLOG_INC_PATH ${FLUID_INFER_LIB}/third_party/install/glog/include)
+SET(GLOG_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/glog/lib)
+
+SET(GFLAGS_INC_PATH ${FLUID_INFER_LIB}/third_party/install/gflags/include)
+SET(GFLAGS_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/gflags/lib)
+SET(MKLDNN_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/mkldnn/lib)
+SET(MKLML_LIB_PATH ${FLUID_INFER_LIB}/third_party/install/mklml/lib)
+
+INCLUDE_DIRECTORIES(${FLUID_INC_PATH})
+INCLUDE_DIRECTORIES(${GLOG_INC_PATH})
+INCLUDE_DIRECTORIES(${GFLAGS_INC_PATH})
+
+LINK_DIRECTORIES(${FLUID_LIB_PATH})
+LINK_DIRECTORIES(${GLOG_LIB_PATH})
+LINK_DIRECTORIES(${GFLAGS_LIB_PATH})
+LINK_DIRECTORIES(${MKLML_LIB_PATH})
+LINK_DIRECTORIES(${MKLDNN_LIB_PATH})
+
+ADD_EXECUTABLE(inference inference.cc)
+TARGET_LINK_LIBRARIES(inference dl paddle_fluid glog gflags pthread)
+

inference/gpu/inference.cc

+// Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+#include <paddle_inference_api.h>
+#include <chrono>
+#include <fstream>
+#include <iostream>
+#include <numeric>
+#include <sstream>
+#include <string>
+#include <vector>
+
+DEFINE_string(model_dir, "", "model directory");
+DEFINE_string(data, "", "input data path");
+DEFINE_int32(repeat, 1, "repeat");
+DEFINE_bool(output_prediction, false, "Whether to output the prediction results.");
+DEFINE_bool(use_gpu, false, "Whether to use GPU for prediction.");
+DEFINE_int32(device, 0, "device.");
+
+
+template <typename T>
+void GetValueFromStream(std::stringstream *ss, T *t) {
+  (*ss) >> (*t);
+}
+
+template <>
+void GetValueFromStream<std::string>(std::stringstream *ss, std::string *t) {
+  *t = ss->str();
+}
+
+// Split string to vector
+template <typename T>
+void Split(const std::string &line, char sep, std::vector<T> *v) {
+  std::stringstream ss;
+  T t;
+  for (auto c : line) {
+    if (c != sep) {
+      ss << c;
+    } else {
+      GetValueFromStream<T>(&ss, &t);
+      v->push_back(std::move(t));
+      ss.str({});
+      ss.clear();
+    }
+  }
+
+  if (!ss.str().empty()) {
+    GetValueFromStream<T>(&ss, &t);
+    v->push_back(std::move(t));
+    ss.str({});
+    ss.clear();
+  }
+}
+
+template <typename T>
+constexpr paddle::PaddleDType GetPaddleDType();
+
+template <>
+constexpr paddle::PaddleDType GetPaddleDType<int64_t>() {
+  return paddle::PaddleDType::INT64;
+}
+
+template <>
+constexpr paddle::PaddleDType GetPaddleDType<float>() {
+  return paddle::PaddleDType::FLOAT32;
+}
+
+// Parse tensor from string
+template <typename T>
+bool ParseTensor(const std::string &field, paddle::PaddleTensor *tensor) {
+  std::vector<std::string> data;
+  Split(field, ':', &data);
+  if (data.size() < 2) return false;
+
+  std::string shape_str = data[0];
+
+  std::vector<int> shape;
+  Split(shape_str, ' ', &shape);
+
+  std::string mat_str = data[1];
+
+  std::vector<T> mat;
+  Split(mat_str, ' ', &mat);
+
+  tensor->shape = shape;
+  auto size =
+      std::accumulate(shape.begin(), shape.end(), 1, std::multiplies<int>()) *
+      sizeof(T);
+  tensor->data.Resize(size);
+  std::copy(mat.begin(), mat.end(), static_cast<T *>(tensor->data.data()));
+  tensor->dtype = GetPaddleDType<T>();
+
+  return true;
+}
+
+// Parse input tensors from string
+bool ParseLine(const std::string &line,
+               std::vector<paddle::PaddleTensor> *tensors) {
+  std::vector<std::string> fields;
+  Split(line, ';', &fields);
+
+  if (fields.size() <= 2) return false;
+
+  tensors->clear();
+  tensors->reserve(4);
+
+  int i = 0;
+  paddle::PaddleTensor src_ids;
+  ParseTensor<int64_t>(fields[i++], &src_ids);
+  src_ids.name = "eval_placeholder_0";
+  tensors->push_back(src_ids);
+
+  // sent_ids
+  paddle::PaddleTensor sent_ids;
+  ParseTensor<int64_t>(fields[i++], &sent_ids);
+  sent_ids.name = "eval_placeholder_1";
+  tensors->push_back(sent_ids);
+
+  // pos_ids
+  paddle::PaddleTensor pos_ids;
+  ParseTensor<int64_t>(fields[i++], &pos_ids);
+  pos_ids.name = "eval_placeholder_2";
+  tensors->push_back(pos_ids);
+
+
+  // input_mask
+  paddle::PaddleTensor input_mask;
+  ParseTensor<float>(fields[i++], &input_mask);
+  input_mask.name = "eval_placeholder_3";
+  tensors->push_back(input_mask);
+
+  return true;
+}
+
+// Print outputs to log
+void PrintOutputs(const std::vector<paddle::PaddleTensor> &outputs) {
+  //LOG(INFO) << "example_id\tcontradiction\tentailment\tneutral";
+  for (size_t i = 0; i < outputs.front().data.length() / sizeof(float) / 3; i += 1) {
+    std::cout << static_cast<float *>(outputs[0].data.data())[3 * i] << "\t"
+         << static_cast<float *>(outputs[0].data.data())[3 * i + 1] << "\t"
+         << static_cast<float *>(outputs[0].data.data())[3 * i + 2] << std::endl;
+  }
+}
+
+bool LoadInputData(std::vector<std::vector<paddle::PaddleTensor>> *inputs) {
+  if (FLAGS_data.empty()) {
+    LOG(ERROR) << "please set input data path";
+    return false;
+  }
+
+  std::ifstream fin(FLAGS_data);
+  std::string line;
+
+  int lineno = 0;
+  while (std::getline(fin, line)) {
+    std::vector<paddle::PaddleTensor> feed_data;
+    if (!ParseLine(line, &feed_data)) {
+      LOG(ERROR) << "Parse line[" << lineno << "] error!";
+    } else {
+      inputs->push_back(std::move(feed_data));
+    }
+  }
+
+  return true;
+}
+
+// ernie inference demo
+// Options:
+//     --model_dir: ernie model file directory
+//     --data: data path
+//     --repeat: repeat num
+//     --use_gpu: use gpu
+int main(int argc, char *argv[]) {
+  google::InitGoogleLogging(*argv);
+  gflags::ParseCommandLineFlags(&argc, &argv, true);
+
+  if (FLAGS_model_dir.empty()) {
+    LOG(ERROR) << "please set model dir";
+    return -1;
+  }
+
+  paddle::AnalysisConfig config;
+  config.SetModel(FLAGS_model_dir);
+  config.EnableUseGpu(100, 0);
+  config.SwitchSpecifyInputNames(true);
+  config.EnableCUDNN();
+  config.SwitchIrOptim(true);
+  config.EnableMemoryOptim();
+  auto predictor = CreatePaddlePredictor(config);
+
+  std::vector<std::vector<paddle::PaddleTensor>> inputs;
+  if (!LoadInputData(&inputs)) {
+    LOG(ERROR) << "load input data error!";
+    return -1;
+  }
+
+  std::vector<paddle::PaddleTensor> fetch;
+  int total_time{0};
+  // auto predict_timer = []()
+  int num_samples{0};
+  int count{0};
+  for (int i = 0; i < FLAGS_repeat; i++) {
+    for (auto feed : inputs) {
+      fetch.clear();
+      auto start = std::chrono::system_clock::now();
+      predictor->Run(feed, &fetch);
+      if (FLAGS_output_prediction && i == 0) {
+        PrintOutputs(fetch);
+      }
+      auto end = std::chrono::system_clock::now();
+      count += 1;
+      if (!fetch.empty()) {
+        total_time +=
+            std::chrono::duration_cast<std::chrono::milliseconds>(end - start)
+                .count();
+        //num_samples += fetch.front().data.length() / 2 / sizeof(float);
+        num_samples += fetch.front().data.length() / (sizeof(float) * 2);
+      }
+    }
+  }
+
+  auto per_sample_ms =
+      static_cast<float>(total_time) / num_samples;
+  LOG(INFO) << "Run " << num_samples
+            << " samples, average latency: " << per_sample_ms
+            << "ms per sample.";
+  LOG(INFO) << count;
+
+  return 0;
+}

inference/gpu/run.sh

+set -x
+(($# != 2)) && echo "${0} data model" && exit -1
+
+export LD_LIBRARY_PATH=fluid_inference/third_party/install/mkldnn/lib:fluid_inference/third_party/install/mklml/lib:fluid_inference/paddle/lib/:/home/work/cuda-9.0/lib64/:/home/work/cudnn/cudnn_v7_3_1_cuda9.0/lib64/:$LD_LIBRARY_PATH
+
+./build/inference --logtostderr \
+    --model_dir $2 \
+    --data $1 \
+    --repeat 5 \
+    --output_prediction true \
+    --use_gpu true \
+    --device 0 \

propeller/README.en.md

@@ -18,11 +18,6 @@ Propeller provide the following benefits:
     -   create checkpoint files and recover from failures
     -   save visualizable results
 
-## install
-
-```script
-pip install --user .
-```
 
 ## Getting Started
 ```python
@@ -71,6 +66,7 @@ pip install --user .
     # Start training!
     propeller.train_and_eval(BowModel, hparams, run_config, train_ds, eval_ds)
 ```
+More detail see example/toy/
 
 ## Main Feature
 1. train_and_eval

propeller/README.md

@@ -19,10 +19,6 @@ Propeller 具有下列优势：
     -   创建检查点文件并从故障中恢复
     -   保存可视化的摘要结果
 
-## install|安装
-
-pip install --user .
-
 ## Getting Started|快速开始
 ```python
 
@@ -70,6 +66,7 @@ pip install --user .
     # 开始训练！
     propeller.train_and_eval(BowModel, hparams, run_config, train_ds, eval_ds)
 ```
+详细详细请见example/toy/
 
 ## 主要构件
 1. train_and_eval

propeller/__init__.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""Propeller"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals

propeller/data/__init__.py

@@ -11,3 +11,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+doc
+"""

propeller/data/functional.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""Basic Dataset API"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -39,7 +40,7 @@ __all__ = ['Dataset']
 
 
 @contextmanager
-def open_file(filename, format=None):
+def _open_file(filename, format=None):
     if format is None:
         fd = open(filename, 'rb')
     elif format == 'GZIP':
@@ -50,9 +51,9 @@ def open_file(filename, format=None):
     fd.close()
 
 
-def open_record(filename):
-    def gen():
-        with open_file(filename, format='GZIP') as f:
+def _open_record(filename):
+    def _gen():
+        with _open_file(filename, format='GZIP') as f:
             while True:
                 data = f.read(struct.calcsize('i'))
                 if not len(data):
@@ -61,11 +62,11 @@ def open_record(filename):
                 data = f.read(l)
                 yield data
 
-    return gen
+    return _gen
 
 
-def shuffle_func(dataset, buffer_size):
-    def gen():
+def _shuffle_func(dataset, buffer_size):
+    def _gen():
         buf = []
         iterable = dataset()
         try:
@@ -82,11 +83,11 @@ def shuffle_func(dataset, buffer_size):
                 for i in buf:
                     yield i
 
-    return gen
+    return _gen
 
 
-def interleave_func(iterable, map_fn, cycle_length, block_length):
-    def gen():
+def _interleave_func(iterable, map_fn, cycle_length, block_length):
+    def _gen():
         ls = itertools.tee(iterable(), cycle_length)
         buf = []
         for i, j in enumerate(ls):
@@ -99,11 +100,11 @@ def interleave_func(iterable, map_fn, cycle_length, block_length):
             for ii in (i for i in tup if i is not None):
                 yield ii
 
-    return gen
+    return _gen
 
 
-def repeat_func(dataset, n):
-    def gen():
+def _repeat_func(dataset, n):
+    def _gen():
         iterable = dataset()
         if n >= 0:
             ret = itertools.chain(*itertools.tee(iterable, n))
@@ -113,11 +114,11 @@ def repeat_func(dataset, n):
         for i in ret:
             yield i
 
-    return gen
+    return _gen
 
 
-def filter_func(dataset, fn):
-    def gen():
+def _filter_func(dataset, fn):
+    def _gen():
         for i in dataset():
             if isinstance(i, tuple) or isinstance(i, list):
                 if fn(*i) is True:
@@ -126,41 +127,41 @@ def filter_func(dataset, fn):
                 if fn(i) is True:
                     yield i
 
-    return gen
+    return _gen
 
 
-def map_func(dataset, fn):
-    def gen():
+def _map_func(dataset, fn):
+    def _gen():
         for i in dataset():
             if isinstance(i, tuple) or isinstance(i, list):
                 yield fn(*i)
             else:
                 yield fn(i)
 
-    return gen
+    return _gen
 
 
-def shard_func(dataset, num_shards, index):
-    def gen():
+def _shard_func(dataset, num_shards, index):
+    def _gen():
         iterable = dataset()
         ret = itertools.islice(iterable, index, None, num_shards)
         for i in ret:
             yield i
 
-    return gen
+    return _gen
 
 
-def take_func(dataset, count):
-    def gen():
+def _take_func(dataset, count):
+    def _gen():
         iterable = dataset()
         ret = itertools.islice(iterable, count)
         for i in ret:
             yield i
 
-    return gen
+    return _gen
 
 
-def buffered_func(dataset, size):
+def _buffered_func(dataset, size):
     """
     Creates a buffered data reader.
 
@@ -176,21 +177,21 @@ def buffered_func(dataset, size):
     :returns: the buffered data reader.
     """
 
-    class EndSignal():
+    class _EndSignal(object):
         pass
 
-    end = EndSignal()
+    end = _EndSignal()
 
-    def read_worker(r, q):
+    def _read_worker(r, q):
         for d in r:
             q.put(d)
         q.put(end)
 
-    def data_reader():
+    def _data_reader():
         r = dataset()
         q = multiprocessing.Queue(maxsize=size)
         t = multiprocessing.Process(
-            target=read_worker, args=(
+            target=_read_worker, args=(
                 r,
                 q, ))
         t.daemon = True
@@ -200,14 +201,14 @@ def buffered_func(dataset, size):
             yield e
             e = q.get()
 
-    return data_reader
+    return _data_reader
 
 
-def padded_batch_func(dataset, batch_size, pad_value=0, max_seqlen=None):
+def _padded_batch_func(dataset, batch_size, pad_value=0, max_seqlen=None):
     if not isinstance(batch_size, int):
         raise ValueError('unknown batch_size: %s' % repr(batch_size))
 
-    def gen():
+    def _gen():
         iterable = dataset()
         pad_value_t = pad_value
         while True:
@@ -226,71 +227,86 @@ def padded_batch_func(dataset, batch_size, pad_value=0, max_seqlen=None):
                 if (not np.isscalar(elem)) and elem.shape != ():
                     max_len = max(map(len,
                                       e)) if max_seqlen is None else max_seqlen
-                    e = map(lambda i: np.pad(i, [0, max_len - len(i)], 'constant', constant_values=pv) if max_len >= len(i) else i[: max_len], e)
+
+                    def _fn(i):
+                        if max_len >= len(i):
+                            return np.pad(i, [0, max_len - len(i)],
+                                          'constant',
+                                          constant_values=pv)
+                        else:
+                            return i[:max_len]
+
+                    e = map(_fn, e)
                 padded.append(np.stack(list(e)))
             yield padded
 
-    return gen
+    return _gen
 
 
 class Dataset(object):
+    """Python Wrapper for PyReader"""
+
     @classmethod
-    def from_generator_func(cls, gen, data_shapes=None, data_types=None):
-        if not inspect.isgeneratorfunction(gen):
-            raise ValueError('expect generator function, got %s' % repr(gen))
+    def from_generator_func(cls, _gen, data_shapes=None, data_types=None):
+        """doc"""
+        if not inspect.isgeneratorfunction(_gen):
+            raise ValueError('expect generator function, got %s' % repr(_gen))
 
-        def wrapper():  #compat to py3.7
+        def _wrapper():  #compat to py3.7
             try:
-                for item in gen():
+                for item in _gen():
                     yield item
             except RuntimeError as e:
                 if str(e) != 'generator raised StopIteration':
                     raise e
 
         ret = cls()
-        ret.generator = wrapper
+        ret.generator = _wrapper
         ret.data_shapes = data_shapes
         ret.data_types = data_types
         return ret
 
     @classmethod
     def from_file(cls, filename, format=None):
+        """doc"""
         if os.path.getsize(filename) == 0:
             raise RuntimeError('%s is empty' % filename)
 
-        def gen():
-            with open_file(filename, format) as f:
+        def _gen():
+            with _open_file(filename, format) as f:
                 for line in f:
                     yield line
 
         ret = cls()
-        ret.generator = gen
+        ret.generator = _gen
         ret.data_shapes = []
         ret.data_types = str
         return ret
 
     @classmethod
     def from_record_file(cls, filename):
+        """doc"""
         if os.path.getsize(filename) == 0:
             raise RuntimeError('%s is empty' % filename)
-        gen = open_record(filename)
+        _gen = _open_record(filename)
         ret = cls()
-        ret.generator = gen
+        ret.generator = _gen
         ret.data_shapes = []
         ret.data_types = str
         return ret
 
     @classmethod
     def from_list(cls, ls):
+        """doc"""
         if not isinstance(ls, list):
             raise ValueError('expect list, got %s' % repr(ls))
 
-        def gen():
+        def _gen():
             for i in ls:
                 yield i
 
         ret = cls()
-        ret.generator = gen
+        ret.generator = _gen
         ret.data_shapes = []
         ret.data_types = str
         return ret
@@ -339,6 +355,7 @@ class Dataset(object):
 
     @property
     def data_shapes(self):
+        """doc"""
         if self._data_shapes is None:
             self._infer_shapes_and_types()
             return self._data_shapes
@@ -347,10 +364,12 @@ class Dataset(object):
 
     @data_shapes.setter
     def data_shapes(self, val):
+        """doc"""
         self._data_shapes = val
 
     @property
     def data_types(self):
+        """doc"""
         if self._data_types is None:
             self._infer_shapes_and_types()
             return self._data_types
@@ -359,9 +378,11 @@ class Dataset(object):
 
     @data_types.setter
     def data_types(self, val):
+        """doc"""
         self._data_types = val
 
     def apply(self, transform_func):
+        """apply transform func to datasets"""
         #input_shapes = transform_func.input_shapes
         #input_types = transform_func.input_types
         #data_shapes = transform_func.data_shapes
@@ -377,46 +398,55 @@ class Dataset(object):
         return ret
 
     def shuffle(self, buffer_size):
-        func = functools.partial(shuffle_func, buffer_size=buffer_size)
+        """doc"""
+        func = functools.partial(_shuffle_func, buffer_size=buffer_size)
         return self.apply(func)
 
     def repeat(self, n=-1):
-        func = functools.partial(repeat_func, n=n)
+        """doc"""
+        func = functools.partial(_repeat_func, n=n)
         return self.apply(func)
 
     def map(self, fn):
-        func = functools.partial(map_func, fn=fn)
+        """doc"""
+        func = functools.partial(_map_func, fn=fn)
         return self.apply(func)
 
     def filter(self, fn):
-        func = functools.partial(filter_func, fn=fn)
+        """doc"""
+        func = functools.partial(_filter_func, fn=fn)
         return self.apply(func)
 
     def shard(self, num_shards, index):
+        """doc"""
         func = functools.partial(
-            shard_func, num_shards=num_shards, index=index)
+            _shard_func, num_shards=num_shards, index=index)
         return self.apply(func)
 
     def interleave(self, map_fn, cycle_length, block_length):
+        """doc"""
         func = functools.partial(
-            interleave_func,
+            _interleave_func,
             map_fn=map_fn,
             cycle_length=cycle_length,
             block_length=block_length)
         return self.apply(func)
 
     def padded_batch(self, batch_size, pad_value=0, max_seqlen=None):
+        """doc"""
         func = functools.partial(
-            padded_batch_func,
+            _padded_batch_func,
             batch_size=batch_size,
             pad_value=pad_value,
             max_seqlen=max_seqlen)
         return self.apply(func)
 
     def take(self, count=1):
-        func = functools.partial(take_func, count=count)
+        """doc"""
+        func = functools.partial(_take_func, count=count)
         return self.apply(func)
 
     def buffered(self, size=10):
-        func = functools.partial(buffered_func, size=size)
+        """doc"""
+        func = functools.partial(_buffered_func, size=size)
         return self.apply(func)

propeller/paddle/__init__.py

@@ -11,6 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+doc
+"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals

propeller/paddle/collection.py

@@ -11,6 +11,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""global collections"""
+
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -43,13 +45,16 @@ class Collections(object):
         _global_collection = None
 
     def add(self, key, val):
+        """doc"""
         self.col.setdefault(key, []).append(val)
 
     def get(self, key):
+        """doc"""
         return self.col.get(key, None)
 
 
 def default_collection():
+    """return global collection"""
     global _global_collection
     if _global_collection is None:
         _global_collection = Collections()

propeller/paddle/data/__init__.py

@@ -11,6 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+doc
+"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals

propeller/paddle/data/feature_column.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""FeatureColumns and many Column"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -30,7 +31,7 @@ import numpy as np
 from glob import glob
 from propeller.paddle.train import distribution
 
-from propeller.data.functional import interleave_func
+from propeller.data.functional import _interleave_func
 from propeller.paddle.data.functional import Dataset
 from propeller.paddle.data import example_pb2, feature_pb2
 
@@ -43,35 +44,47 @@ __all__ = [
 
 
 def basic_tokenizer(sen):
+    """doc"""
     seg = sen.split(b' ')
     seg = filter(lambda i: i != b' ', seg)
     return seg
 
 
-class Column():
+class Column(object):
+    """doc"""
+
     def __init__(self, name):
+        """doc"""
         pass
 
     def raw_to_proto(self, raw):
+        """doc"""
         return feature_pb2.Feature()
 
     @property
     def output_shapes(self):
+        """doc"""
         pass
 
     @property
     def output_types(self):
+        """doc"""
         pass
 
     def proto_to_instance(self, proto):
+        """doc"""
         raise NotImplementedError()
 
     def raw_to_instance(self, raw):
+        """doc"""
         raise NotImplementedError()
 
 
 class LabelColumn(Column):
+    """doc"""
+
     def __init__(self, name, vocab_dict=None, vocab_file=None):
+        """doc"""
         self.name = name
         self.vocab = None
         if vocab_file:
@@ -84,13 +97,16 @@ class LabelColumn(Column):
 
     @property
     def output_shapes(self):
+        """doc"""
         return [1]
 
     @property
     def output_types(self):
+        """doc"""
         return 'int64'
 
     def raw_to_proto(self, raw):
+        """doc"""
         if self.vocab is None:
             ids = [int(raw)]
         else:
@@ -99,10 +115,12 @@ class LabelColumn(Column):
         return fe
 
     def proto_to_instance(self, feature):
+        """doc"""
         ret = np.array(feature.int64_list.value[0], dtype=np.int64)
         return ret
 
     def raw_to_instance(self, raw):
+        """doc"""
         if self.vocab is None:
             ids = int(raw)
         else:
@@ -111,6 +129,8 @@ class LabelColumn(Column):
 
 
 class TextColumn(Column):
+    """doc"""
+
     def __init__(self,
                  name,
                  unk_id,
@@ -132,63 +152,75 @@ class TextColumn(Column):
 
     @property
     def output_shapes(self):
+        """doc"""
         return [-1]
 
     @property
     def output_types(self):
+        """doc"""
         return 'int64'
 
     def raw_to_proto(self, raw):
+        """doc"""
         ids = [self.vocab.get(s, self.unk_id) for s in self.tokenizer(raw)]
         fe = feature_pb2.Feature(int64_list=feature_pb2.Int64List(value=ids))
         return fe
 
     def proto_to_instance(self, feature):
+        """doc"""
         ret = np.array(feature.int64_list.value, dtype=np.int64)
         return ret
 
     def raw_to_instance(self, raw):
+        """doc"""
         ids = [self.vocab.get(s, self.unk_id) for s in self.tokenizer(raw)]
         return np.array(ids, dtype=np.int64)
 
 
 class TextIDColumn(Column):
+    """doc"""
+
     def __init__(self, name):
+        """doc"""
         self.name = name
 
     @property
     def output_shapes(self):
+        """doc"""
         return [-1]
 
     @property
     def output_types(self):
+        """doc"""
         return 'int64'
 
     def raw_to_proto(self, raw):
+        """doc"""
         ids = [int(s) for s in raw.split(b' ')]
         fe = feature_pb2.Feature(int64_list=feature_pb2.Int64List(value=ids))
         return fe
 
     def proto_to_instance(self, feature):
+        """doc"""
         ret = np.array(feature.int64_list.value, dtype=np.int64)
         return ret
 
     def raw_to_instance(self, raw):
+        """doc"""
         ret = np.array([int(i) for i in raw.split(b' ')], dtype=np.int64)
         return ret
 
 
-class FeatureColumns(object):
-    def __init__(self, columns, pad_id=0):
-        self._columns = columns
+def _list_files(raw_dir):
+    return [os.path.join(raw_dir, p) for p in os.listdir(raw_dir)]
+
 
-    def raw_files(self, raw_dir):
-        return [os.path.join(raw_dir, p) for p in os.listdir(raw_dir)]
+class FeatureColumns(object):
+    """A Dataset Factory object"""
 
-    def gz_files(self, gz_dir):
-        return None if gz_dir is None else [
-            os.path.join(gz_dir, p) for p in os.listdir(gz_dir)
-        ]
+    def __init__(self, columns):
+        """doc"""
+        self._columns = columns
 
     def _make_gz_dataset(self, raw_dir, gz_dir):
         assert raw_dir or gz_dir, 'data_dir not specified when using gz mode'
@@ -237,7 +269,7 @@ class FeatureColumns(object):
         if shuffle:
             dataset = dataset.shuffle(buffer_size=len(gz_files))
         fn = partial(
-            interleave_func,
+            _interleave_func,
             map_fn=lambda filename: Dataset.from_record_file(filename),
             cycle_length=len(gz_files),
             block_length=1)
@@ -271,7 +303,7 @@ class FeatureColumns(object):
             dataset = dataset.shuffle(buffer_size=len(data_files))
 
         fn = partial(
-            interleave_func,
+            _interleave_func,
             map_fn=lambda filename: Dataset.from_file(filename),
             cycle_length=len(data_files),
             block_length=1)
@@ -294,9 +326,9 @@ class FeatureColumns(object):
     def _read_stdin_dataset(self, encoding='utf8', shuffle=False, **kwargs):
         log.info('reading raw files stdin')
 
-        def gen():
+        def _gen():
             if six.PY3:
-                source = sys.stdin.buffer 
+                source = sys.stdin.buffer
             else:
                 source = sys.stdin
             while True:
@@ -305,12 +337,12 @@ class FeatureColumns(object):
                     break
                 yield line,
 
-        dataset = Dataset.from_generator_func(gen)
+        dataset = Dataset.from_generator_func(_gen)
         if shuffle:
             dataset = dataset.shuffle(buffer_size=1000)
 
         def _parse_stdin(record_str):
-            '''function that takes python_str as input'''
+            """function that takes python_str as input"""
             features = record_str.strip(b'\n').split(b'\t')
             ret = [
                 column.raw_to_instance(feature)
@@ -346,13 +378,17 @@ class FeatureColumns(object):
                       gz_dir=None,
                       data_file=None,
                       **kwargs):
+        """
+        build `Dataset` from `data_dir` or `data_file`
+        if `use_gz`, will try to convert data_files to gz format and save to `gz_dir`, if `gz_dir` not given, will create one.
+        """
         if use_gz:
             gz_dir = self._make_gz_dataset(data_dir, gz_dir)
-            gz_files = self.gz_files(gz_dir)
+            gz_files = _list_files(gz_dir) if gz_dir is not None else gz_dir
             ds = self._read_gz_dataset(gz_files, **kwargs)
         else:
             if data_dir is not None:
-                data_files = self.raw_files(data_dir)
+                data_files = _list_files(data_dir)
             elif data_file is not None:
                 data_files = [data_file]
             else:
@@ -362,6 +398,7 @@ class FeatureColumns(object):
         return ds
 
     def build_dataset_from_stdin(self, name, **kwargs):
+        """doc"""
         ds = self._read_stdin_dataset(**kwargs)
         ds.name = name
         return ds

propeller/paddle/data/functional.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""Pyreader based Dataset"""
 
 import sys
 import numpy as np
@@ -25,7 +26,10 @@ log = logging.getLogger(__name__)
 
 
 class Dataset(DatasetBase):
+    """Pyreader based Dataset"""
+
     def placeholders(self):
+        """doc"""
         if self.name is None:
             raise ValueError('can not get feature from unnamed Dataset')
 
@@ -41,7 +45,7 @@ class Dataset(DatasetBase):
         return ret
 
     def features(self):
-        '''start point of net building. call this in a program scope'''
+        """start point of net building. call this in a program scope"""
         if self.name is None:
             raise ValueError('can not get feature from unnamed Dataset')
 
@@ -51,9 +55,13 @@ class Dataset(DatasetBase):
                 (repr(self._data_shapes), repr(self._data_types)))
         return self.placeholders()
 
-    def start(self, places=F.cuda_places()):
+    def start(self, places=None):
+        """start Pyreader"""
+        if places is None:
+            places = F.cuda_places() if F.core.is_compiled_with_cuda(
+            ) else F.cpu_places()
         #assert self.pyreader is not None, 'use Dataset.features to build net first, then start dataset'
-        def gen():
+        def _gen():
             try:
                 for idx, i in enumerate(self.generator()):
                     yield i
@@ -63,5 +71,5 @@ class Dataset(DatasetBase):
 
         r = F.io.PyReader(
             feed_list=self.placeholders(), capacity=50, iterable=True)
-        r.decorate_batch_generator(gen, places=places)
+        r.decorate_batch_generator(_gen, places=places)
         return r()

propeller/paddle/service/__init__.py

-# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.

propeller/paddle/service/server.py

-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import division
-from __future__ import absolute_import
-from __future__ import print_function
-from __future__ import unicode_literals
-
-import logging
-import six
-import asyncio
-import threading
-
-import grpc
-from propeller.service import interface_pb2
-from propeller.service import interface_pb2_grpc
-import propeller.paddle.service.utils as serv_utils
-
-from concurrent.futures import ThreadPoolExecutor
-
-import paddle.fluid as F
-
-from time import sleep, time
-
-log = logging.getLogger(__name__)
-
-
-def profile(msg):
-    def decfn(fn):
-        def retfn(*args, **kwargs):
-            start = time()
-            ret = fn(*args, **kwargs)
-            end = time()
-            log.debug('%s timecost: %.5f' % (msg, end - start))
-            return ret
-
-        return retfn
-
-    return decfn
-
-
-def serve(model_dir, host, num_concurrent=None):
-    if six.PY2:
-        raise RuntimeError('propeller service work in python3 only')
-    num_worker = len(F.cuda_places(
-    )) if num_concurrent is None else num_concurrent
-    pool = ThreadPoolExecutor(num_worker)
-
-    class Predictor(object):
-        def __init__(self, did):
-            log.debug('create predictor on card %d' % did)
-            config = F.core.AnalysisConfig(model_dir)
-            config.enable_use_gpu(5000, did)
-            self._predictor = F.core.create_paddle_predictor(config)
-
-        @profile('paddle')
-        def __call__(self, args):
-            for i, a in enumerate(args):
-                a.name = 'placeholder_%d' % i
-            res = self._predictor.run(args)
-            return res
-
-    predictor_context = {}
-
-    class InferenceService(interface_pb2_grpc.InferenceServicer):
-        @profile('service')
-        def Infer(self, request, context):
-            try:
-                slots = request.slots
-                current_thread = threading.current_thread()
-                log.debug('%d slots received dispatch to thread %s' %
-                          (len(slots), current_thread))
-                if current_thread not in predictor_context:
-                    did = list(pool._threads).index(current_thread)
-                    log.debug('spawning worker thread %d' % did)
-                    predictor = Predictor(did)
-                    predictor_context[current_thread] = predictor
-                else:
-                    predictor = predictor_context[current_thread]
-                slots = [serv_utils.slot_to_paddlearray(s) for s in slots]
-                ret = predictor(slots)
-                response = [serv_utils.paddlearray_to_slot(r) for r in ret]
-            except Exception as e:
-                log.exception(e)
-                raise e
-            return interface_pb2.Slots(slots=response)
-
-    server = grpc.server(pool)
-    interface_pb2_grpc.add_InferenceServicer_to_server(InferenceService(),
-                                                       server)
-    server.add_insecure_port(host)
-    server.start()
-    log.info('server started on %s...' % host)
-    try:
-        while True:
-            sleep(100000)
-    except KeyboardInterrupt as e:
-        pass
-    log.info('server stoped...')
-
-
-if __name__ == '__main__':
-    from propeller import log
-    log.setLevel(logging.DEBUG)
-    serve(
-        '/home/work/chenxuyi/playground/grpc_play/ernie2.0/',
-        '10.255.138.19:8334',
-        num_concurrent=3)

propeller/paddle/service/utils.py

-#   Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import division
-from __future__ import absolute_import
-from __future__ import print_function
-from __future__ import unicode_literals
-
-import struct
-
-from propeller.service import interface_pb2
-from propeller.service import interface_pb2_grpc
-
-import paddle.fluid.core as core
-
-
-def slot_to_paddlearray(slot):
-    if slot.type == interface_pb2.Slot.FP32:
-        type_str = 'f'
-        dtype = core.PaddleDType.FLOAT32
-    elif slot.type == interface_pb2.Slot.INT32:
-        type_str = 'i'
-        dtype = core.PaddleDType.INT32
-    elif slot.type == interface_pb2.Slot.INT64:
-        type_str = 'q'
-        dtype = core.PaddleDType.INT64
-    else:
-        raise RuntimeError('know type %s' % slot.type)
-    ret = core.PaddleTensor()
-    ret.shape = slot.dims
-    ret.dtype = dtype
-    num = len(slot.data) // struct.calcsize(type_str)
-    arr = struct.unpack('%d%s' % (num, type_str), slot.data)
-    ret.data = core.PaddleBuf(arr)
-    return ret
-
-
-def paddlearray_to_slot(arr):
-    if arr.dtype == core.PaddleDType.FLOAT32:
-        dtype = interface_pb2.Slot.FP32
-        type_str = 'f'
-        arr_data = arr.data.float_data()
-    elif arr.dtype == core.PaddleDType.INT32:
-        dtype = interface_pb2.Slot.INT32
-        type_str = 'i'
-        arr_data = arr.data.int32_data()
-    elif arr.dtype == core.PaddleDType.INT64:
-        dtype = interface_pb2.Slot.INT64
-        type_str = 'q'
-        arr_data = arr.data.int64_data()
-    else:
-        raise RuntimeError('know type %s' % arr.dtype)
-    data = struct.pack('%d%s' % (len(arr_data), type_str), *arr_data)
-    pb = interface_pb2.Slot(type=dtype, dims=list(arr.shape), data=data)
-    return pb

propeller/paddle/summary.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""record summary tensor in a collection scope"""
 
 from __future__ import print_function
 from __future__ import absolute_import
@@ -23,6 +24,7 @@ from propeller.paddle.collection import default_collection, Key
 
 
 def scalar(name, tensor):
+    """scalar summary"""
     if not isinstance(tensor, F.framework.Variable):
         raise ValueError('expect paddle Variable, got %s' % repr(tensor))
     tensor.persistable = True
@@ -30,6 +32,7 @@ def scalar(name, tensor):
 
 
 def histogram(name, tensor):
+    """histogram summary"""
     if not isinstance(tensor, F.framework.Variable):
         raise ValueError('expect paddle Variable, got %s' % repr(tensor))
     tensor.persistable = True

propeller/paddle/train/__init__.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""Propeller training"""
 
 from __future__ import print_function
 from __future__ import absolute_import

propeller/paddle/train/distribution.py

@@ -128,6 +128,7 @@ def init_distribuition_env(program):
     elif status.mode == DistributionMode.NCCL:
         config = F.DistributeTranspilerConfig()
         config.mode = "nccl2"
+        config.nccl_comm_num = 1
         F.DistributeTranspiler(config=config).transpile(
             status.replica_id,
             trainers=','.join(status._env),

propeller/paddle/train/exporter.py

@@ -11,6 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+exporters
+"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -19,6 +22,7 @@ import sys
 import os
 import itertools
 import six
+import inspect
 import abc
 import logging
 
@@ -28,24 +32,36 @@ import paddle.fluid.layers as L
 
 from propeller.paddle.train import Saver
 from propeller.types import InferenceSpec
+from propeller.train.model import Model
+from propeller.paddle.train.trainer import _build_net
+from propeller.paddle.train.trainer import _build_model_fn
+from propeller.types import RunMode
+from propeller.types import ProgramPair
 
 log = logging.getLogger(__name__)
 
 
 @six.add_metaclass(abc.ABCMeta)
-class Exporter():
+class Exporter(object):
+    """base exporter"""
+
     @abc.abstractmethod
     def export(self, exe, program, eval_result, state):
+        """export"""
         raise NotImplementedError()
 
 
 class BestExporter(Exporter):
+    """export saved model accordingto `cmp_fn`"""
+
     def __init__(self, export_dir, cmp_fn):
+        """doc"""
         self._export_dir = export_dir
         self._best = None
         self.cmp_fn = cmp_fn
 
     def export(self, exe, program, eval_model_spec, eval_result, state):
+        """doc"""
         log.debug('New evaluate result: %s \nold: %s' %
                   (repr(eval_result), repr(self._best)))
         if self._best is None or self.cmp_fn(old=self._best, new=eval_result):
@@ -65,40 +81,85 @@ class BestExporter(Exporter):
 
 
 class BestInferenceModelExporter(Exporter):
-    def __init__(self, export_dir, cmp_fn):
+    """export inference model accordingto `cmp_fn`"""
+
+    def __init__(self,
+                 export_dir,
+                 cmp_fn,
+                 model_class_or_model_fn=None,
+                 hparams=None,
+                 dataset=None):
+        """doc"""
         self._export_dir = export_dir
         self._best = None
         self.cmp_fn = cmp_fn
+        self.model_class_or_model_fn = model_class_or_model_fn
+        self.hparams = hparams
+        self.dataset = dataset
 
     def export(self, exe, program, eval_model_spec, eval_result, state):
+        """doc"""
+        if self.model_class_or_model_fn is not None and self.hparams is not None \
+                and self.dataset is not None:
+            log.info('Building program by user defined model function')
+            if issubclass(self.model_class_or_model_fn, Model):
+                _model_fn = _build_model_fn(self.model_class_or_model_fn)
+            elif inspect.isfunction(self.model_class_or_model_fn):
+                _model_fn = self.model_class_or_model_fn
+            else:
+                raise ValueError('unknown model %s' %
+                                 self.model_class_or_model_fn)
+
+            # build net
+            infer_program = F.Program()
+            startup_prog = F.Program()
+            with F.program_guard(infer_program, startup_prog):
+                #share var with Train net
+                with F.unique_name.guard():
+                    log.info('Building Infer Graph')
+                    infer_fea = self.dataset.features()
+                    # run_config is None
+                    self.model_spec = _build_net(_model_fn, infer_fea,
+                                                 RunMode.PREDICT, self.hparams,
+                                                 None)
+                    log.info('Done')
+            infer_program = infer_program.clone(for_test=True)
+            self.program = ProgramPair(
+                train_program=infer_program, startup_program=startup_prog)
+
+        else:
+            self.program = program
+            self.model_spec = eval_model_spec
+
         log.debug('New evaluate result: %s \nold: %s' %
                   (repr(eval_result), repr(self._best)))
         if self._best is None or self.cmp_fn(old=self._best, new=eval_result):
             log.debug('[Best Exporter]: export to %s' % self._export_dir)
-            if eval_model_spec.inference_spec is None:
+            if self.model_spec.inference_spec is None:
                 raise ValueError('model_fn didnt return InferenceSpec')
 
-            inf_sepc_dict = eval_model_spec.inference_spec
-            if not isinstance(inf_sepc_dict, dict):
-                inf_sepc_dict = {'inference': inf_sepc_dict}
-            for inf_sepc_name, inf_sepc in six.iteritems(inf_sepc_dict):
-                if not isinstance(inf_sepc, InferenceSpec):
-                    raise ValueError('unkonw inference spec type: %s' % v)
+            inf_spec_dict = self.model_spec.inference_spec
+            if not isinstance(inf_spec_dict, dict):
+                inf_spec_dict = {'inference': inf_spec_dict}
+            for inf_spec_name, inf_spec in six.iteritems(inf_spec_dict):
+                if not isinstance(inf_spec, InferenceSpec):
+                    raise ValueError('unknow inference spec type: %s' %
+                                     inf_spec)
 
-                save_dir = os.path.join(self._export_dir, inf_sepc_name)
+                save_dir = os.path.join(self._export_dir, inf_spec_name)
                 log.debug('[Best Exporter]: save inference model: "%s" to %s' %
-                          (inf_sepc_name, save_dir))
-                feed_var = [i.name for i in inf_sepc.inputs]
-                fetch_var = inf_sepc.outputs
+                          (inf_spec_name, save_dir))
+                feed_var = [i.name for i in inf_spec.inputs]
+                fetch_var = inf_spec.outputs
 
-                eval_program = program.train_program
+                infer_program = self.program.train_program
                 startup_prog = F.Program()
                 F.io.save_inference_model(
                     save_dir,
                     feed_var,
                     fetch_var,
                     exe,
-                    main_program=eval_program)
+                    main_program=infer_program)
             self._best = eval_result
         else:
             log.debug('[Best Exporter]: skip step %s' % state.gstep)

propeller/paddle/train/hooks.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""train hooks"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -38,44 +39,56 @@ log = logging.getLogger(__name__)
 
 
 class RunHook(object):
+    """RunHook Base class"""
+
     def __init__(self):
+        """doc"""
         pass
 
-    def before_train(self):
+    def before_train(self, program):
+        """doc"""
         pass
 
     def before_run(self, state):
+        """doc"""
         return []
 
     def after_run(self, res_list, state):
+        """doc"""
         pass
 
     def should_stop(self, state):
+        """doc"""
         return False
 
     def after_train(self):
+        """doc"""
         pass
 
 
 class TqdmProgressBarHook(RunHook):
+    """show a progress bar when training"""
+
     def __init__(self, max_steps, desc=None):
+        """doc"""
         self.tqdm = None
         import tqdm
         from propeller import log as main_log
         hdl = main_log.handlers[0]
 
-        class TqdmLogginHandler(logging.Handler):
+        class _TqdmLogginHandler(logging.Handler):
             def emit(self, record):
+                """doc"""
                 try:
                     msg = self.format(record)
                     tqdm.tqdm.write(msg, file=sys.stderr)
                     self.flush()
-                except (KeyboardInterrupt, SystemExit):
-                    raise
+                except (KeyboardInterrupt, SystemExit) as e:
+                    raise e
                 except:
                     self.handleError(record)
 
-        tqdm_hdl = TqdmLogginHandler()
+        tqdm_hdl = _TqdmLogginHandler()
         tqdm_hdl.setFormatter(hdl.formatter)
         main_log.removeHandler(hdl)
         main_log.addHandler(tqdm_hdl)
@@ -91,46 +104,55 @@ class TqdmProgressBarHook(RunHook):
 
 
 class TqdmNotebookProgressBarHook(RunHook):
+    """show a progress bar when training"""
+
     def __init__(self, max_steps, desc=None):
+        """doc"""
         self.tqdm = None
         import tqdm
         from propeller import log as main_log
         hdl = main_log.handlers[0]
 
-        class TqdmLogginHandler(logging.Handler):
+        class _TqdmLogginHandler(logging.Handler):
             def emit(self, record):
+                """doc"""
                 try:
                     msg = self.format(record)
                     tqdm.tqdm.write(msg, file=sys.stderr)
                     self.flush()
-                except (KeyboardInterrupt, SystemExit):
-                    raise
+                except (KeyboardInterrupt, SystemExit) as e:
+                    raise e
                 except:
                     self.handleError(record)
 
-        tqdm_hdl = TqdmLogginHandler()
+        tqdm_hdl = _TqdmLogginHandler()
         tqdm_hdl.setFormatter(hdl.formatter)
         main_log.removeHandler(hdl)
         main_log.addHandler(tqdm_hdl)
         self.tqdm = tqdm.tqdm_notebook(total=max_steps, desc=None)
 
     def before_run(self, state):
+        """doc"""
         self.tqdm.n = state.gstep
         self.tqdm.refresh()
         return []
 
     def __del__(self):
+        """doc"""
         if self.tqdm:
             self.tqdm.close()
 
 
 class LoggingHook(RunHook):
+    """log tensor in to screan and tensorboard"""
+
     def __init__(self,
                  loss,
                  per_step=10,
                  skip_step=100,
                  summary_writer=None,
                  summary_record=None):
+        """doc"""
         if per_step is None or skip_step is None:
             raise ValueError('wrong step argument, per step: %d skip_step %d' %
                              (per_step, skip_step))
@@ -141,7 +163,8 @@ class LoggingHook(RunHook):
         self.writer = summary_writer
         self.last_state = None
 
-    def before_train(self):
+    def before_train(self, program):
+        """doc"""
         if self.summary_record:
             if self.summary_record.scalar:
                 self.s_name, self.s_tolog = zip(*self.summary_record.scalar)
@@ -154,6 +177,7 @@ class LoggingHook(RunHook):
                 self.h_name, self.h_tolog = [], []
 
     def before_run(self, state):
+        """doc"""
         if state.gstep % self.per_step == 0 and state.step > self.skip_step:
             ret = [self.loss]
             if self.summary_record:
@@ -164,6 +188,7 @@ class LoggingHook(RunHook):
             return []
 
     def after_run(self, res_list, state):
+        """doc"""
         if state.gstep % self.per_step == 0 and state.step > self.skip_step:
             if not self.summary_record:
                 return
@@ -209,11 +234,15 @@ class LoggingHook(RunHook):
 
 
 class StopAtStepHook(RunHook):
+    """stop training at some step"""
+
     def __init__(self, stop_global_step, stop_step):
+        """doc"""
         self._stop_gstep = stop_global_step
         self._stop_step = stop_step
 
     def should_stop(self, state):
+        """doc"""
         if (self._stop_gstep and state.gstep >= self._stop_gstep) or \
            (self._stop_step and state.step >= self._stop_step):
             log.info('StopAtStepHook called stop')
@@ -226,6 +255,7 @@ class EvalHook(RunHook):
     """hook this on a eval Executor"""
 
     def __init__(self, metrics, summary_writer=None):
+        """doc"""
         self.writer = summary_writer
         self._result = None
 
@@ -244,11 +274,13 @@ class EvalHook(RunHook):
         else:
             self.names, self.metrics = [], []
 
-    def before_train(self):
+    def before_train(self, program):
+        """doc"""
         for m in self.metrics:
             m.reset()
 
     def before_run(self, state):
+        """doc"""
         ls = [m.tensor for m in self.metrics]
         for i in ls:
             if not (isinstance(i, list) or isinstance(i, tuple)):
@@ -265,15 +297,18 @@ class EvalHook(RunHook):
         return ls_flt
 
     def after_run(self, res_list, state):
+        """doc"""
         res = util.unflatten(res_list, self.schema)
         for r, m in zip(res, self.metrics):
             m.update(r)
 
     @property
     def result(self):
+        """doc"""
         return self._result
 
     def after_train(self):
+        """doc"""
         printable = []
         self._result = {}
         for n, m in zip(self.names, self.metrics):
@@ -284,12 +319,16 @@ class EvalHook(RunHook):
 
 
 class CheckpointSaverHook(RunHook):
+    """Save checkpoint every n step"""
+
     def __init__(self, saver, per_step=10, skip_step=100):
+        """doc"""
         self.saver = saver
         self.per_step = per_step
         self.skip_step = skip_step
 
     def after_run(self, res_list, state):
+        """doc"""
         if state.gstep % self.per_step == 0 and \
                 state.step > self.skip_step:
             self.saver.save(state)

propeller/paddle/train/metrics.py

@@ -11,9 +11,12 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""predefined metrics"""
 
 import sys
 import os
+import six
+
 import numpy as np
 import itertools
 import logging
@@ -31,98 +34,132 @@ __all__ = [
 
 
 class Metrics(object):
+    """Metrics base class"""
+
     def __init__(self):
+        """doc"""
         self.saver = []
 
     @property
     def tensor(self):
+        """doc"""
         pass
 
     def update(self, *args):
+        """doc"""
         pass
 
     def eval(self):
+        """doc"""
         pass
 
 
 class Mean(Metrics):
+    """doc"""
+
     def __init__(self, t):
+        """doc"""
         self.t = t
         self.reset()
 
     def reset(self):
+        """doc"""
         self.saver = np.array([])
 
     @property
     def tensor(self):
+        """doc"""
         self.t.persistable = True
         return self.t,
 
     def update(self, args):
+        """doc"""
         t, = args
         t = t.reshape([-1])
         self.saver = np.concatenate([self.saver, t])
 
     def eval(self):
+        """doc"""
         return self.saver.mean()
 
 
 class Ppl(Mean):
+    """doc"""
+
     def eval(self):
+        """doc"""
         return np.exp(self.saver.mean())
 
 
 class Acc(Mean):
+    """doc"""
+
     def __init__(self, label, pred):
+        """doc"""
         self.eq = L.equal(pred, label)
         self.reset()
 
     @property
     def tensor(self):
+        """doc"""
         self.eq.persistable = True
         return self.eq,
 
 
 class MSE(Mean):
+    """doc"""
+
     def __init__(self, label, pred):
+        """doc"""
         diff = pred - label
         self.mse = diff * diff
         self.reset()
 
     @property
     def tensor(self):
+        """doc"""
         self.mse.persistable = True
         return self.mse,
 
 
 class Cosine(Mean):
+    """doc"""
+
     def __init__(self, label, pred):
+        """doc"""
         self.cos = L.cos_sim(label, pred)
         self.reset()
 
     @property
     def tensor(self):
+        """doc"""
         self.cos.persistable = True
         return self.cos,
 
 
 class Precision(Metrics):
+    """doc"""
+
     def __init__(self, label, pred):
+        """doc"""
         self.label = label
         self.pred = pred
         self.reset()
 
     def reset(self):
+        """doc"""
         self.label_saver = np.array([], dtype=np.bool)
         self.pred_saver = np.array([], dtype=np.bool)
 
     @property
     def tensor(self):
+        """doc"""
         self.label.persistable = True
         self.pred.persistable = True
         return self.label, self.pred
 
     def update(self, args):
+        """doc"""
         label, pred = args
         label = label.reshape([-1]).astype(np.bool)
         pred = pred.reshape([-1]).astype(np.bool)
@@ -134,20 +171,27 @@ class Precision(Metrics):
         self.pred_saver = np.concatenate([self.pred_saver, pred])
 
     def eval(self):
+        """doc"""
         tp = (self.label_saver & self.pred_saver).astype(np.int64).sum()
         t = self.label_saver.astype(np.int64).sum()
         return tp / t
 
 
 class Recall(Precision):
+    """doc"""
+
     def eval(self):
+        """doc"""
         tp = (self.label_saver & self.pred_saver).astype(np.int64).sum()
         p = (self.label_saver).astype(np.int64).sum()
         return tp / p
 
 
 class F1(Precision):
+    """doc"""
+
     def eval(self):
+        """doc"""
         tp = (self.label_saver & self.pred_saver).astype(np.int64).sum()
         t = self.label_saver.astype(np.int64).sum()
         p = self.pred_saver.astype(np.int64).sum()
@@ -157,22 +201,28 @@ class F1(Precision):
 
 
 class Auc(Metrics):
+    """doc"""
+
     def __init__(self, label, pred):
+        """doc"""
         self.pred = pred
         self.label = label
         self.reset()
 
     def reset(self):
+        """doc"""
         self.pred_saver = np.array([], dtype=np.float32)
         self.label_saver = np.array([], dtype=np.bool)
 
     @property
     def tensor(self):
+        """doc"""
         self.pred.persistable = True
         self.label.persistable = True
         return [self.pred, self.label]
 
     def update(self, args):
+        """doc"""
         pred, label = args
         pred = pred.reshape([-1]).astype(np.float32)
         label = label.reshape([-1]).astype(np.bool)
@@ -180,6 +230,7 @@ class Auc(Metrics):
         self.label_saver = np.concatenate([self.label_saver, label])
 
     def eval(self):
+        """doc"""
         fpr, tpr, thresholds = sklearn.metrics.roc_curve(
             self.label_saver.astype(np.int64), self.pred_saver)
         auc = sklearn.metrics.auc(fpr, tpr)
@@ -187,11 +238,15 @@ class Auc(Metrics):
 
 
 class RecallAtPrecision(Auc):
+    """doc"""
+
     def __init__(self, label, pred, precision=0.9):
+        """doc"""
         super(RecallAtPrecision, self).__init__(label, pred)
         self.precision = precision
 
     def eval(self):
+        """doc"""
         self.pred_saver = self.pred_saver.reshape(
             [self.label_saver.size, -1])[:, -1]
         precision, recall, thresholds = sklearn.metrics.precision_recall_curve(
@@ -202,11 +257,15 @@ class RecallAtPrecision(Auc):
 
 
 class PrecisionAtThreshold(Auc):
+    """doc"""
+
     def __init__(self, label, pred, threshold=0.5):
+        """doc"""
         super().__init__(label, pred)
         self.threshold = threshold
 
     def eval(self):
+        """doc"""
         infered = self.pred_saver > self.threshold
         correct_num = np.array(infered & self.label_saver).sum()
         infer_num = infered.sum()
@@ -214,25 +273,31 @@ class PrecisionAtThreshold(Auc):
 
 
 class Mrr(Metrics):
+    """doc"""
+
     def __init__(self, qid, label, pred):
+        """doc"""
         self.qid = qid
         self.label = label
         self.pred = pred
         self.reset()
 
     def reset(self):
+        """doc"""
         self.qid_saver = np.array([], dtype=np.int64)
         self.label_saver = np.array([], dtype=np.int64)
         self.pred_saver = np.array([], dtype=np.float32)
 
     @property
     def tensor(self):
+        """doc"""
         self.qid.persistable = True
         self.label.persistable = True
         self.pred.persistable = True
         return [self.qid, self.label, self.pred]
 
     def update(self, args):
+        """doc"""
         qid, label, pred = args
         if not (qid.shape[0] == label.shape[0] == pred.shape[0]):
             raise ValueError(
@@ -246,10 +311,12 @@ class Mrr(Metrics):
             [self.pred_saver, pred.reshape([-1]).astype(np.float32)])
 
     def eval(self):
-        def key_func(tup):
+        """doc"""
+
+        def _key_func(tup):
             return tup[0]
 
-        def calc_func(tup):
+        def _calc_func(tup):
             ranks = [
                 1. / (rank + 1.)
                 for rank, (_, l, p) in enumerate(
@@ -262,19 +329,22 @@ class Mrr(Metrics):
                 return 0.
 
         mrr_for_qid = [
-            calc_func(tup)
+            _calc_func(tup)
             for _, tup in itertools.groupby(
                 sorted(
                     zip(self.qid_saver, self.label_saver, self.pred_saver),
-                    key=key_func),
-                key=key_func)
+                    key=_key_func),
+                key=_key_func)
         ]
         mrr = np.float32(sum(mrr_for_qid) / len(mrr_for_qid))
         return mrr
 
 
 class ChunkF1(Metrics):
+    """doc"""
+
     def __init__(self, label, pred, seqlen, num_label):
+        """doc"""
         self.label = label
         self.pred = pred
         self.seqlen = seqlen
@@ -327,18 +397,21 @@ class ChunkF1(Metrics):
         return chunks
 
     def reset(self):
+        """doc"""
         self.label_cnt = 0
         self.pred_cnt = 0
         self.correct_cnt = 0
 
     @property
     def tensor(self):
+        """doc"""
         self.pred.persistable = True
         self.label.persistable = True
         self.seqlen.persistable = True
         return [self.pred, self.label, self.seqlen]
 
     def update(self, args):
+        """doc"""
         pred, label, seqlen = args
         pred = pred.reshape([-1]).astype(np.int32).tolist()
         label = label.reshape([-1]).astype(np.int32).tolist()
@@ -374,6 +447,7 @@ class ChunkF1(Metrics):
                     label_index += 1
 
     def eval(self):
+        """doc"""
         if self.pred_cnt == 0:
             precision = 0.0
         else:
@@ -393,23 +467,29 @@ class ChunkF1(Metrics):
 
 
 class PNRatio(Metrics):
+    """doc"""
+
     def __init__(self, qid, label, pred):
+        """doc"""
         self.qid = qid
         self.label = label
         self.pred = pred
         self.saver = {}
 
     def reset(self):
+        """doc"""
         self.saver = {}
 
     @property
     def tensor(self):
+        """doc"""
         self.qid.persistable = True
         self.label.persistable = True
         self.pred.persistable = True
         return [self.qid, self.label, self.pred]
 
     def update(self, args):
+        """doc"""
         qid, label, pred = args
         if not (qid.shape[0] == label.shape[0] == pred.shape[0]):
             raise ValueError('dimention not match: qid[%s] label[%s], pred[%s]'
@@ -424,6 +504,7 @@ class PNRatio(Metrics):
             self.saver[q].append((l, p))
 
     def eval(self):
+        """doc"""
         p = 0
         n = 0
         for qid, outputs in self.saver.items():
@@ -446,10 +527,14 @@ class PNRatio(Metrics):
 
 
 class BinaryPNRatio(PNRatio):
+    """doc"""
+
     def __init__(self, qid, label, pred):
+        """doc"""
         super(BinaryPNRatio, self).__init__(qid, label, pred)
 
     def eval(self):
+        """doc"""
         p = 0
         n = 0
         for qid, outputs in self.saver.items():
@@ -474,7 +559,10 @@ class BinaryPNRatio(PNRatio):
 
 
 class PrecisionAtK(Metrics):
+    """doc"""
+
     def __init__(self, qid, label, pred, k=1):
+        """doc"""
         self.qid = qid
         self.label = label
         self.pred = pred
@@ -482,16 +570,19 @@ class PrecisionAtK(Metrics):
         self.saver = {}
 
     def reset(self):
+        """doc"""
         self.saver = {}
 
     @property
     def tensor(self):
+        """doc"""
         self.qid.persistable = True
         self.label.persistable = True
         self.pred.persistable = True
         return [self.qid, self.label, self.pred]
 
     def update(self, args):
+        """doc"""
         qid, label, pred = args
         if not (qid.shape[0] == label.shape[0] == pred.shape[0]):
             raise ValueError('dimention not match: qid[%s] label[%s], pred[%s]'
@@ -507,6 +598,7 @@ class PrecisionAtK(Metrics):
             self.saver[q].append((l, p))
 
     def eval(self):
+        """doc"""
         right = 0
         total = 0
         for v in self.saver.values():

propeller/paddle/train/monitored_executor.py

@@ -11,6 +11,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+doc
+"""
+
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -37,9 +41,17 @@ log = logging.getLogger(__name__)
 __all__ = ['MonitoredExecutor', 'Saver']
 
 
+def _get_one_place():
+    return F.cuda_places()[0] if F.core.is_compiled_with_cuda(
+    ) else F.cpu_places()[0]
+
+
 class RunState(object):
+    """serializable Run state object"""
+
     @classmethod
     def from_str(cls, s):
+        """doc"""
         j = json.loads(s)
         ret = RunState()
         ret._gstep = j['global_step']
@@ -48,29 +60,36 @@ class RunState(object):
         return ret
 
     def __init__(self):
+        """doc"""
         self._gstep = 0
         self._step = 0
         self._time = time()
 
     @property
     def gstep(self):
+        """doc"""
         return self._gstep
 
     @property
     def step(self):
+        """doc"""
         return self._step
 
     @property
     def time(self):
+        """doc"""
         return self._time
 
     def __repr__(self):
+        """doc"""
         return repr({'global_step': self._gstep, 'time': self._time})
 
     def serialize(self):
+        """doc"""
         return json.dumps({'global_step': self._gstep, 'time': self._time})
 
     def next(self):
+        """doc"""
         ret = RunState()
         ret._gstep = self._gstep + 1
         ret._step = self._step + 1
@@ -79,12 +98,15 @@ class RunState(object):
 
 
 class Saver(object):
+    """checkpoint saver and manager"""
+
     def __init__(self,
                  save_dir,
                  exe,
                  program,
                  save_prefix='model',
                  max_ckpt_to_keep=None):
+        """doc"""
         if exe is not None:
             assert isinstance(
                 exe, F.Executor
@@ -108,9 +130,11 @@ class Saver(object):
 
     @property
     def last_ckpt(self):
+        """doc"""
         return self.ckpt_list[-1] if len(self.ckpt_list) else None
 
     def save(self, state):
+        """doc"""
         save_name = '%s_%d' % (self._save_prefix, state.gstep)
         save_dir = os.path.join(self._save_dir, save_name)
         tmp_dir = os.path.join(self._save_dir, 'tmp')
@@ -139,28 +163,26 @@ class Saver(object):
         open(self.ckpt_info_path, 'w').write('\n'.join(self.ckpt_list))
 
     def restore(self, ckpt=-1):
-        if not isinstance(ckpt, (int, ) + six.string_types):
-            raise ValueError('ckpt type not understood %s' % repr(ckpt))
+        """doc"""
         if isinstance(ckpt, int):
             try:
-                ckpt = self.ckpt_list[ckpt]
+                path = os.path.join(self._save_dir, self.ckpt_list[ckpt])
             except IndexError:
                 raise ValueError('invalid restore ckpt number %d' % ckpt)
-        if isinstance(ckpt, six.string_types):
-            try:
-                ckpt = self.ckpt_list.index(ckpt)
-            except ValueError:
-                raise ValueError('ckpt: %s not in ckpt list: %s' %
-                                 (ckpt, self.ckpt_list))
+        elif isinstance(ckpt, six.string_types):
+            if not os.path.exists(ckpt):
+                raise ValueError('ckpt: %s not found' % ckpt)
+            path = ckpt
+        else:
+            raise ValueError('ckpt type not understood %s' % repr(ckpt))
 
-        path = os.path.join(self._save_dir, self.ckpt_list[ckpt])
         meta_file = os.path.join(path, 'meta')
         if not os.path.exists(meta_file):
             raise RuntimeError('meta not found in restore dir: %s' % path)
         state = RunState.from_str(open(meta_file).read())
         log.info('restore from ckpt %s, ckpt-status: %s' % (path, repr(state)))
 
-        def fn(v):
+        def _fn(v):
             vpath = os.path.join(path, v.name)
             if F.io.is_persistable(v):
                 if os.path.exists(vpath):
@@ -171,12 +193,12 @@ class Saver(object):
             return False
 
         F.io.load_vars(
-            self._exe, path, main_program=self._program, predicate=fn)
+            self._exe, path, main_program=self._program, predicate=_fn)
         return state
 
 
 class MonitoredExecutor(object):
-    """A wrapper handling the train loop"""
+    """An Executor wrapper handling the train loop"""
 
     def __init__(
             self,
@@ -209,13 +231,18 @@ class MonitoredExecutor(object):
 
     @property
     def state(self):
+        """doc"""
         return self._state
 
-    def init_or_restore_variables(self):
+    def init_or_restore_variables(self, ckpt=-1):
+        """
+        init vars or restore vars from model_dir
+        call before train
+        """
         # The order of this 2 steps really matters
         # 1. init train
 
-        F.Executor(F.cuda_places()[0]).run(self._program.startup_program)
+        F.Executor(_get_one_place()).run(self._program.startup_program)
         # 2. restore param
         if self._warm_start_setting is not None:
             if not os.path.exists(self._warm_start_setting.from_dir):
@@ -224,29 +251,34 @@ class MonitoredExecutor(object):
             log.info("warm start from %s" % self._warm_start_setting.from_dir)
             if self._warm_start_setting.predicate_fn is not None:
 
-                def fn(v):
+                def _fn(v):
                     ret = self._warm_start_setting.predicate_fn(v)
                     if ret:
                         log.info('warm start: %s' % v.name)
                     return ret
 
                 F.io.load_vars(
-                    F.Executor(F.cuda_places()[0]),
+                    F.Executor(_get_one_place()),
                     self._warm_start_setting.from_dir,
                     main_program=self._program.train_program,
-                    predicate=fn)
+                    predicate=_fn)
             else:
                 raise NotImplementedError()
 
         self._saver = Saver(
             self._model_dir,
-            F.Executor(F.cuda_places()[0]),
+            F.Executor(_get_one_place()),
             program=self._program.train_program,
             max_ckpt_to_keep=self._max_ckpt)
         if self._saver.last_ckpt is not None:
-            self._state = self._saver.restore()
-
-    def freeze(self):
+            self._state = self._saver.restore(ckpt)
+
+    def _freeze(self):
+        """
+        call before enter train loop
+        convert program to compiled program
+        will do nothing if loss is None i.e. not in train mode
+        """
         if self._loss is None:
             log.debug('will not freeze a program without loss')
             return
@@ -278,8 +310,16 @@ class MonitoredExecutor(object):
             startup_program=self._program.startup_program)
 
     def __enter__(self):
+        """
+        prepapre before enter train loop
+        """
+        if F.core.is_compiled_with_cuda():
+            log.info('propeller runs in CUDA mode')
+        else:
+            log.info('propeller runs in CPU mode')
+
         log.debug('freezing program')
-        self.freeze()
+        self._freeze()
         log.debug('done freezing')
         log.info('********** Start Loop ************')
         # TODO init
@@ -287,10 +327,13 @@ class MonitoredExecutor(object):
         self.result = None
         for h in self._hooks:
             log.debug('train loop has hook %s' % h)
-            h.before_train()
+            h.before_train(self._program)
         return self
 
     def run(self, fetch_list=[], *args, **kwargs):
+        """
+        wrapper for Executor.run
+        """
         #log.debug('Executor running step %d' % self._state.gstep)
         if self._hooks:
             fetch_list = [fetch_list]
@@ -306,11 +349,12 @@ class MonitoredExecutor(object):
             ]
             #if len(set(fetch_list)) != len(fetch_list):
             #    log.error('strange shit happend when fetch list has idetity tensors %s' % fetch_list)
+            #log.debug(fetch_list)
             res = self._exe.run(self._program.train_program,
                                 fetch_list=fetch_list,
                                 *args,
                                 **kwargs)
-            res = [self.merge_result(r) for r in res]
+            res = [self._merge_result(r) for r in res]
             #log.debug(res)
 
             res = util.unflatten(res, schema)
@@ -330,6 +374,9 @@ class MonitoredExecutor(object):
         return ret
 
     def __exit__(self, err_type, err_value, trace):
+        """
+        clean up things and report hook result when exit train loop
+        """
         if (err_type is None) or isinstance(err_value, (
                 F.core.EOFException, StopException, KeyboardInterrupt)):
             try:
@@ -344,7 +391,10 @@ class MonitoredExecutor(object):
             log.exception('error occur during loop %s: %s' %
                           (err_type, err_value))
 
-    def merge_result(self, ls):
+    def _merge_result(self, ls):
+        """
+        merge results from multi gpu cards
+        """
         dev_count = len(self._program.train_program._places) if isinstance(
             self._program.train_program, F.compiler.CompiledProgram) else 1
         if dev_count == 1:

propeller/paddle/train/trainer.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""common ML train and eval procedure"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -28,7 +29,8 @@ from time import time
 import paddle.fluid as F
 import paddle.fluid.layers as L
 
-from propeller.types import RunMode, StopException, SummaryRecord, StopException, ModelSpec, InferenceSpec, ProgramPair, RunConfig
+from propeller.types import RunMode, StopException, SummaryRecord, StopException
+from propeller.types import ModelSpec, InferenceSpec, ProgramPair, RunConfig
 from propeller.paddle import summary, collection
 from propeller.paddle.data.functional import Dataset
 from propeller.paddle.train import distribution
@@ -43,7 +45,7 @@ log = logging.getLogger(__name__)
 __all__ = ['train_and_eval', 'Learner']
 
 
-def get_summary_writer(path):
+def _get_summary_writer(path):
     summary_writer = None
     try:
         from tensorboardX import SummaryWriter
@@ -54,7 +56,12 @@ def get_summary_writer(path):
     return summary_writer
 
 
-def log_eval_result(name, eval_result, swriter, state):
+def _get_one_place():
+    return F.cuda_places()[0] if F.core.is_compiled_with_cuda(
+    ) else F.cpu_places()[0]
+
+
+def _log_eval_result(name, eval_result, swriter, state):
     log.debug(eval_result)
     printable = []
     for n, val in six.iteritems(eval_result):
@@ -71,7 +78,7 @@ def log_eval_result(name, eval_result, swriter, state):
         log.info('******************************')
 
 
-def build_net(model_fn, features, mode, params, run_config):
+def _build_net(model_fn, features, mode, params, run_config):
     model_spec = model_fn(
         features=features, mode=mode, params=params, run_config=run_config)
 
@@ -97,12 +104,14 @@ def build_net(model_fn, features, mode, params, run_config):
 
 
 class Learner(object):
+    """A Learner can train / eval / predict on a Dataset"""
+
     def __init__(self,
                  model_class_or_model_fn,
                  run_config,
                  params=None,
                  warm_start_setting=None):
-        '''
+        """
         model_class_or_model_fn(callable|propeller.train.Model): `model_class_or_model_fn` be specified in 2 ways:
             1. subclass of propeller.train.Model which implements:
                 1. \_\_init\_\_       (hyper_param, mode, run_config)
@@ -121,58 +130,23 @@ class Learner(object):
         params: any python object, will pass to your `model_fn` or `propeller.train.Model`
         run_config (propeller.RunConfig): run_config.max_steps should not be None.
         warm_start_setting (propeller.WarmStartSetting): Optional. warm start variable will overwrite model variable.
-        '''
+        """
         if run_config.model_dir is None:
             raise ValueError('model_dir should specified in run_config')
 
         if issubclass(model_class_or_model_fn, Model):
-
-            def model_fn(features, mode, params, run_config):
-                if mode != RunMode.PREDICT:
-                    fea, label = features[:-1], features[-1]
-                else:
-                    fea = features
-
-                model = model_class_or_model_fn(
-                    params, mode, run_config=run_config)
-                pred = model.forward(fea)
-                if isinstance(pred, F.framework.Variable):
-                    prediction = [pred]
-                else:
-                    prediction = pred
-                if mode == RunMode.TRAIN:
-                    loss = model.loss(pred, label)
-                    model.backward(loss)
-                    return ModelSpec(
-                        loss=loss, predictions=prediction, mode=mode)
-                elif mode == RunMode.EVAL:
-                    loss = model.loss(pred, label)
-                    me = model.metrics(pred, label)
-
-                    inf_spec = InferenceSpec(inputs=fea, outputs=prediction)
-                    if 'loss' not in me:
-                        me['loss'] = metrics.Mean(loss)
-                    return ModelSpec(
-                        loss=loss,
-                        predictions=prediction,
-                        metrics=me,
-                        mode=mode,
-                        inference_spec=inf_spec)
-                elif mode == RunMode.PREDICT:
-                    return ModelSpec(predictions=prediction, mode=mode)
-                else:
-                    raise RuntimeError('unknown run mode %s' % mode)
+            _model_fn = _build_model_fn(model_class_or_model_fn)
         elif inspect.isfunction(model_class_or_model_fn):
-            model_fn = model_class_or_model_fn
+            _model_fn = model_class_or_model_fn
         else:
             raise ValueError('unknown model %s' % model_class_or_model_fn)
 
-        self.model_fn = model_fn
+        self.model_fn = _model_fn
         self.params = params
         self.run_config = run_config
         self.warm_start_setting = warm_start_setting
 
-    def build_for_train(self, train_dataset):
+    def _build_for_train(self, train_dataset):
         train_dataset.name = 'train'
         train_program = F.Program()
         startup_prog = F.Program()
@@ -181,8 +155,8 @@ class Learner(object):
                 with collection.Collections() as collections:
                     log.info('Building Train Graph...')
                     fea = train_dataset.features()
-                    model_spec = build_net(self.model_fn, fea, RunMode.TRAIN,
-                                           self.params, self.run_config)
+                    model_spec = _build_net(self.model_fn, fea, RunMode.TRAIN,
+                                            self.params, self.run_config)
                     log.info('Building Train Graph: Done')
 
                 scalars = collections.get(collection.Key.SUMMARY_SCALAR)
@@ -208,7 +182,7 @@ class Learner(object):
             train_program=train_program,
             startup_program=startup_prog), model_spec, summary_record
 
-    def build_for_eval(self, ds):
+    def _build_for_eval(self, ds):
         ds.name = 'eval'
         program = F.Program()
         startup_prog = F.Program()
@@ -217,8 +191,8 @@ class Learner(object):
             with F.unique_name.guard():
                 log.info('Building Eval Graph')
                 fea = ds.features()
-                model_spec = build_net(self.model_fn, fea, RunMode.EVAL,
-                                       self.params, self.run_config)
+                model_spec = _build_net(self.model_fn, fea, RunMode.EVAL,
+                                        self.params, self.run_config)
                 log.info('Done')
         program = program.clone(for_test=True)
         log.info(
@@ -227,7 +201,7 @@ class Learner(object):
         return ProgramPair(
             train_program=program, startup_program=startup_prog), model_spec
 
-    def build_for_predict(self, ds):
+    def _build_for_predict(self, ds):
         ds.name = 'predict'
         program = F.Program()
         startup_prog = F.Program()
@@ -236,8 +210,8 @@ class Learner(object):
             with F.unique_name.guard():
                 log.info('Building Predict Graph')
                 fea = ds.features()
-                model_spec = build_net(self.model_fn, fea, RunMode.PREDICT,
-                                       self.params, self.run_config)
+                model_spec = _build_net(self.model_fn, fea, RunMode.PREDICT,
+                                        self.params, self.run_config)
                 log.info('Done')
 
         program = program.clone(for_test=True)
@@ -249,11 +223,12 @@ class Learner(object):
             train_program=program, startup_program=startup_prog), model_spec
 
     def train(self, train_ds, train_hooks=[]):
+        """train on a `Dataset`"""
         if not isinstance(train_ds, Dataset):
             raise ValueError('expect dataset to be instance of Dataset, got %s'
                              % repr(train_ds))
 
-        train_program, model_spec, summary_record = self.build_for_train(
+        train_program, model_spec, summary_record = self._build_for_train(
             train_ds)
         train_run_hooks = [
             hooks.StopAtStepHook(self.run_config.max_steps,
@@ -261,13 +236,16 @@ class Learner(object):
             hooks.LoggingHook(
                 model_spec.loss,
                 summary_record=summary_record,
-                summary_writer=get_summary_writer(
+                summary_writer=_get_summary_writer(
                     os.path.join(self.run_config.model_dir, 'train_history')),
                 per_step=self.run_config.log_steps,
                 skip_step=self.run_config.skip_steps),
         ]
+        if model_spec.train_hooks is not None:
+            train_run_hooks.extend(model_spec.train_hooks)
         train_run_hooks.extend(train_hooks)
-        train_executor = F.Executor(F.cuda_places()[0])
+
+        train_executor = F.Executor(_get_one_place())
 
         mon_exe = MonitoredExecutor(
             train_executor,
@@ -297,24 +275,29 @@ class Learner(object):
         return mon_exe.result
 
     def evaluate(self, eval_dataset, eval_hooks=[]):
+        """eval on a `Dataset`"""
         if not isinstance(eval_dataset, Dataset):
             raise ValueError('expect dataset to be instance of Dataset, got %s'
                              % repr(eval_dataset))
-        program, model_spec = self.build_for_eval(eval_dataset)
-        single_card_place = F.cuda_places()[0]
+        program, model_spec = self._build_for_eval(eval_dataset)
+        single_card_place = _get_one_place()
         eval_executor = F.Executor(single_card_place)
 
-        eval_hooks = [
+        eval_run_hooks = [
             hooks.StopAtStepHook(self.run_config.eval_max_steps,
                                  self.run_config.eval_max_steps),
             hooks.EvalHook(model_spec.metrics, )
         ]
 
+        if model_spec.eval_hooks is not None:
+            eval_run_hooks.extend(model_spec.eval_hooks)
+        eval_run_hooks.extend(eval_hooks)
+
         mon_exe = MonitoredExecutor(
             eval_executor,
             program,
             run_config=self.run_config,
-            run_hooks=eval_hooks)
+            run_hooks=eval_run_hooks)
         mon_exe.init_or_restore_variables()
 
         try:
@@ -326,32 +309,43 @@ class Learner(object):
 
         _, eval_result = mon_exe.result
 
-        summary_writer = get_summary_writer(
+        summary_writer = _get_summary_writer(
             os.path.join(self.run_config.model_dir, 'eval_history'))
-        log_eval_result('eval', eval_result, summary_writer, mon_exe.state)
+        _log_eval_result('eval', eval_result, summary_writer, mon_exe.state)
 
         return mon_exe.result
 
-    def predict(self, predict_dataset, ckpt=None, steps=-1, split_batch=True):
-        '''
+    def predict(self,
+                predict_dataset,
+                ckpt=-1,
+                ckpt_path=None,
+                steps=-1,
+                split_batch=True):
+        """
         Perform predictoin
         will call `model_fn` and initiate user-specifed model in `propeller.RunMode.PREDICT` mode 
 
         Args:
             infer_dataset (propeller.data.Dataset): should not `shuffle` or `repeat`
-            steps (int): steps to predict, if -1 is specifed, will stop when `StopException` is raised in `infer_dataset`
+            steps (int): steps to predict, if None is specifed, 
+                will stop when `StopException` is raised in `infer_dataset`
+            ckpt_path (None|str): Path of a specific checkpoint to predict. 
+                If None, the latest checkpoint in model_dir is used. 
+                If there are no checkpoints in model_dir, 
+                prediction is run with newly initialized Variables instead of ones restored from checkpoint.
+            ckpt (int): deprecated args
             split_batch (bool): if True, prediction of each example in a batch is returned.
 
         Yields:
             Evaluated values of predictions tensors.
 
-        '''
+        """
         if not isinstance(predict_dataset, Dataset):
             raise ValueError('expect dataset to be instance of Dataset, got %s'
                              % repr(predict_dataset))
 
-        program, model_spec = self.build_for_predict(predict_dataset)
-        single_card_place = F.cuda_places()[0]
+        program, model_spec = self._build_for_predict(predict_dataset)
+        single_card_place = _get_one_place()
         executor = F.Executor(single_card_place)
         pred_run_config = RunConfig(
             run_steps=steps if steps == -1 else None,
@@ -360,11 +354,12 @@ class Learner(object):
             executor,
             program,
             run_config=pred_run_config, )
-        mon_exe.init_or_restore_variables()
+        mon_exe.init_or_restore_variables(ckpt
+                                          if ckpt_path is None else ckpt_path)
         try:
             with mon_exe:
                 log.info('Runining predict from dir: %s' % repr(mon_exe.state))
-                single_card_place = F.cuda_places()[0]
+                single_card_place = _get_one_place()
                 for data in predict_dataset.start(places=[single_card_place]):
                     res = mon_exe.run(fetch_list=model_spec.predictions,
                                       feed=data)
@@ -379,7 +374,7 @@ class Learner(object):
             pass
 
 
-def train_and_eval(_shit=None,
+def train_and_eval(_placeholder=None,
                    model_class_or_model_fn=None,
                    params=None,
                    run_config=None,
@@ -389,36 +384,27 @@ def train_and_eval(_shit=None,
                    train_hooks=[],
                    eval_hooks=[],
                    exporters=[]):
-    '''
+    """
     Perform train and evaluate procesure. 
     will call `model_fn` and initiate user-specifed model in `propeller.RunMode.PREDICT` mode 
 
     Args:
         model_class_or_model_fn(callable|propeller.train.Model): `model_class_or_model_fn` be specified in 2 ways:
-            1. subclass of propeller.train.Model which implements:
-                1. \_\_init\_\_       (hyper_param, mode, run_config)
-                2. forward            (features) => (prediction)
-                3. backword           (loss) => None
-                4. loss               (predictoin) => (loss)
-                5. metrics (optional) (prediction) => (dict of propeller.Metrics)
-                
-            2. a model_fn takes following args:
-                1. features
-                2. param
-                3. mode
-                4. run_config(optional)
+            1. subclass of propeller.train.Model
+            2. a model_fn takes following args: 1. features; 2. param; 3. mode; 4. run_config(optional)
                and returns a `propeller.ModelSpec`
 
         params: any python object, will pass to your `model_fn` or `propeller.train.Model`
         run_config (propeller.RunConfig): run_config.max_steps should not be None.
         train_dataset (propeller.paddle.data.Dataset): training will stop if global_step > run_config.max_steps.
-        eval_dataset (propeller.paddle.data.Dataset|dict): Optional, if Dict of propeller.data.Dataset were specified, will perform evluatation on every evaluation sets and report results.
+        eval_dataset (propeller.paddle.data.Dataset|dict): Optional, if Dict of propeller.data.Dataset were specified, 
+            will perform evluatation on every evaluation sets and report results.
         warm_start_setting (propeller.WarmStartSetting): Optional. warm start variable will overwrite model variable.
         train_hooks (list of propeller.paddle.train.RunHook): Optional.
         eval_hooks (list of propeller.paddle.train.RunHook): Optional.
         exporters (list of propeller.paddle.train.Exporter): Optional.
-    '''
-    if _shit is not None:
+    """
+    if _placeholder is not None:
         raise ValueError('specify keyword args to this function')
     if model_class_or_model_fn is None or params is None or run_config is None or train_dataset is None:
         raise ValueError(
@@ -454,13 +440,13 @@ def train_and_eval(_shit=None,
         params,
         warm_start_setting=warm_start_setting)
 
-    class EvalHookOnTrainLoop(hooks.RunHook):
+    class _EvalHookOnTrainLoop(hooks.RunHook):
         def __init__(self):
-            self.program, self.model_spec = est.build_for_eval(
+            self.program, self.model_spec = est._build_for_eval(
                 list(eval_dataset.values())[
                     0])  #eval_datasets must have same output shapes
             self.summary_writers = {
-                ds_name: get_summary_writer(
+                ds_name: _get_summary_writer(
                     os.path.join(
                         os.path.join(run_config.model_dir, 'eval_history'),
                         ds_name))
@@ -468,6 +454,7 @@ def train_and_eval(_shit=None,
             }
 
         def after_run(self, _, state):
+            """doc"""
             if state.step > run_config.skip_steps and state.gstep % run_config.eval_steps == 0:
                 eval_results = {}
                 for name, ds in six.iteritems(eval_dataset):
@@ -478,7 +465,7 @@ def train_and_eval(_shit=None,
                             self.model_spec.metrics,
                             summary_writer=self.summary_writers[name], )
                     ]
-                    single_card_place = F.cuda_places()[0]
+                    single_card_place = _get_one_place()
                     eval_executor = F.Executor(single_card_place)
                     mon_exe = MonitoredExecutor(
                         eval_executor,
@@ -495,8 +482,8 @@ def train_and_eval(_shit=None,
                     eval_res = hook_results[
                         1]  # hook_results:  [StopAtStepHook, EvalHook, ...]
                     eval_results[name] = eval_res
-                    log_eval_result(name, eval_res, self.summary_writers[name],
-                                    state)
+                    _log_eval_result(name, eval_res,
+                                     self.summary_writers[name], state)
                 for exporter in exporters:
                     exporter.export(eval_executor, self.program,
                                     self.model_spec, eval_results, state)
@@ -505,6 +492,46 @@ def train_and_eval(_shit=None,
             return eval_results
 
     if distribution.status.is_master:
-        train_hooks.append(EvalHookOnTrainLoop())
+        train_hooks.append(_EvalHookOnTrainLoop())
     res = est.train(train_dataset, train_hooks=train_hooks)
     return res
+
+
+def _build_model_fn(model_class):
+    def _model_fn(features, mode, params, run_config):
+        if mode != RunMode.PREDICT:
+            fea, label = features[:-1], features[-1]
+        else:
+            fea = features
+
+        model = model_class(params, mode, run_config=run_config)
+        pred = model.forward(fea)
+        if isinstance(pred, F.framework.Variable):
+            prediction = [pred]
+        else:
+            prediction = pred
+        if mode == RunMode.TRAIN:
+            loss = model.loss(pred, label)
+            model.backward(loss)
+            return ModelSpec(loss=loss, predictions=prediction, mode=mode)
+        elif mode == RunMode.EVAL:
+            loss = model.loss(pred, label)
+            me = model.metrics(pred, label)
+
+            inf_spec = InferenceSpec(inputs=fea, outputs=prediction)
+            if 'loss' not in me:
+                me['loss'] = metrics.Mean(loss)
+            return ModelSpec(
+                loss=loss,
+                predictions=prediction,
+                metrics=me,
+                mode=mode,
+                inference_spec=inf_spec)
+        elif mode == RunMode.PREDICT:
+            inf_spec = InferenceSpec(inputs=fea, outputs=prediction)
+            return ModelSpec(
+                predictions=prediction, mode=mode, inference_spec=inf_spec)
+        else:
+            raise RuntimeError('unknown run mode %s' % mode)
+
+    return _model_fn

propeller/service/__init__.py

@@ -11,3 +11,4 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""server"""

propeller/service/server.py

@@ -11,6 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+""" 
+Never Never Never import paddle.fluid in main process, or any module would import fluid.
+"""
 
 from __future__ import division
 from __future__ import absolute_import
@@ -24,27 +27,27 @@ from time import sleep, time
 import multiprocessing
 
 import zmq
-""" Never Never Never import paddle.fluid in main process, or any module would import fluid.
-"""
 
 log = logging.getLogger(__name__)
 
 
-def profile(msg):
-    def decfn(fn):
-        def retfn(*args, **kwargs):
+def _profile(msg):
+    def _decfn(fn):
+        def _retfn(*args, **kwargs):
             start = time()
             ret = fn(*args, **kwargs)
             end = time()
             log.debug('%s timecost: %.5f' % (msg, end - start))
             return ret
 
-        return retfn
+        return _retfn
 
-    return decfn
+    return _decfn
 
 
 class Predictor(object):
+    """paddle predictor wrapper"""
+
     def __init__(self, model_dir, device_idx=0):
         import paddle.fluid as F
         log.debug('create predictor on card %d' % device_idx)
@@ -52,7 +55,7 @@ class Predictor(object):
         config.enable_use_gpu(5000, device_idx)
         self._predictor = F.core.create_paddle_predictor(config)
 
-    @profile('paddle')
+    @_profile('paddle')
     def __call__(self, args):
         for i, a in enumerate(args):
             a.name = 'placeholder_%d' % i
@@ -61,6 +64,7 @@ class Predictor(object):
 
 
 def run_worker(model_dir, device_idx, endpoint="ipc://worker.ipc"):
+    """worker process entrence"""
     try:
         log.debug("run_worker %s" % device_idx)
         os.environ["CUDA_VISIBLE_DEVICES"] = os.getenv(
@@ -97,6 +101,8 @@ def run_worker(model_dir, device_idx, endpoint="ipc://worker.ipc"):
 
 
 class InferencePredictor(object):
+    """control Predictor for multi gpu card"""
+
     def __init__(self, backend_addr, model_dir, n_devices=1):
         self.backend_addr = backend_addr
         self.model_dir = model_dir
@@ -104,6 +110,7 @@ class InferencePredictor(object):
         self.children = []
 
     def start(self):
+        """doc"""
         for device_idx in range(self.n_devices):
             p = multiprocessing.Process(
                 target=run_worker,
@@ -113,21 +120,27 @@ class InferencePredictor(object):
         return self
 
     def join(self):
+        """doc"""
         for p in self.children:
             p.join()
 
     def term(self):
+        """doc"""
         for p in self.children:
             log.debug("terminating children %s" % repr(p))
             p.terminate()
 
 
 class InferenceProxy(object):
+    """zmq proxy"""
+
     def __init__(self):
+        """doc"""
         self.backend = None
         self.frontend = None
 
     def listen(self, frontend_addr, backend_addr):
+        """doc"""
         log.info("InferenceProxy starting...")
         try:
             context = zmq.Context(1)
@@ -152,11 +165,15 @@ class InferenceProxy(object):
 
 
 class InferenceServer(object):
+    """start InferencePredictor and InferenceProxy"""
+
     def __init__(self, model_dir, n_devices):
+        """doc"""
         self.model_dir = model_dir
         self.n_devices = n_devices
 
     def listen(self, port):
+        """doc"""
         frontend_addr = "tcp://*:%s" % port
         backend_addr = "ipc://backend.ipc"
         predictor = InferencePredictor(backend_addr, self.model_dir,

propeller/service/utils.py

@@ -13,6 +13,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""utils for server"""
 
 from __future__ import division
 from __future__ import absolute_import
@@ -26,6 +27,7 @@ from propeller.service import interface_pb2
 
 
 def slot_to_numpy(slot):
+    """doc"""
     if slot.type == interface_pb2.Slot.FP32:
         dtype = np.float32
         type_str = 'f'
@@ -45,6 +47,7 @@ def slot_to_numpy(slot):
 
 
 def numpy_to_slot(arr):
+    """doc"""
     if arr.dtype == np.float32:
         dtype = interface_pb2.Slot.FP32
     elif arr.dtype == np.int32:
@@ -59,28 +62,27 @@ def numpy_to_slot(arr):
 
 
 def slot_to_paddlearray(slot):
+    """doc"""
     import paddle.fluid.core as core
     if slot.type == interface_pb2.Slot.FP32:
+        dtype = np.float32
         type_str = 'f'
-        dtype = core.PaddleDType.FLOAT32
     elif slot.type == interface_pb2.Slot.INT32:
+        dtype = np.int32
         type_str = 'i'
-        dtype = core.PaddleDType.INT32
     elif slot.type == interface_pb2.Slot.INT64:
+        dtype = np.int64
         type_str = 'q'
-        dtype = core.PaddleDType.INT64
     else:
         raise RuntimeError('know type %s' % slot.type)
-    ret = core.PaddleTensor()
-    ret.shape = slot.dims
-    ret.dtype = dtype
     num = len(slot.data) // struct.calcsize(type_str)
     arr = struct.unpack('%d%s' % (num, type_str), slot.data)
-    ret.data = core.PaddleBuf(arr)
+    ret = core.PaddleTensor(data=np.array(arr, dtype=dtype).reshape(slot.dims))
     return ret
 
 
 def paddlearray_to_slot(arr):
+    """doc"""
     import paddle.fluid.core as core
     if arr.dtype == core.PaddleDType.FLOAT32:
         dtype = interface_pb2.Slot.FP32
@@ -102,12 +104,14 @@ def paddlearray_to_slot(arr):
 
 
 def nparray_list_serialize(arr_list):
+    """doc"""
     slot_list = [numpy_to_slot(arr) for arr in arr_list]
     slots = interface_pb2.Slots(slots=slot_list)
     return slots.SerializeToString()
 
 
 def nparray_list_deserialize(string):
+    """doc"""
     slots = interface_pb2.Slots()
     slots.ParseFromString(string)
     return [slot_to_numpy(slot) for slot in slots.slots]

propeller/tools/ckpt_inspector.py

@@ -72,6 +72,10 @@ def parse(filename):
         elif proto.data_type == framework_pb2.VarType.INT8:
             arr = np.array(
                 gen_arr(f.read(), 'B'), dtype=np.int8).reshape(proto.dims)
+        elif proto.data_type == framework_pb2.VarType.FP16:
+            arr = np.array(
+                gen_arr(f.read(), 'H'),
+                dtype=np.uint16).view(np.float16).reshape(proto.dims)
         else:
             raise RuntimeError('Unknown dtype %s' % proto.data_type)
 

propeller/train/__init__.py

@@ -11,3 +11,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+doc
+"""

propeller/train/model.py

@@ -11,6 +11,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""
+Model template
+"""
+
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -26,7 +30,11 @@ import numpy as np
 
 
 @six.add_metaclass(abc.ABCMeta)
-class Model():
+class Model(object):
+    """
+    template
+    """
+
     def __init__(self, config, mode):
         """
         Args:
@@ -39,9 +47,9 @@ class Model():
     def forward(self, features):
         """
         Args:
-            features (list of Tensor): depends on your Dataset.output_shapes
+            features (list of Tensor): inputs features that depends on your Dataset.output_shapes
         Returns:
-            return (Tensor): 
+            return (Tensor): prediction
         """
         pass
 
@@ -53,8 +61,6 @@ class Model():
             label (Tensor): depends on your Dataset.output_shapes
         Returns:
             return (paddle scalar): loss
-        
-
         """
         pass
 

propeller/types.py

@@ -11,6 +11,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""Basic types"""
+
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -21,12 +23,15 @@ from collections import namedtuple
 
 
 class RunMode(object):
+    """model_fn will be called in 3 modes"""
     TRAIN = 1
     PREDICT = 2
     EVAL = 3
 
 
 class HParams(object):
+    """Hyper paramerter"""
+
     def __init__(self, **kwargs):
         for k, v in kwargs.items():
             self.__dict__[k] = v
@@ -45,30 +50,36 @@ class HParams(object):
     def __setitem__(self, key, val):
         self.__dict__[key] = val
 
-    @staticmethod
-    def from_json(self, json_str):
+    @classmethod
+    def from_json(cls, json_str):
+        """doc"""
         d = json.loads(json_str)
         if type(d) != dict:
             raise ValueError('json object must be dict.')
         return HParams.from_dict(d)
 
     def get(self, key, default=None):
+        """doc"""
         return self.__dict__.get(key, default)
 
-    @staticmethod
-    def from_dict(self, d):
+    @classmethod
+    def from_dict(cls, d):
+        """doc"""
         if type(d) != dict:
             raise ValueError('input must be dict.')
         hp = HParams(**d)
         return hp
 
     def to_json(self):
+        """doc"""
         return json.dumps(self.__dict__)
 
     def to_dict(self):
+        """doc"""
         return self.__dict__
 
     def join(self, other):
+        """doc"""
         if not isinstance(other, HParams):
             raise ValueError('input must be HParams instance.')
         self.__dict__.update(**other.__dict__)
@@ -95,9 +106,12 @@ ModelSpec = namedtuple('ModelSpec', [
     'metrics',
     'mode',
     'inference_spec',
+    'train_hooks',
+    'eval_hooks',
 ])
 ModelSpec.__new__.__defaults__ = (None, ) * len(ModelSpec._fields)
 
 
 class StopException(Exception):
+    """doc"""
     pass

propeller/util.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+"""global utils"""
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import unicode_literals
@@ -31,6 +32,7 @@ log = logging.getLogger(__name__)
 
 
 def ArgumentParser(name):
+    """predefined argparser"""
     parser = argparse.ArgumentParser('propeller model')
     parser.add_argument('--run_config', type=str, default='')
     parser.add_argument(
@@ -59,6 +61,7 @@ def _get_dict_from_environ_or_json_or_file(args, env_name):
 
 
 def parse_file(filename):
+    """useless api"""
     d = _get_dict_from_environ_or_json_or_file(filename, None)
     if d is None:
         raise ValueError('file(%s) not found' % filename)
@@ -66,6 +69,7 @@ def parse_file(filename):
 
 
 def parse_runconfig(args=None):
+    """get run_config from env or file"""
     d = _get_dict_from_environ_or_json_or_file(args.run_config,
                                                'PROPELLER_RUNCONFIG')
     if d is None:
@@ -74,6 +78,7 @@ def parse_runconfig(args=None):
 
 
 def parse_hparam(args=None):
+    """get hparam from env or file"""
     if args is not None:
         hparam_strs = reduce(list.__add__, args.hparam)
     else:
@@ -91,6 +96,7 @@ def parse_hparam(args=None):
 
 
 def flatten(s):
+    """doc"""
     assert is_struture(s)
     schema = [len(ss) for ss in s]
     flt = list(itertools.chain(*s))
@@ -98,6 +104,7 @@ def flatten(s):
 
 
 def unflatten(structure, schema):
+    """doc"""
     start = 0
     res = []
     for _range in schema:
@@ -107,10 +114,12 @@ def unflatten(structure, schema):
 
 
 def is_struture(s):
+    """doc"""
     return isinstance(s, list) or isinstance(s, tuple)
 
 
 def map_structure(func, s):
+    """same sa tf.map_structure"""
     if isinstance(s, list) or isinstance(s, tuple):
         return [map_structure(func, ss) for ss in s]
     elif isinstance(s, dict):

requirements.txt

@@ -5,3 +5,5 @@ scikit-learn==0.20.3
 scipy==1.2.1
 six==1.11.0
 sklearn==0.0
+sentencepiece==0.1.8
+paddlepaddle-gpu==1.6.3.post107