Merge pull request #2 from xixiaoyao/master

add pretrain_model dir

README.md

@@ -32,10 +32,12 @@ git clone https://github.com/PaddlePaddle/PALM.git
 
 框架给出了三个添加完成的任务示例：*Machine Reading Comprehension*、*Mask Language Model*和*Question Answer Matching*。其中在`mtl_config.yaml`中将*Machine Reading Comprehension*设置为了主任务，其他为辅助任务，用户可通过如下命令启动多任务学习
 
-```
+```shell
 bash run.sh
 ```
 
+*提示：首次运行时，脚本会自动下载预训练的bert和ernie模型，请耐心等待*
+
 ### 多任务学习配置
 
 在`mtl_config.yaml`中完成对多任务训练和推理的主配置，配置包含如下

backbone/README.md

+
+该目录用来存放模型backbone，用户可通过实现backbone的接口完成自定义。
+

backbone/ernie_model.py

@@ -22,10 +22,9 @@ from __future__ import absolute_import
 import paddle.fluid as fluid
 
 import backbone.utils.transformer4ernie as transformer
-from backbone.interface import backbone
 
 
-class Model(backbone):
+class Model(object):
     def __init__(self,
                  config,
                  is_training=False,

download_pretrain.sh

+#!/bin/bash
+
+set -e
+
+if [[ $# != 1 ]]; then
+    echo "Usage: bash download_pretrain.sh <bert|ernie>"
+    exit 1
+fi
+
+if [[ $1 == 'bert' ]]; then
+    name="bert"
+    link="https://bert-models.bj.bcebos.com/uncased_L-24_H-1024_A-16.tar.gz"
+    packname="uncased_L-24_H-1024_A-16.tar.gz"
+    dirname="uncased_L-24_H-1024_A-16"
+elif [[ $1 == 'ernie' ]]; then
+    name="ernie"
+    link="https://ernie.bj.bcebos.com/ERNIE_Large_en_stable-2.0.0.tar.gz"
+    packname="ERNIE_Large_en_stable-2.0.0.tar.gz"
+else
+    echo "$1 is currently not supported."
+    exit 1
+fi
+
+cd pretrain_model
+mkdir $name
+cd $name
+echo "downloading ${name}..."
+wget --no-check-certificate $link
+echo "decompressing..."
+tar -zxf $packname
+rm -rf $packname
+if [[ $dirname != "" ]]; then
+    mv $dirname/* .
+    rm -rf $dirname
+fi
+
+cd ../..
+
+

mtl_config.yaml

@@ -4,6 +4,8 @@ auxiliary_task: "mask_language_model answer_matching"
 do_train: True
 do_predict: True
 
+use_cuda: False
+
 checkpoint_path: "output_model/firstrun"
 
 backbone_model: "bert_model"
@@ -22,10 +24,9 @@ lr_scheduler: "linear_warmup_decay"
 skip_steps: 10
 save_steps: 10000
 epoch: 2
-use_cuda: True
 warmup_proportion: 0.1
 weight_decay: 0.1
-do_lower_case: False
+do_lower_case: True
 max_seq_len: 512
 use_ema: True
 ema_decay: 0.9999

optimizer/README.md

+
+该目录存放优化器，用户可通过实现相关接口完成自定义

paradigm/README.md

+
+该目录用来存放任务范式，用户可通过实现paradigm的接口完成自定义。
+

pretrain_model/README.md

+
+该目录用于存放预训练及其配置文件，用户可通过运行`download_pretrain.sh`下载内置的预训练模型。
+

reader/README.md

+
+该目录存放数据集载入与处理模块reader，用户可通过实现相关接口完成自定义
+

run.sh

 #!/bin/bash
 
+# for gpu memory optimization
 export FLAGS_sync_nccl_allreduce=0
 export FLAGS_eager_delete_tensor_gb=1
 
 export CUDA_VISIBLE_DEVICES=0
 
-if  [ ! "$CUDA_VISIBLE_DEVICES" ]
-then
-    export CPU_NUM=1
-    use_cuda=false
-else
-    use_cuda=true
+if [[ ! -d pretrain_model/bert ]]; then
+    bash download_pretrain.sh bert
+fi
+
+if [[ ! -d pretrain_model/ernie ]]; then
+    bash download_pretrain.sh ernie
 fi
 
 python -u mtl_run.py