PaddlePALM
PaddlePALM (PArallel Learning from Multi-tasks) is a flexible, general and easy-to-use NLP large-scale pretraining and multi-task learning friendly framework. PALM is a high level framework aiming at fastly develop high-performance NLP models.
With PaddlePALM, it is easy to achieve effecient exploration of robust learning of reading comprehension models with multiple auxilary tasks, and the produced model, D-Net, achieve the 1st place in EMNLP2019 MRQA track.
MRQA2019 Leaderboard
Beyond the research scope, PaddlePALM has been applied on Baidu Search Engine to seek for more accurate user query understanding and answer mining, which implies the high reliability and performance of PaddlePALM.
Features:
- Easy-to-use: with PALM, 8 steps to achieve a typical NLP task. Moreover, the model backbone, dataset reader and task output layers have been decoupled, which allows the replacement of any component to other candidates with quite minor changes of your code.
- Multi-task Learning friendly: 6 steps to achieve multi-task learning for prepared tasks.
- Large Scale and Pre-training freiendly: automatically utilize multi-gpus (if exists) to accelerate training and inference. Minor codes is required for distributed training on clusters.
- Popular NLP Backbones and Pre-trained models: multiple state-of-the-art general purpose model architectures and pretrained models (e.g., BERT,ERNIE,RoBERTa,...) are built-in.
- Easy to Customize: support customized development of any component (e.g, backbone, task head, reader and optimizer) with reusement of pre-defined ones, which gives developers high flexibility and effeciency to adapt for diverse NLP scenes.
You can easily re-produce following competitive results with minor codes, which covers most of NLP tasks such as classification, matching, sequence labeling, reading comprehension, dialogue understanding and so on. More details can be found in examples
.
Dataset
|
chnsenticorp | Quora Question Pairs matching | MSRA-NER (SIGHAN2006) |
CMRC2018 | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Metric
|
precision
|
recall
|
f1-score
|
precision
|
recall
|
f1-score
|
precision
|
recall
|
f1-score
|
em
|
f1-score
|
test
|
test
|
test
|
dev
|
||||||||
ERNIE Base | 95.7 | 95.0 | 95.7 | 85.8 | 82.4 | 81.5 | 94.9 | 94.5 | 94.7 | 96.3 | 84.0 |
Package Overview
module | illustration |
---|---|
paddlepalm | an open source NLP pretraining and multitask learning framework, built on paddlepaddle. |
paddlepalm.reader | a collection of elastic task-specific dataset readers. |
paddlepalm.backbone | a collection of classic NLP representation models, e.g., BERT, ERNIE, RoBERTa. |
paddlepalm.head | a collection of task-specific output layers. |
paddlepalm.lr_sched | a collection of learning rate schedualers. |
paddlepalm.optimizer | a collection of optimizers. |
paddlepalm.downloader | a download module for pretrained models with configure and vocab files. |
paddlepalm.Trainer | the core unit to start a single task training/predicting session. A trainer is to build computation graph, manage training and evaluation process, achieve model/checkpoint saving and pretrain_model/checkpoint loading. |
paddlepalm.MultiHeadTrainer | the core unit to start a multi-task training/predicting session. A MultiHeadTrainer is built based on several Trainers. Beyond the inheritance of Trainer, it additionally achieves model backbone reuse across tasks, trainer sampling for multi-task learning, and multi-head inference for effective evaluation and prediction. |
Installation
PaddlePALM support both python2 and python3, linux and windows, CPU and GPU. The preferred way to install PaddlePALM is via pip
. Just run following commands in your shell.
pip install paddlepalm
Installing via source
git clone https://github.com/PaddlePaddle/PALM.git
cd PALM && python setup.py install
Library Dependencies
- Python >= 2.7
- cuda >= 9.0
- cudnn >= 7.0
- PaddlePaddle >= 1.7.0 (请参考安装指南进行安装)
Downloading pretrain models
We incorporate many pretrained models to initialize model backbone parameters. Training big NLP model, e.g., 12-layer transformers, with pretrained models is practically much more effective than that with randomly initialized parameters. To see all the available pretrained models and download, run following code in python interpreter (input command python
in shell):
>>> from paddlepalm import downloader
>>> downloader.ls('pretrain')
Available pretrain items:
=> roberta-cn-base
=> roberta-cn-large
=> bert-cn-base
=> bert-cn-large
=> bert-en-uncased-base
=> bert-en-uncased-large
=> bert-en-cased-base
=> bert-en-cased-large
=> ernie-en-uncased-base
=> ernie-en-uncased-large
...
>>> downloader.download('pretrain', 'bert-en-uncased-base', './pretrain_models')
...
Usage
8 steps to start a typical NLP training task.
- use
paddlepalm.reader
to create a reader for dataset loading and input features generation, then callreader.load_data
method to load your training data. - use
paddlepalm.backbone
to create a model backbone to extract text features (e.g., contextual word embedding, sentence embedding). - register your reader with your backbone through
reader.register_with
method. After this step, your reader is able to yield input features used by backbone. - use
paddlepalm.head
to create a task output head. This head can provide task loss for training and predicting results for model inference. - create a task trainer with
paddlepalm.Trainer
, then build forward graph with backbone and task head (created in step 2 and 4) throughtrainer.build_forward
. - use
paddlepalm.optimizer
(andpaddlepalm.lr_sched
if is necessary) to create a optimizer, then build backward throughtrainer.build_backward
. - fit prepared reader and data (achieved in step 1) to trainer with
trainer.fit_reader
method. - load pretrain model with
trainer.load_pretrain
, or load checkpoint withtrainer.load_ckpt
or nothing to do for training from scratch, then do training withtrainer.train
.
For more implementation details, see following demos:
set saver
To save models/checkpoints and logs during training, just call trainer.set_saver
method. More implementation details see this.
do prediction
To do predict/evaluation after a training stage, just create another three reader, backbone and head instance with phase='predict'
(repeat step 1~4 above). Then do predicting with predict
method in trainer (no need to create another trainer). More implementation details see this.
multi-task learning
To run with multi-task learning mode:
- repeatedly create components (i.e., reader, backbone and head) for each task followed with step 1~5 above.
- create empty trainers (each trainer is corresponded to one task) and pass them to create a
MultiHeadTrainer
. - build multi-task forward graph with
multi_head_trainer.build_forward
method. - use
paddlepalm.optimizer
(andpaddlepalm.lr_sched
if is necessary) to create a optimizer, then build backward throughmulti_head_trainer.build_backward
. - fit all prepared readers and data to multi_head_trainer with
multi_head_trainer.fit_readers
method. - randomly initialize model parameters with
multi_head_trainer.random_init_params
(andmulti_head_trainer.load_pretrain
if needed), then do training withmulti_head_trainer.train
.
The save/load and predict operations of a multi_head_trainer is the same as a trainer.
For more implementation details with multi_head_trainer
, see
- ATIS: joint training of dialogue intent recognition and slot filling
- MRQA: learning reading comprehension auxilarized with mask language model (初次发版先不用加)
License
This tutorial is contributed by PaddlePaddle and licensed under the Apache-2.0 license.
许可证书
此向导由PaddlePaddle贡献,受Apache-2.0 license许可认证。