README.md 12.9 KB
Newer Older
X
Xiaoyao Xi 已提交
1
# PaddlePALM
X
xixiaoyao 已提交
2

X
Xiaoyao Xi 已提交
3 4
English | [简体中文](./README_cn.md)

X
Xiaoyao Xi 已提交
5
PaddlePALM (PArallel Learning from Multi-tasks) is a fast, flexible, extensible and easy-to-use NLP large-scale pretraining and multi-task learning framework. PaddlePALM is a high level framework aiming at **fastly** developing **high-performance** NLP models. 
W
wangxiao1021 已提交
6

X
Xiaoyao Xi 已提交
7
With PaddlePALM, it is easy to achieve effecient exploration of robust learning of NLP models with multiple auxilary tasks. For example, based on PaddlePALM, the produced robust MRC model, [D-Net](https://github.com/PaddlePaddle/models/tree/develop/PaddleNLP/Research/MRQA2019-D-NET), has achieved **the 1st place** in [EMNLP2019 MRQA](https://mrqa.github.io) track.
X
Xiaoyao Xi 已提交
8

X
Xiaoyao Xi 已提交
9
<p align="center">
X
Xiaoyao Xi 已提交
10
	<img src="https://tva1.sinaimg.cn/large/006tNbRwly1gbjkuuwrmlj30hs0hzdh2.jpg" alt="Sample"  width="300" height="333">
X
Xiaoyao Xi 已提交
11
	<p align="center">
X
Xiaoyao Xi 已提交
12
		<em>MRQA2019 Leaderboard</em>
X
Xiaoyao Xi 已提交
13 14 15
	</p>
</p>

X
Xiaoyao Xi 已提交
16 17
Beyond the research scope, PaddlePALM has been applied on **Baidu Search Engine** to seek for more accurate user query understanding and answer mining, which implies the high reliability and performance of PaddlePALM.

X
Xiaoyao Xi 已提交
18
#### Features:
X
Xiaoyao Xi 已提交
19

X
Xiaoyao Xi 已提交
20
- **Easy-to-use:** with PALM, *8 steps* to achieve a typical NLP task. Moreover, all basic components (e.g., the model backbone, dataset reader, task output head, optimizer...) have been decoupled, which allows the replacement of any component to other candidates with quite minor changes of your code. 
X
Xiaoyao Xi 已提交
21 22 23 24 25
- **Multi-task Learning friendly:** *6 steps* to achieve multi-task learning for prepared tasks. 
- **Large Scale and Pre-training freiendly:** automatically utilize multi-gpus (if exists) to accelerate training and inference. Minor codes is required for distributed training on clusters.
- **Popular NLP Backbones and Pre-trained models:** multiple state-of-the-art general purpose model architectures and pretrained models (e.g., BERT,ERNIE,RoBERTa,...) are built-in. 
- **Easy to Customize:** support customized development of any component (e.g, backbone, task head, reader and optimizer) with reusement of pre-defined ones, which gives developers high flexibility and effeciency to adapt for diverse NLP scenes. 

X
Xiaoyao Xi 已提交
26
You can easily re-produce following competitive results with minor codes, which covers most of NLP tasks such as classification, matching, sequence labeling, reading comprehension, dialogue understanding and so on. More details can be found in `examples`.
W
wangxiao1021 已提交
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113

<table>
  <tbody>
    <tr>
      <th><strong>Dataset</strong>
        <br></th>
      <th colspan="3"><center><strong>chnsenticorp</strong></center></th>
      <th colspan="3"><center><strong>Quora Question Pairs matching</strong><center></th>
      <th colspan="3"><strong>MSRA-NER<br>(SIGHAN2006)</strong></th>
      <th colspan="2"><strong>CMRC2018</strong></th>
    </tr>
    <tr>
      <td rowspan="2">
        <p>
          <strong>Metric</strong>
          <br></p>
      </td>
      <td colspan="1">
        <center><strong>precision</strong></center>
        <br></td>
      <td colspan="1">
        <strong>recall</strong>
        <br></td>
      <td colspan="1">
        <strong>f1-score</strong>
        <strong></strong>
        <br></td>
      <td colspan="1">
        <center><strong>precision</strong></center>
        <br></td>
      <td colspan="1">
        <strong>recall</strong>
        <br></td>
      <td colspan="1">
        <strong>f1-score</strong>
        <strong></strong>
        <br></td>
      <td colspan="1">
        <center><strong>precision</strong></center>
        <br></td>
      <td colspan="1">
        <strong>recall</strong>
        <br></td>
      <td colspan="1">
        <strong>f1-score</strong>
        <strong></strong>
        <br></td>
      <td colspan="1">
        <strong>em</strong>
        <br></td>
      <td colspan="1">
        <strong>f1-score</strong>
        <br></td>
    </tr>
    <tr>
      <td colspan="3" width="">
        <strong>test</strong>
        <br></td>
      <td colspan="3" width="">
        <strong>test</strong>
        <br></td>
      <td colspan="3" width="">
        <strong>test</strong>
        <br></td>
      <td colspan="2" width="">
        <strong>dev</strong>
        <br></td>
    </tr>
    <tr>
      <td><strong>ERNIE Base</strong></td>
      <td>95.7</td>
      <td>95.0</td>
      <td>95.7</td>
      <td>85.8</td>
      <td>82.4</td>
      <td>81.5</td>
      <td>94.9</td>
      <td>94.5</td>
      <td>94.7</td>
      <td>96.3</td>
      <td>84.0</td>
    </tr>

  </tbody>
</table>


X
Xiaoyao Xi 已提交
114

X
Xiaoyao Xi 已提交
115
## Overview
W
wangxiao1021 已提交
116

W
wangxiao1021 已提交
117 118 119 120 121 122 123
<p align="center">
	<img src="https://github.com/PaddlePaddle/PALM/blob/master/img/architecture.png" alt="Sample"  width="582" height="289">
	<p align="center">
		<em>Architecture Diagram</em>
	</p>
</p>

X
Xiaoyao Xi 已提交
124 125
PaddlePALM is a well-designed high-level NLP framework. You can efficiently achieve **supervised learning, unsupervised/self-supervised learning, multi-task learning and transfer learning** with minor codes based on PaddlePALM. There are three layers in PaddlePALM architecture, i.e., component layer, trainer layer and high-level trainer layer from bottom to top. 

X
Xiaoyao Xi 已提交
126
In component layer, PaddlePALM supplies 6 **decoupled** components to achieve a NLP task. Each component contains rich `pre-defined` classes and a `Base` class. Pre-defined classes are aiming at typical NLP tasks, and the base class is to help users develop a new Class (based on pre-defined ones or from the base). 
X
Xiaoyao Xi 已提交
127

X
Xiaoyao Xi 已提交
128 129 130 131
The trainer layer is to establish a computation graph with selected components and do training and predicting. The training strategy, model saving and loading, evaluation and predicting procedures are described in this layer. Noted a trainer can only process one task. 

The high-level trainer layer is for complicated learning and inference strategy, e.g., multi-task learning. You can add auxilary tasks to train robust NLP models (improve test set and out-of-domain performance of a model), or jointly training multiple related tasks to gain more performance for each task.

W
wangxiao1021 已提交
132 133 134 135 136 137 138 139 140 141 142 143 144 145 146
| module | illustration | 
| - | - |
| **paddlepalm** | an open source NLP pretraining and multitask learning framework, built on paddlepaddle. |
| **paddlepalm.reader** | a collection of elastic task-specific dataset readers. |
| **paddlepalm.backbone** | a collection of classic NLP representation models, e.g., BERT, ERNIE, RoBERTa. |
| **paddlepalm.head** | a collection of task-specific output layers. |
| **paddlepalm.lr_sched** | a collection of learning rate schedualers. |
| **paddlepalm.optimizer** | a collection of optimizers. |
| **paddlepalm.downloader** | a download module for pretrained models with configure and vocab files. |
| **paddlepalm.Trainer** | the core unit to start a single task training/predicting session. A trainer is to build computation graph, manage training and evaluation process, achieve model/checkpoint saving and pretrain_model/checkpoint loading.|
| **paddlepalm.MultiHeadTrainer** | the core unit to start a multi-task training/predicting session. A MultiHeadTrainer is built based on several Trainers. Beyond the inheritance of Trainer, it additionally achieves model backbone reuse across tasks, trainer sampling for multi-task learning, and multi-head inference for effective evaluation and prediction. |


## Installation

X
Xiaoyao Xi 已提交
147
PaddlePALM support both python2 and python3, linux and windows, CPU and GPU. The preferred way to install PaddlePALM is via `pip`. Just run following commands in your shell.
W
wangxiao1021 已提交
148 149

```bash
X
xixiaoyao 已提交
150
pip install paddlepalm
X
Xiaoyao Xi 已提交
151 152
```

W
wangxiao1021 已提交
153
### Installing via source
X
Xiaoyao Xi 已提交
154 155 156

```shell
git clone https://github.com/PaddlePaddle/PALM.git
X
Xiaoyao Xi 已提交
157
cd PALM && python setup.py install
X
Xiaoyao Xi 已提交
158 159
```

W
wangxiao1021 已提交
160
### Library Dependencies
X
Xiaoyao Xi 已提交
161
- Python >= 2.7
X
Xiaoyao Xi 已提交
162 163
- cuda >= 9.0
- cudnn >= 7.0
X
Xiaoyao Xi 已提交
164
- PaddlePaddle >= 1.7.0 (Please refer to [this](http://www.paddlepaddle.org/#quick-start) to install)
X
Xiaoyao Xi 已提交
165 166


W
wangxiao1021 已提交
167 168
### Downloading pretrain models
We incorporate many pretrained models to initialize model backbone parameters. Training big NLP model, e.g., 12-layer transformers, with pretrained models is practically much more effective than that with randomly initialized parameters. To see all the available pretrained models and download, run following code in python interpreter (input command `python` in shell):
X
Xiaoyao Xi 已提交
169

X
Xiaoyao Xi 已提交
170
```python
W
wangxiao1021 已提交
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186
>>> from paddlepalm import downloader
>>> downloader.ls('pretrain')
Available pretrain items:
  => roberta-cn-base
  => roberta-cn-large
  => bert-cn-base
  => bert-cn-large
  => bert-en-uncased-base
  => bert-en-uncased-large
  => bert-en-cased-base
  => bert-en-cased-large
  => ernie-en-uncased-base
  => ernie-en-uncased-large
  ...

>>> downloader.download('pretrain', 'bert-en-uncased-base', './pretrain_models')
X
Xiaoyao Xi 已提交
187 188 189
...
```

X
Xiaoyao Xi 已提交
190

W
wangxiao1021 已提交
191
## Usage
X
Xiaoyao Xi 已提交
192

X
Xiaoyao Xi 已提交
193 194
#### Quick Start

W
wangxiao1021 已提交
195
8 steps to start a typical NLP training task.
X
Xiaoyao Xi 已提交
196

W
wangxiao1021 已提交
197 198 199 200 201 202 203 204
1. use `paddlepalm.reader` to create a *reader* for dataset loading and input features generation, then call `reader.load_data` method to load your training data.
2. use `paddlepalm.backbone` to create a model *backbone* to extract text features (e.g., contextual word embedding, sentence embedding).
3. register your *reader* with your *backbone* through `reader.register_with` method. After this step, your reader is able to yield input features used by backbone.
4. use `paddlepalm.head` to create a task output *head*. This head can provide task loss for training and predicting results for model inference.
5. create a task *trainer* with `paddlepalm.Trainer`, then build forward graph with backbone and task head (created in step 2 and 4) through `trainer.build_forward`.
6. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `trainer.build_backward`.
7. fit prepared reader and data (achieved in step 1) to trainer with `trainer.fit_reader` method.
8. load pretrain model with `trainer.load_pretrain`, or load checkpoint with `trainer.load_ckpt` or nothing to do for training from scratch, then do training with `trainer.train`.
X
Xiaoyao Xi 已提交
205

X
Xiaoyao Xi 已提交
206
For more implementation details, see following demos: 
X
Xiaoyao Xi 已提交
207

W
wangxiao1021 已提交
208
- [Sentiment Classification](https://github.com/PaddlePaddle/PALM/tree/master/examples/classification)
X
Xiaoyao Xi 已提交
209 210 211
- [Question Pairs matching](https://github.com/PaddlePaddle/PALM/tree/master/examples/matching)
- [Named Entity Recognition](https://github.com/PaddlePaddle/PALM/tree/master/examples/tagging)
- [SQuAD-like Machine Reading Comprehension](https://github.com/PaddlePaddle/PALM/tree/master/examples/mrc).
X
Xiaoyao Xi 已提交
212

X
Xiaoyao Xi 已提交
213

X
Xiaoyao Xi 已提交
214
#### Multi-task Learning
W
wangxiao1021 已提交
215
To run with multi-task learning mode:
X
Xiaoyao Xi 已提交
216

W
wangxiao1021 已提交
217 218 219 220 221
1. repeatedly create components (i.e., reader, backbone and head) for each task followed with step 1~5 above. 
2. create empty trainers (each trainer is corresponded to one task) and pass them to create a `MultiHeadTrainer`. 
3. build multi-task forward graph with `multi_head_trainer.build_forward` method.
4. use `paddlepalm.optimizer` (and `paddlepalm.lr_sched` if is necessary) to create a *optimizer*, then build backward through `multi_head_trainer.build_backward`.
5. fit all prepared readers and data to multi_head_trainer with `multi_head_trainer.fit_readers` method.
W
wangxiao1021 已提交
222
6. load pretrain model with `multi_head_trainer.load_pretrain`, or load checkpoint with `multi_head_trainer.load_ckpt` or nothing to do for training from scratch, then do training with `multi_head_trainer.train`.
X
Xiaoyao Xi 已提交
223

W
wangxiao1021 已提交
224
The save/load and predict operations of a multi_head_trainer is the same as a trainer.
X
Xiaoyao Xi 已提交
225

X
Xiaoyao Xi 已提交
226
For more implementation details with `multi_head_trainer`, see
X
Xiaoyao Xi 已提交
227

W
wangxiao1021 已提交
228
- [ATIS: joint training of dialogue intent recognition and slot filling](https://github.com/PaddlePaddle/PALM/tree/master/examples/multi-task)
X
Xiaoyao Xi 已提交
229

X
Xiaoyao Xi 已提交
230 231 232 233 234 235
#### Save models

To save models/checkpoints and logs during training, just call `trainer.set_saver` method. More implementation details see [this](https://github.com/PaddlePaddle/PALM/tree/master/examples).

#### Evaluation/Inference
To do predict/evaluation after a training stage, just create another three reader, backbone and head instance with `phase='predict'` (repeat step 1~4 above). Then do predicting with `predict` method in trainer (no need to create another trainer). More implementation details see [this](https://github.com/PaddlePaddle/PALM/tree/master/examples/predict).
X
Xiaoyao Xi 已提交
236

X
Xiaoyao Xi 已提交
237
#### Multiple GPUs
X
Xiaoyao Xi 已提交
238
If there exists multiple GPUs in your environment, you can control the number and index of these GPUs through the environment variable [CUDA_VISIBLE_DEVICES](https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/). For example, if 4 GPUs in your enviroment, indexed with 0,1,2,3, you can run with GPU2 only with following commands
X
Xiaoyao Xi 已提交
239 240 241 242 243 244 245 246 247 248 249 250 251

```shell
CUDA_VISIBLE_DEVICES=2 python run.py
```

Multiple GPUs should be seperated with `,`. For example, running with GPU2 and GPU3, following commands is refered:

```shell
CUDA_VISIBLE_DEVICES=2,3 python run.py
```

On multi-gpu mode, PaddlePALM will automatically split each batch onto the available cards. For example, if the `batch_size` is set 64, and there are 4 cards visible for PaddlePALM, then the batch_size in each card is actually 64/4=16. Therefore, when running with multiple cards, **you need to ensure that the set batch_size can be divided by the number of cards.**

X
Xiaoyao Xi 已提交
252 253 254 255
## License

This tutorial is contributed by [PaddlePaddle](https://github.com/PaddlePaddle/Paddle) and licensed under the [Apache-2.0 license](https://github.com/PaddlePaddle/models/blob/develop/LICENSE).