diff --git a/official/nlp/MODEL_GARDEN.md b/official/nlp/MODEL_GARDEN.md index 5d590a9337cf9cb84294eff5ca1da7a74984e375..09294e942e8d6a2e001f36b0f3650a8496bfe69f 100644 --- a/official/nlp/MODEL_GARDEN.md +++ b/official/nlp/MODEL_GARDEN.md @@ -2,53 +2,69 @@ ## Introduction -This TF-NLP library provides a collection of scripts for the training and -evaluation of transformer-based models, on various tasks such as sentence +The TF-NLP library provides a collection of scripts for training and +evaluating transformer-based models, on various tasks such as sentence classification, question answering, and translation. Additionally, we provide checkpoints of pretrained models which can be finetuned on downstream tasks. ### How to Train Models -Model Garden can be easily installed using PIP -(`pip install tf-models-nightly`). After installation, check out +Model Garden can be easily installed with +`pip install tf-models-nightly`. After installation, check out [this instruction](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md) on how to train models with this codebase. -## Available Tasks -There are two available model configs (we will add more) under -`configs/experiments/`: +By default, the experiment runs on GPUs. To run on TPUs, one should overwrite +`runtime.distribution_strategy` and set the tpu address. See [RuntimeConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) for details. + +In general, the experiments can run with the folloing command by setting the +corresponding `${TASK}`, `${TASK_CONFIG}`, `${MODEL_CONFIG}`. +``` +EXPERIMENT=??? +TASK_CONFIG=??? +MODEL_CONFIG=??? +EXRTRA_PARAMS=??? +MODEL_DIR=??? # a-folder-to-hold-checkpoints-and-logs +python3 train.py \ + --experiment=${EXPERIMENT} \ + --mode=train_and_eval \ + --model_dir=${MODEL_DIR} \ + --config_file=${TASK_CONFIG} \ + --config_file=${MODEL_CONFIG} \ + --params_override=${EXRTRA_PARAMS} +``` + +* `EXPERIMENT` can be found under `configs/` +* `TASK_CONFIG` can be found under `configs/experiments/` +* `MODEL_CONFIG` can be found under `configs/models/` + +#### Order of params override: +1. `train.py` looks up the registered `ExperimentConfig` with `${EXPERIMENT}` +2. Overrides params in `TaskConfig` in `${TASK_CONFIG}` +3. Overrides params `model` in `TaskConfig` with `${MODEL_CONFIG}` +4. Overrides any params in `ExperimentConfig` with `${EXTRA_PARAMS}` + +Note that +1. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` can be optional when EXPERIMENT default is enough. +2. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` are only guaranteed to be compatible to it's `${EXPERIMENT}` that defines it. + +## Experiments + +| NAME | EXPERIMENT | TASK_CONFIG | MODEL_CONFIG | EXRTRA_PARAMS | +| ----------------- | ------------------------ | ------- | -------- | ----------- | +| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) |
data and bert-base hub inittask.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4
| +| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) |
data and bert-base ckpt inittask.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.init_checkpoint=gs://tf_model_garden/nlp/bert/uncased_L-12_H-768_A-12/bert_model.ckpt
| +| BERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) |
data and bert-base hub inittask.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4
| +|ALBERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml)|
data and albert-base hub inittask.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3
| +| Transformer-large WMT14/en-de scratch |[wmt_transformer/large](https://github.com/tensorflow/models/blob/master/official/nlp/configs/wmt_transformer_experiments.py)| | |
ende-32k sentencepiecetask.sentencepiece_model_path='gs://tf_model_garden/nlp/transformer_wmt/ende_bpe_32k.model'
| -| Dataset | Task | Config | Example command | -| ----------------- | ------------------------ | ------- | ---- | -| GLUE/MNLI-matched | bert/sentence_prediction | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) |
finetune BERT-base on this task PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4

python3 train.py \\
--experiment=bert/sentence_prediction \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/bert_en_uncased_base.yaml \\
--config_file=configs/experiments/glue_mnli_matched.yaml \\
--params_override=${PARAMS}
| -| SQuAD v1.1 | bert/squad | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) |
finetune BERT-base on this task PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4

python3 train.py \\
--experiment=bert/squad \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/bert_en_uncased_base.yaml \\
--config_file=configs/experiments/squad_v1.yaml \\
--params_override=${PARAMS}
| - -One example on how to use the config file: if you want to work on the SQuAD -question answering task, set -`--config_file=configs/experiments/squad_v1.yaml` and -`--experiment=bert/squad` -as arguments to `train.py`. - -## Available Model Configs - -There are two available model configs (we will add more) under -`configs/models/`: - -| Model | Config | Pretrained checkpoint & Vocabulary | TF-HUB SavedModel | Example command | -| ------------ | ------- | ---------------------------------- | ----------------- | --------------- | -| BERT-base | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | [uncased_L-12_H-768_A-12](https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/uncased_L-12_H-768_A-12.tar.gz) | [uncased_L-12_H-768_A-12](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/) |
finetune on SQuAD v1.1 PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4

python3 train.py \\
--experiment=bert/squad \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/bert_en_uncased_base.yaml \\
--config_file=configs/experiments/squad_v1.yaml \\
--params_override=${PARAMS}
| -| ALBERT-base | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml) | [albert_en_base](https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_base.tar.gz) | [albert_en_base](https://tfhub.dev/tensorflow/albert_en_base/3) |
finetune on SQuAD v1.1 PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3

python3 train.py \\
--experiment=bert/squad \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/albert_base.yaml \\
--config_file=configs/experiments/squad_v1.yaml \\
--params_override=${PARAMS}
| - -One example on how to use the config file: if you want to train an ALBERT-base -model, set `--config_file=configs/models/albert_base.yaml` as an argument to -`train.py`. ## Useful links [How to Train Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md) -[List of Pretrained Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md) +[List of Pretrained Models for finetuning](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md) [How to Publish Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/tfhub.md)