diff --git a/official/nlp/MODEL_GARDEN.md b/official/nlp/MODEL_GARDEN.md
index 5d590a9337cf9cb84294eff5ca1da7a74984e375..09294e942e8d6a2e001f36b0f3650a8496bfe69f 100644
--- a/official/nlp/MODEL_GARDEN.md
+++ b/official/nlp/MODEL_GARDEN.md
@@ -2,53 +2,69 @@
## Introduction
-This TF-NLP library provides a collection of scripts for the training and
-evaluation of transformer-based models, on various tasks such as sentence
+The TF-NLP library provides a collection of scripts for training and
+evaluating transformer-based models, on various tasks such as sentence
classification, question answering, and translation. Additionally, we provide
checkpoints of pretrained models which can be finetuned on downstream tasks.
### How to Train Models
-Model Garden can be easily installed using PIP
-(`pip install tf-models-nightly`). After installation, check out
+Model Garden can be easily installed with
+`pip install tf-models-nightly`. After installation, check out
[this instruction](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md)
on how to train models with this codebase.
-## Available Tasks
-There are two available model configs (we will add more) under
-`configs/experiments/`:
+By default, the experiment runs on GPUs. To run on TPUs, one should overwrite
+`runtime.distribution_strategy` and set the tpu address. See [RuntimeConfig](https://github.com/tensorflow/models/blob/master/official/core/config_definitions.py) for details.
+
+In general, the experiments can run with the folloing command by setting the
+corresponding `${TASK}`, `${TASK_CONFIG}`, `${MODEL_CONFIG}`.
+```
+EXPERIMENT=???
+TASK_CONFIG=???
+MODEL_CONFIG=???
+EXRTRA_PARAMS=???
+MODEL_DIR=??? # a-folder-to-hold-checkpoints-and-logs
+python3 train.py \
+ --experiment=${EXPERIMENT} \
+ --mode=train_and_eval \
+ --model_dir=${MODEL_DIR} \
+ --config_file=${TASK_CONFIG} \
+ --config_file=${MODEL_CONFIG} \
+ --params_override=${EXRTRA_PARAMS}
+```
+
+* `EXPERIMENT` can be found under `configs/`
+* `TASK_CONFIG` can be found under `configs/experiments/`
+* `MODEL_CONFIG` can be found under `configs/models/`
+
+#### Order of params override:
+1. `train.py` looks up the registered `ExperimentConfig` with `${EXPERIMENT}`
+2. Overrides params in `TaskConfig` in `${TASK_CONFIG}`
+3. Overrides params `model` in `TaskConfig` with `${MODEL_CONFIG}`
+4. Overrides any params in `ExperimentConfig` with `${EXTRA_PARAMS}`
+
+Note that
+1. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` can be optional when EXPERIMENT default is enough.
+2. `${TASK_CONFIG}`, `${MODEL_CONFIG}`, `${EXTRA_PARAMS}` are only guaranteed to be compatible to it's `${EXPERIMENT}` that defines it.
+
+## Experiments
+
+| NAME | EXPERIMENT | TASK_CONFIG | MODEL_CONFIG | EXRTRA_PARAMS |
+| ----------------- | ------------------------ | ------- | -------- | ----------- |
+| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | data and bert-base hub init
task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 |
+| BERT-base GLUE/MNLI-matched finetune | [bert/sentence_prediction](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | data and bert-base ckpt init
task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.init_checkpoint=gs://tf_model_garden/nlp/bert/uncased_L-12_H-768_A-12/bert_model.ckpt |
+| BERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | data and bert-base hub init
task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 |
+|ALBERT-base SQuAD v1.1 finetune | [bert/squad](https://github.com/tensorflow/models/blob/master/official/nlp/configs/finetuning_experiments.py) | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml)| data and albert-base hub init
task.train_data.input_path=/path-to-your-training-data,task.validation_data.input_path=/path-to-your-val-data,task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3 |
+| Transformer-large WMT14/en-de scratch |[wmt_transformer/large](https://github.com/tensorflow/models/blob/master/official/nlp/configs/wmt_transformer_experiments.py)| | | ende-32k sentencepiece
task.sentencepiece_model_path='gs://tf_model_garden/nlp/transformer_wmt/ende_bpe_32k.model' |
-| Dataset | Task | Config | Example command |
-| ----------------- | ------------------------ | ------- | ---- |
-| GLUE/MNLI-matched | bert/sentence_prediction | [glue_mnli_matched.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/glue_mnli_matched.yaml) | finetune BERT-base on this task
PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4
python3 train.py \\
--experiment=bert/sentence_prediction \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/bert_en_uncased_base.yaml \\
--config_file=configs/experiments/glue_mnli_matched.yaml \\
--params_override=${PARAMS} |
-| SQuAD v1.1 | bert/squad | [squad_v1.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/experiments/squad_v1.yaml) | finetune BERT-base on this task
PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4
python3 train.py \\
--experiment=bert/squad \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/bert_en_uncased_base.yaml \\
--config_file=configs/experiments/squad_v1.yaml \\
--params_override=${PARAMS} |
-
-One example on how to use the config file: if you want to work on the SQuAD
-question answering task, set
-`--config_file=configs/experiments/squad_v1.yaml` and
-`--experiment=bert/squad`
-as arguments to `train.py`.
-
-## Available Model Configs
-
-There are two available model configs (we will add more) under
-`configs/models/`:
-
-| Model | Config | Pretrained checkpoint & Vocabulary | TF-HUB SavedModel | Example command |
-| ------------ | ------- | ---------------------------------- | ----------------- | --------------- |
-| BERT-base | [bert_en_uncased_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/bert_en_uncased_base.yaml) | [uncased_L-12_H-768_A-12](https://storage.googleapis.com/tf_model_garden/nlp/bert/v3/uncased_L-12_H-768_A-12.tar.gz) | [uncased_L-12_H-768_A-12](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/) | finetune on SQuAD v1.1
PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4
python3 train.py \\
--experiment=bert/squad \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/bert_en_uncased_base.yaml \\
--config_file=configs/experiments/squad_v1.yaml \\
--params_override=${PARAMS} |
-| ALBERT-base | [albert_base.yaml](https://github.com/tensorflow/models/blob/master/official/nlp/configs/models/albert_base.yaml) | [albert_en_base](https://storage.googleapis.com/tf_model_garden/nlp/albert/albert_base.tar.gz) | [albert_en_base](https://tfhub.dev/tensorflow/albert_en_base/3) | finetune on SQuAD v1.1
PARAMS=runtime.distribution_strategy=mirrored
PARAMS=${PARAMS},task.train_data.input_path=/path-to-your-training-data/
PARAMS=${PARAMS},task.hub_module_url=https://tfhub.dev/tensorflow/albert_en_base/3
python3 train.py \\
--experiment=bert/squad \\
--mode=train \\
--model_dir=/a-folder-to-hold-checkpoints-and-logs/ \\
--config_file=configs/models/albert_base.yaml \\
--config_file=configs/experiments/squad_v1.yaml \\
--params_override=${PARAMS} |
-
-One example on how to use the config file: if you want to train an ALBERT-base
-model, set `--config_file=configs/models/albert_base.yaml` as an argument to
-`train.py`.
## Useful links
[How to Train Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md)
-[List of Pretrained Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md)
+[List of Pretrained Models for finetuning](https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrained_models.md)
[How to Publish Models](https://github.com/tensorflow/models/blob/master/official/nlp/docs/tfhub.md)