| -c | ALL | Select config file | None | **The whole description of configure can refer to [config_example](config_example)** |
| -o | ALL | Set parameters in configure file | None | `-o` has higher priority to file configured by `-c`. Such as `-o use_gpu=False max_iter=10000` |
| -r/--resume_checkpoint | train | Checkpoint path for resuming training | None | `-r output/faster_rcnn_r50_1x/10000` |
| --eval | train | Whether to perform evaluation in training | False | |
| --output_eval | train/eval | json path in evalution | current path | `--output_eval ./json_result` |
| -d/--dataset_dir | train/eval | path for dataset, same as dataset_dir in configs | None | `-d dataset/coco` |
| --fp16 | train | Whether to enable mixed precision training | False | GPU training is required |
| --loss_scale | train | Loss scaling factor for mixed precision training | 8.0 | enable when `--fp16` is True |
| --json_eval | eval | Whether to evaluate with already existed bbox.json or mask.json | False | json path is set in `--output_eval` |
| --output_dir | infer | Directory for storing the output visualization files | `./output` | `--output_dir output` |
| --draw_threshold | infer | Threshold to reserve the result for visualization | 0.5 | `--draw_threshold 0.7` |
| --save\_inference_model | infer | Whether to save inference model in output_dir | False | save_inference_model is saved in `--output_dir` |
| --infer_dir | infer | Directory for images to perform inference on | None | |
| --use_tb | train/infer | Whether to record the data with [tb-paddle](https://github.com/linshuliang/tb-paddle), so as to display in Tensorboard | False | |
-`-r` or `--resume_checkpoint`: Checkpoint path for resuming training. Such as: `-r output/faster_rcnn_r50_1x/10000`
-`--eval`: Whether to perform evaluation in training, default is `False`
-`--output_eval`: If perform evaluation in training, this edits evaluation directory, default is current directory.
-`-d` or `--dataset_dir`: Dataset path, same as `dataset_dir` of configs. Such as: `-d dataset/coco`
-`-c`: Select config file and all files are saved in `configs/`
-`-o`: Set configuration options in config file. Such as: `-o max_iters=180000`. `-o` has higher priority to file configured by `-c`
-`--use_tb`: Whether to record the data with [tb-paddle](https://github.com/linshuliang/tb-paddle), so as to display in Tensorboard, default is `False`
-`--tb_log_dir`: tb-paddle logging directory for scalar, default is `tb_log_dir/scalar`
-`--fp16`: Whether to enable mixed precision training (requires GPU), default is `False`
-`--loss_scale`: Loss scaling factor for mixed precision training, default is `8.0`
Alternating between training epoch and evaluation run is possible, simply pass
```bash
in `--eval` to do so and evaluate at each snapshot_iter. It can be modified at `snapshot_iter` of the configuration file. If evaluation dataset is large and
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
causes time-consuming in training, we suggest decreasing evaluation times or evaluating after training. When perform evaluation in training,
the best model with highest MAP is saved at each `snapshot_iter`. `best_model` has the same path as `model_final`.
```
Perform training and evalution alternatively and evaluate at each snapshot_iter. Meanwhile, the best model with highest MAP is saved at each `snapshot_iter` which has the same path as `model_final`.
- Configure dataset path
If evaluation dataset is large, we suggest decreasing evaluation times or evaluating after training.
Mixed precision training can be enabled with `--fp16` flag. Currently Faster-FPN, Mask-FPN and Yolov3 have been verified to be working with little to no loss of precision (less than 0.2 mAP)
To speed up mixed precision training, it is recommended to train in multi-process mode, for example
If loss becomes `NaN` during training, try tweak the `--loss_scale` value. Please refer to the Nvidia [documentation](https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#mptrain) on mixed precision training for details.
Also, please note mixed precision training currently requires changing `norm_type` from `affine_channel` to `bn`.
##### NOTES
##### NOTES
-`CUDA_VISIBLE_DEVICES` can specify different gpu numbers. Such as: `export CUDA_VISIBLE_DEVICES=0,1,2,3`. GPU calculation rules can refer [FAQ](#faq)
-`CUDA_VISIBLE_DEVICES` can specify different gpu numbers. Such as: `export CUDA_VISIBLE_DEVICES=0,1,2,3`. GPU calculation rules can refer [FAQ](#faq)
- Dataset is stored in `dataset/coco` by default (configurable).
- Dataset will be downloaded automatically and cached in `~/.cache/paddle/dataset` if not be found locally.
- Dataset will be downloaded automatically and cached in `~/.cache/paddle/dataset` if not be found locally.
- Pretrained model is downloaded automatically and cached in `~/.cache/paddle/weights`.
- Pretrained model is downloaded automatically and cached in `~/.cache/paddle/weights`.
- Model checkpoints are saved in `output` by default (configurable).
- Checkpoints are saved in `output` by default, and can be revised from save_dir in configure files.
- When finetuning, users could set `pretrain_weights` to the models published by PaddlePaddle. Parameters matched by fields in finetune_exclude_pretrained_params will be ignored in loading and fields can be wildcard matching. For detailed information, please refer to [Transfer Learning](TRANSFER_LEARNING.md).
- To check out hyper parameters used, please refer to the [configs](../configs).
- RCNN models training on CPU is not supported on PaddlePaddle<=1.5.1 and will be fixed on later version.
- RCNN models training on CPU is not supported on PaddlePaddle<=1.5.1 and will be fixed on later version.
### Mixed Precision Training
## Evaluation
Mixed precision training can be enabled with `--fp16` flag. Currently Faster-FPN, Mask-FPN and Yolov3 have been verified to be working with little to no loss of precision (less than 0.2 mAP)
To speed up mixed precision training, it is recommended to train in multi-process mode, for example
If loss becomes `NaN` during training, try tweak the `--loss_scale` value. Please refer to the Nvidia [documentation](https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#mptrain) on mixed precision training for details.
Also, please note mixed precision training currently requires changing `norm_type` from `affine_channel` to `bn`.
-`-d` or `--dataset_dir`: Dataset path, same as dataset_dir of configs. Such as: `-d dataset/coco`
-`--output_eval`: Evaluation directory, default is current directory.
-`-o`: Set configuration options in config file. Such as: `-o weights=output/faster_rcnn_r50_1x/model_final`
-`--json_eval`: Whether to eval with already existed bbox.json or mask.json. Default is `False`. Json file directory is assigned by `-f` argument.
#### Examples
### Evaluation
- Evaluate by specified weights path and dataset path
- Evaluate by specified weights path and dataset path
The json file must be named bbox.json or mask.json, placed in the `evaluation/` directory. Or without the `-f` parameter, default is the current directory.
The json file must be named bbox.json or mask.json, placed in the `evaluation/` directory.
#### NOTES
#### NOTES
- Checkpoint is loaded from `output` by default (configurable)
- Multi-GPU evaluation for R-CNN and SSD models is not supported at the
- Multi-GPU evaluation for R-CNN and SSD models is not supported at the
-`--output_dir`: Directory for storing the output visualization files.
-`--draw_threshold`: Threshold to reserve the result for visualization. Default is 0.5.
-`--save_inference_model`: Save inference model in output_dir if True.
-`--use_tb`: Whether to record the data with [tb-paddle](https://github.com/linshuliang/tb-paddle), so as to display in Tensorboard, default is `False`
-`--tb_log_dir`: tb-paddle logging directory for image, default is `tb_log_dir/image`
