add fp16 in image classification (#4087)

PaddleCV/image_classification/README.md

@@ -264,7 +264,15 @@ Mixup相关介绍参考[mixup: Beyond Empirical Risk Minimization](https://arxiv
 
 ### 混合精度训练
 
-FP16相关内容已经迁移至PaddlePaddle/Fleet 中
+通过指定--fp16=True 启动混合训练，在训练过程中会使用float16数据，并输出float32的模型参数。您可能需要同时传入--scale_loss来解决fp16训练的精度问题，通常传入--scale_loss=0.8即可。
+
+```bash
+python train.py \
+    --model=ResNet50 \
+    --fp16=True \
+    --scale_loss=0.8
+```
+具体内容也可参考[Fleet](https://github.com/PaddlePaddle/Fleet/tree/develop/benchmark/collective/resnet)
 
 ### 性能分析
 
@@ -323,7 +331,7 @@ python -m paddle.distributed.launch train.py \
 
 #### 注意事项
 
-1. PaddlePaddle需使用1.6或以上的版本，并且需要使用GCC5.4以上编译器[编译](https://www.paddlepaddle.org.cn/install/doc/source/ubuntu), 另外，请在编译过程中指定-DWITH_DISTRIBUTE=ON 来启动多进程训练模式。
+1. 请务必使用GCC5.4以上编译器[编译安装](https://www.paddlepaddle.org.cn/install/doc/source/ubuntu)的1.6或以上版本paddlepaddle, 另外，请在编译过程中指定-DWITH_DISTRIBUTE=ON 来启动多进程训练模式。注意：官方的paddlepaddle是GCC4.8编译的，请务必检查此项，或参考使用[已经编译好的whl包](https://github.com/NVIDIA/DALI/blob/master/qa/setup_packages.py#L38)
 2. Nvidia DALI需要使用[#1371](https://github.com/NVIDIA/DALI/pull/1371)以后的git版本。请参考[此文档](https://docs.nvidia.com/deeplearning/sdk/dali-master-branch-user-guide/docs/installation.html)安装nightly版本或从源码安装。
 3. 因为DALI使用GPU进行图片预处理，需要占用部分显存，请适当调整 `FLAGS_fraction_of_gpu_memory_to_use`环境变量（如`0.8`）来预留部分显存供DALI使用。
 

PaddleCV/image_classification/README_en.md

@@ -253,7 +253,16 @@ Refer to [mixup: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.
 
 ### Using Mixed-Precision Training
 
-Mixed-precision part is moving to PaddlePaddle/Fleet now.
+Set --fp16=True to sart Mixed-Precision Training.
+
+```bash
+python train.py \
+    --model=ResNet50 \
+    --fp16=True \
+    --scale_loss=0.8
+```
+
+Refer to [PaddlePaddle/Fleet](https://github.com/PaddlePaddle/Fleet/tree/develop/benchmark/collective/resnet)
 
 ### Preprocessing with Nvidia DALI
 

PaddleCV/image_classification/train.py

@@ -63,11 +63,18 @@ def build_program(is_train, main_prog, startup_prog, args):
             if is_train:
                 optimizer = create_optimizer(args)
                 avg_cost = loss_out[0]
-                optimizer.minimize(avg_cost)
                 #XXX: fetch learning rate now, better implement is required here. 
                 global_lr = optimizer._global_learning_rate()
                 global_lr.persistable = True
                 loss_out.append(global_lr)
+
+                if args.use_fp16:
+                    optimizer = fluid.contrib.mixed_precision.decorate(
+                        optimizer,
+                        init_loss_scaling=args.scale_loss,
+                        use_dynamic_loss_scaling=args.use_dynamic_loss_scaling)
+
+                optimizer.minimize(avg_cost)
                 if args.use_ema:
                     global_steps = fluid.layers.learning_rate_scheduler._decay_step_counter(
                     )

PaddleCV/image_classification/utils/utility.py

@@ -131,9 +131,10 @@ def parse_args():
 
     # SWITCH
     add_arg('validate',                 bool,   True,                   "whether to validate when training.")
-    #NOTE: (2019/08/08) FP16 is moving to PaddlePaddle/Fleet now
-    #add_arg('use_fp16',                 bool,   False,                  "Whether to enable half precision training with fp16." )
-    #add_arg('scale_loss',               float,  1.0,                    "The value of scale_loss for fp16." )
+    add_arg('use_fp16',                 bool,   False,                  "Whether to enable half precision training with fp16." )
+    add_arg('scale_loss',               float,  1.0,                    "The value of scale_loss for fp16." )
+    add_arg('use_dynamic_loss_scaling', bool,   True,                   "Whether to use dynamic loss scaling.")
+
     add_arg('use_label_smoothing',      bool,   False,                  "Whether to use label_smoothing")
     add_arg('label_smoothing_epsilon',  float,  0.1,                    "The value of label_smoothing_epsilon parameter")
     #NOTE: (2019/08/08) temporary disable use_distill
@@ -273,6 +274,11 @@ def check_args(args):
 
     assert args.class_dim > 1, "class_dim must greater than 1"
 
+    if args.use_dali:
+        print(
+            "DALI preprocessing is activated!!!\nWarning: 1. Please make sure paddlepaddle is compiled by GCC5.4 or later version!\n\t 2. Please make sure nightly builds DALI is installed correctly.\n----------------------------------------------------"
+        )
+
     #check gpu
     check_gpu()
     check_version()