In PaddleDetection, making use of tb-paddle to record data, so as to display...

In PaddleDetection,  making use of tb-paddle to record data, so as to display in TensorBoard.  (#3295)

* use tb_paddle to log data
including:

* loss
* mAP
* image

* modified:   requirements.txt
* modify the missing argument --use_tb
* modify GETTING_STARTED.md and docs/GETTING_STARTED_cn.md

add the argument of tb-paddle in docs.
* optimize the display name of image displaying.

docs/GETTING_STARTED.md

@@ -38,6 +38,8 @@ python tools/train.py -c configs/faster_rcnn_r50_1x.yml -o use_gpu=false
 - `--output_eval`: If perform evaluation in training, this edits evaluation directory, default is current directory.
 - `-d` or `--dataset_dir`: Dataset path, same as `dataset_dir` of configs. Such as: `-d dataset/coco`
 - `-o`: Set configuration options in config file. Such as: `-o max_iters=180000`
+- `--use_tb`: Whether to record the data with [tb-paddle](https://github.com/linshuliang/tb-paddle), so as to display in Tensorboard, default is `False`
+- `--tb_log_dir`: tb-paddle logging directory for scalar, default is `tb_log_dir/scalar`
 
 
 ##### Examples
@@ -150,10 +152,13 @@ python tools/infer.py -c configs/faster_rcnn_r50_1x.yml --infer_dir=demo
 - `--output_dir`: Directory for storing the output visualization files.
 - `--draw_threshold`: Threshold to reserve the result for visualization. Default is 0.5.
 - `--save_inference_model`: Save inference model in output_dir if True.
+- `--use_tb`: Whether to record the data with [tb-paddle](https://github.com/linshuliang/tb-paddle), so as to display in Tensorboard, default is `False`
+- `--tb_log_dir`: tb-paddle logging directory for image, default is `tb_log_dir/image`
 
 #### Examples
 
 - Output specified directory && Set up threshold
+
 ```bash
 # run on GPU with:
 export PYTHONPATH=$PYTHONPATH:.
@@ -162,11 +167,16 @@ python tools/infer.py -c configs/faster_rcnn_r50_1x.yml \
                       --infer_img=demo/000000570688.jpg \
                       --output_dir=infer_output/ \
                       --draw_threshold=0.5 \
-                      -o weights=output/faster_rcnn_r50_1x/model_final
+                      -o weights=output/faster_rcnn_r50_1x/model_final \
+                      --use_tb=Ture
 ```
-The visualization files are saved in `output` by default, to specify a different
-path, simply add a `--output_dir=` flag.  
-`--draw_threshold` is an optional argument. Default is 0.5. Different thresholds will produce different results depending on the calculation of [NMS](https://ieeexplore.ieee.org/document/1699659). If users want to infer according to customized model path, `-o weights` can be set for specified path.
+
+The visualization files are saved in `output` by default, to specify a different path, simply add a `--output_dir=` flag.  
+`--draw_threshold` is an optional argument. Default is 0.5. 
+Different thresholds will produce different results depending on the calculation of [NMS](https://ieeexplore.ieee.org/document/1699659).
+If users want to infer according to customized model path, `-o weights` can be set for specified path.
+`--use_tb` is an optional argument, if `--use_tb` is `True`, the tb-paddle will record data in directory, 
+so users can see the results in Tensorboard.
 
 - Save inference model
 

docs/GETTING_STARTED_cn.md

@@ -39,20 +39,26 @@ python tools/train.py -c configs/faster_rcnn_r50_1x.yml -o use_gpu=false
 - `--output_eval`: 如果边训练边测试, 这个参数可以编辑评测保存json路径, 默认是当前目录。
 - `-d` or `--dataset_dir`: 数据集路径, 同配置文件里的`dataset_dir`. 例如: `-d dataset/coco`
 - `-o`: 设置配置文件里的参数内容。 例如: `-o max_iters=180000`
+- `--use_tb`: 是否使用[tb-paddle](https://github.com/linshuliang/tb-paddle)记录数据，进而在TensorBoard中显示，默认是False。
+- `--tb_log_dir`: 指定 tb-paddle 记录数据的存储路径，默认是`tb_log_dir/scalar`。
 
 ##### 例子
 
 - 边训练边测试
+
 ```bash
 export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
 export PYTHONPATH=$PYTHONPATH:.
 python -u tools/train.py -c configs/faster_rcnn_r50_1x.yml --eval
 ```
-可通过设置`--eval`在训练epoch中交替执行评估, 评估在每个snapshot_iter时开始。可在配置文件的`snapshot_iter`处修改。
-如果验证集很大，测试将会比较耗时，影响训练速度，建议减少评估次数，或训练完再进行评估。当边训练边测试时，在每次snapshot_iter会评测出最佳mAP模型保存到
+
+可通过设置`--eval`在训练epoch中交替执行评估, 评估在每个snapshot\_iter时开始。可在配置文件的`snapshot_iter`处修改。
+如果验证集很大，测试将会比较耗时，影响训练速度，建议减少评估次数，或训练完再进行评估。
+当边训练边测试时，在每次snapshot\_iter会评测出最佳mAP模型保存到
 `best_model`文件夹下，`best_model`的路径和`model_final`的路径相同。
 
 - 设置配置文件参数 && 指定数据集路径
+
 ```bash
 export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
 export PYTHONPATH=$PYTHONPATH:.
@@ -143,11 +149,14 @@ python tools/infer.py -c configs/faster_rcnn_r50_1x.yml --infer_dir=demo
 
 - `--output_dir`: 输出推断后可视化文件。
 - `--draw_threshold`: 设置推断的阈值。默认是0.5.
-- `--save_inference_model`: 设为`True`时，将预测模型保存到output_dir中.
+- `--save_inference_model`: 设为`True`时，将预测模型保存到output\_dir中.
+- `--use_tb`: 是否使用[tb-paddle](https://github.com/linshuliang/tb-paddle)记录数据，进而在TensorBoard中显示，默认是False。
+- `--tb_log_dir`: 指定 tb-paddle 记录数据的存储路径，默认是`tb_log_dir/image`。
 
 #### 例子
 
 - 设置输出路径 && 设置推断阈值
+
 ```bash
 # GPU推断
 export CUDA_VISIBLE_DEVICES=0
@@ -156,12 +165,15 @@ python tools/infer.py -c configs/faster_rcnn_r50_1x.yml \
                       --infer_img=demo/000000570688.jpg \
                       --output_dir=infer_output/ \
                       --draw_threshold=0.5 \
-                      -o weights=output/faster_rcnn_r50_1x/model_final
+                      -o weights=output/faster_rcnn_r50_1x/model_final \
+                      --use_tb=True
 ```
 
 
 可视化文件默认保存在`output`中，可通过`--output_dir=`指定不同的输出路径。  
-`--draw_threshold` 是个可选参数. 根据 [NMS](https://ieeexplore.ieee.org/document/1699659) 的计算，不同阈值会产生不同的结果。如果用户需要对自定义路径的模型进行推断，可以设置`-o weights`指定模型路径。
+`--draw_threshold` 是个可选参数. 根据 [NMS](https://ieeexplore.ieee.org/document/1699659) 的计算，
+不同阈值会产生不同的结果。如果用户需要对自定义路径的模型进行推断，可以设置`-o weights`指定模型路径。
+`--use_tb`是个可选参数，当为`True`时，可使用 TensorBoard 来可视化参数的变化趋势和图片。
 
 - 保存推断模型
 

requirements.txt

 tqdm
 docstring_parser @ http://github.com/willthefrog/docstring_parser/tarball/master
 typeguard ; python_version >= '3.4'
+tb-paddle
+tb-nightly

tools/infer.py

@@ -211,6 +211,13 @@ def main():
             callable(model.is_bbox_normalized):
         is_bbox_normalized = model.is_bbox_normalized()
 
+    # use tb-paddle to log image
+    if FLAGS.use_tb:
+        from tb_paddle import SummaryWriter
+        tb_writer = SummaryWriter(FLAGS.tb_log_dir)
+        tb_image_step = 0
+        tb_image_frame = 0 # each frame can display ten pictures at most. 
+
     imid2path = reader.imid2path
     for iter_id, data in enumerate(reader()):
         outs = exe.run(infer_prog,
@@ -236,10 +243,34 @@ def main():
         for im_id in im_ids:
             image_path = imid2path[int(im_id)]
             image = Image.open(image_path).convert('RGB')
+
+            # use tb-paddle to log original image           
+            if FLAGS.use_tb:
+                original_image_np = np.array(image)
+                tb_writer.add_image(
+                    "original/frame_{}".format(tb_image_frame),
+                    original_image_np,
+                    tb_image_step,
+                    dataformats='HWC')
+
             image = visualize_results(image,
                                       int(im_id), catid2name,
                                       FLAGS.draw_threshold, bbox_results,
                                       mask_results)
+            
+            # use tb-paddle to log image with bbox
+            if FLAGS.use_tb:
+                infer_image_np = np.array(image)
+                tb_writer.add_image(
+                    "bbox/frame_{}".format(tb_image_frame),
+                    infer_image_np,
+                    tb_image_step,
+                    dataformats='HWC')
+                tb_image_step += 1
+                if tb_image_step % 10 == 0:
+                    tb_image_step = 0
+                    tb_image_frame += 1
+
             save_name = get_save_image_name(FLAGS.output_dir, image_path)
             logger.info("Detection bbox results save in {}".format(save_name))
             image.save(save_name, quality=95)
@@ -272,5 +303,15 @@ if __name__ == '__main__':
         action='store_true',
         default=False,
         help="Save inference model in output_dir if True.")
+    parser.add_argument(
+        "--use_tb",
+        type=bool,
+        default=False,
+        help="whether to record the data to Tensorboard.")
+    parser.add_argument(
+        '--tb_log_dir',
+        type=str,
+        default="tb_log_dir/image",
+        help='Tensorboard logging directory for image.')
     FLAGS = parser.parse_args()
     main()

tools/train.py

@@ -174,6 +174,14 @@ def main():
     save_dir = os.path.join(cfg.save_dir, cfg_name)
     time_stat = deque(maxlen=cfg.log_iter)
     best_box_ap_list = [0.0, 0]  #[map, iter]
+
+    # use tb-paddle to log data
+    if FLAGS.use_tb:
+        from tb_paddle import SummaryWriter
+        tb_writer = SummaryWriter(FLAGS.tb_log_dir)
+        tb_loss_step = 0
+        tb_mAP_step = 0
+
     for it in range(start_iter, cfg.max_iters):
         start_time = end_time
         end_time = time.time()
@@ -183,6 +191,14 @@ def main():
         eta = str(datetime.timedelta(seconds=int(eta_sec)))
         outs = exe.run(train_compile_program, fetch_list=train_values)
         stats = {k: np.array(v).mean() for k, v in zip(train_keys, outs[:-1])}
+
+        # use tb-paddle to log loss
+        if FLAGS.use_tb:
+            if it % cfg.log_iter == 0:
+                for loss_name, loss_value in stats.items():
+                    tb_writer.add_scalar(loss_name, loss_value, tb_loss_step)
+                tb_loss_step += 1
+
         train_stats.update(stats)
         logs = train_stats.log()
         if it % cfg.log_iter == 0:
@@ -204,6 +220,12 @@ def main():
                 box_ap_stats = eval_results(
                     results, eval_feed, cfg.metric, cfg.num_classes, resolution,
                     is_bbox_normalized, FLAGS.output_eval, map_type)
+                
+                # use tb_paddle to log mAP
+                if FLAGS.use_tb:
+                    tb_writer.add_scalar("mAP", box_ap_stats[0], tb_mAP_step)
+                    tb_mAP_step += 1
+                
                 if box_ap_stats[0] > best_box_ap_list[0]:
                     best_box_ap_list[0] = box_ap_stats[0]
                     best_box_ap_list[1] = it
@@ -239,5 +261,15 @@ if __name__ == '__main__':
         default=None,
         type=str,
         help="Dataset path, same as DataFeed.dataset.dataset_dir")
+    parser.add_argument(
+        "--use_tb",
+        type=bool,
+        default=False,
+        help="whether to record the data to Tensorboard.")
+    parser.add_argument(
+        '--tb_log_dir',
+        type=str,
+        default="tb_log_dir/scalar",
+        help='Tensorboard logging directory for scalar.')
     FLAGS = parser.parse_args()
     main()