Created by: zqx1609

使用命令python3 -u tools/train.py -c configs/mask_rcnn_r50_fpn_1x.yml --eval loss_bbox在10000iter时是0，eval报warning：The number of valid bbox detected is zero. Please use reasonable model and check input data. 这个是不是梯度消失了，训练完也是这个错不能eval。从10000次迭代重新训练的详细报错如下：

2020-06-19 03:20:37,279-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000100] in Optimizer will not take effect, and it will only be applied to other Parameters! loading annotations into memory... Done (t=0.54s) creating index... index created! 2020-06-19 03:20:41,311-INFO: places would be ommited when DataLoader is not iterable W0619 03:20:41.363803 2047 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 60, Driver API Version: 10.2, Runtime API Version: 10.0 W0619 03:20:41.370936 2047 device_context.cc:260] device: 0, cuDNN Version: 7.6. loading annotations into memory... Done (t=13.94s) creating index... index created! 2020-06-19 03:21:13,350-WARNING: Found an invalid bbox in annotations: im_id: 550395, area: 0.0 x1: 9.98, y1: 188.56, x2: 14.52, y2: 188.56. 2020-06-19 03:22:27,912-WARNING: Found an invalid bbox in annotations: im_id: 200365, area: 0.0 x1: 296.65, y1: 388.33, x2: 296.67999999999995, y2: 388.33. 2020-06-19 03:22:53,498-INFO: places would be ommited when DataLoader is not iterable 2020-06-19 03:22:54,197-INFO: iter: 10000, lr: 0.010000, 'loss': '0.745188', 'loss_rpn_bbox': '0.071010', 'loss_bbox': '0.000000', 'loss_mask': '0.413068', 'loss_cls': '0.031566', 'loss_rpn_cls': '0.229543', time: 0.000, eta: 0:00:05 2020-06-19 03:22:54,199-INFO: Save model to output/mask_rcnn_r50_fpn_1x/10000. 2020-06-19 03:23:00,220-INFO: Test iter 0 2020-06-19 03:23:06,929-INFO: Test iter 100 2020-06-19 03:23:13,676-INFO: Test iter 200 2020-06-19 03:23:20,480-INFO: Test iter 300 2020-06-19 03:23:27,271-INFO: Test iter 400 2020-06-19 03:23:34,060-INFO: Test iter 500 2020-06-19 03:23:40,790-INFO: Test iter 600 2020-06-19 03:23:47,534-INFO: Test iter 700 2020-06-19 03:23:54,334-INFO: Test iter 800 2020-06-19 03:24:01,081-INFO: Test iter 900 2020-06-19 03:24:07,821-INFO: Test iter 1000 2020-06-19 03:24:14,483-INFO: Test iter 1100 2020-06-19 03:24:21,094-INFO: Test iter 1200 2020-06-19 03:24:27,814-INFO: Test iter 1300 2020-06-19 03:24:34,583-INFO: Test iter 1400 2020-06-19 03:24:41,332-INFO: Test iter 1500 2020-06-19 03:24:48,070-INFO: Test iter 1600 2020-06-19 03:24:54,763-INFO: Test iter 1700 2020-06-19 03:25:01,495-INFO: Test iter 1800 2020-06-19 03:25:08,216-INFO: Test iter 1900 2020-06-19 03:25:14,899-INFO: Test iter 2000 2020-06-19 03:25:21,600-INFO: Test iter 2100 2020-06-19 03:25:28,309-INFO: Test iter 2200 2020-06-19 03:25:35,095-INFO: Test iter 2300 2020-06-19 03:25:41,760-INFO: Test iter 2400 2020-06-19 03:25:48,538-INFO: Test iter 2500 2020-06-19 03:25:55,360-INFO: Test iter 2600 2020-06-19 03:26:02,018-INFO: Test iter 2700 2020-06-19 03:26:08,782-INFO: Test iter 2800 2020-06-19 03:26:15,477-INFO: Test iter 2900 2020-06-19 03:26:22,185-INFO: Test iter 3000 2020-06-19 03:26:28,920-INFO: Test iter 3100 2020-06-19 03:26:35,626-INFO: Test iter 3200 2020-06-19 03:26:42,265-INFO: Test iter 3300 2020-06-19 03:26:48,960-INFO: Test iter 3400 2020-06-19 03:26:55,673-INFO: Test iter 3500 2020-06-19 03:27:02,280-INFO: Test iter 3600 2020-06-19 03:27:09,046-INFO: Test iter 3700 2020-06-19 03:27:15,836-INFO: Test iter 3800 2020-06-19 03:27:22,577-INFO: Test iter 3900 2020-06-19 03:27:29,411-INFO: Test iter 4000 2020-06-19 03:27:36,115-INFO: Test iter 4100 2020-06-19 03:27:42,872-INFO: Test iter 4200 2020-06-19 03:27:49,675-INFO: Test iter 4300 2020-06-19 03:27:56,415-INFO: Test iter 4400 2020-06-19 03:28:03,162-INFO: Test iter 4500 2020-06-19 03:28:09,873-INFO: Test iter 4600 2020-06-19 03:28:16,634-INFO: Test iter 4700 2020-06-19 03:28:23,303-INFO: Test iter 4800 2020-06-19 03:28:30,030-INFO: Test iter 4900 2020-06-19 03:28:36,641-INFO: Test finish iter 5000 2020-06-19 03:28:36,642-INFO: Total number of images: 5000, inference time: 14.851810193470275 fps. loading annotations into memory... Done (t=0.84s) creating index... index created! 2020-06-19 03:28:37,557-WARNING: The number of valid bbox detected is zero. Please use reasonable model and check input data. stop eval! loading annotations into memory... Done (t=0.46s) creating index... index created! 2020-06-19 03:28:38,154-WARNING: The number of valid mask detected is zero. Please use reasonable model and check input data. 2020-06-19 03:28:38,228-INFO: Best test box ap: 0.0, in iter: 0 2020-06-19 03:28:41,739-INFO: iter: 10020, lr: 0.010000, 'loss': '0.928413', 'loss_rpn_bbox': '0.031707', 'loss_bbox': '0.000004', 'loss_mask': '0.588154', 'loss_cls': '0.070668', 'loss_rpn_cls': '0.239460', time: 17.404, eta: 34 days, 5:45:17 2020-06-19 03:28:45,385-INFO: iter: 10040, lr: 0.010000, 'loss': '1.066859', 'loss_rpn_bbox': '0.050977', 'loss_bbox': '0.000005', 'loss_mask': '0.570866', 'loss_cls': '0.090004', 'loss_rpn_cls': '0.349661', time: 0.181, eta: 8:33:51 2020-06-19 03:28:49,413-INFO: iter: 10060, lr: 0.010000, 'loss': '1.185960', 'loss_rpn_bbox': '0.067141', 'loss_bbox': '0.000005', 'loss_mask': '0.607934', 'loss_cls': '0.117662', 'loss_rpn_cls': '0.366537', time: 0.201, eta: 9:29:34