Created by: Sephirothought

运行到2000iter以后突然出现如下的报错情况，百思不得起解，在运行别的yml时也出现这样的问题，是在运行了一段时间以后才出现这种问题，所以蛮难受的，跪求大神指点。包括是存在脏数据还是数据增广中存在什么问题都在尝试排查，但没找到原因。 2020-07-26 17:17:56,666-INFO: mAP(0.50, 11point) = 39.03 2020-07-26 17:17:56,667-INFO: Save model to output/faster_rcnn_dcn_r50_vd_fpn_3x_server_side/best_model. 2020-07-26 17:18:00,935-INFO: Best test box ap: 39.032466841682144, in iter: 28200 2020-07-26 17:18:09,198-INFO: iter: 28220, lr: 0.010000, 'loss_cls': '0.031215', 'loss_bbox': '0.044445', 'loss_rpn_cls': '0.001842', 'loss_rpn_bbox': '0.001161', 'loss': '0.082873', time: 2.721, eta: 7 days, 14:44:37 2020-07-26 17:18:17,417-INFO: iter: 28240, lr: 0.010000, 'loss_cls': '0.030234', 'loss_bbox': '0.041962', 'loss_rpn_cls': '0.002307', 'loss_rpn_bbox': '0.001171', 'loss': '0.079412', time: 0.411, eta: 1 day, 3:35:10 2020-07-26 17:18:25,557-INFO: iter: 28260, lr: 0.010000, 'loss_cls': '0.030489', 'loss_bbox': '0.043358', 'loss_rpn_cls': '0.001707', 'loss_rpn_bbox': '0.001198', 'loss': '0.074116', time: 0.407, eta: 1 day, 3:19:25 2020-07-26 17:18:31,207-INFO: fail to map op [AutoAugmentImage_f0b51c] with error: tuple index out of range and stack: Traceback (most recent call last): File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/reader.py", line 45, in call data = f(data, ctx) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/operators.py", line 498, in call self.autoaug_type) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 1588, in distort_image_with_autoaugment augmentation_hparams) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 1550, in build_and_apply_nas_policy tf_policies, image, bboxes) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 1498, in select_and_apply_random_policy image, bboxes = policy(image, bboxes) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 1542, in final_policy prob, bboxes_) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 1486, in _apply_func_with_prob augmented_image, augmented_bboxes = func(image, bboxes, *args) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 749, in translate_y_only_bboxes image, bboxes, prob, translate_y, func_changes_bbox, pixels, replace) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 708, in _apply_multi_bbox_augmentation_wrapper new_image, new_bboxes, prob, aug_func, func_changes_bbox, *args) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 689, in _apply_multi_bbox_augmentation idx, (image, new_bboxes) = body(idx, (image, new_bboxes)) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 687, in _images_and_bboxes[1])] File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 657, in wrapped_aug_func = lambda _image, bbox, _new_bboxes: _apply_bbox_augmentation_wrapper(_image, bbox, _new_bboxes, prob, aug_func, func_changes_bbox, *args) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 613, in _apply_bbox_augmentation_wrapper augmentation_func, *args) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 537, in _apply_bbox_augmentation augmented_bbox_content = augmentation_func(bbox_content, *args) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 861, in translate_y return unwrap(np.array(image), replace) File "/home/aistudio/work/PaddleDetection_traffic/ppdet/data/transform/autoaugment_utils.py", line 1167, in unwrap flattened_image = np.reshape(image, [-1, image_shape[2]]) IndexError: tuple index out of range