Created by: kriswangjie

Ubuntu机器上在训练网络时，我用的VOC格式的数据集，一直卡在一条log这不往下走，log的内容是“2020-09-04 19:21:32,589-WARNING: recv endsignal from outq with errmsg[consumer[consumer-5cf-4] failed to map with error:[target_ not in samples]] ”。我用的是自定义的loss去掉了obj_loss输出中（5+num_class）改成了（4+num_class）以及更改了合适的yolo_box后出现现在的问题，请问如何才能解决这个问题？我的yml如下:

architecture: YOLOv3_tiny
use_gpu: true
max_iters: 500000
log_smooth_window: 20
save_dir: output_v2_sar
snapshot_iter: 500
metric: VOC
map_type: 11point
pretrain_weights:
weights:
num_classes: 1
use_fine_grained_loss: true

我们查看有相关的配置文件确认如下：

architecture: YOLOv3_tiny
use_gpu: true
max_iters: 500000
log_smooth_window: 20
save_dir: output_v2_sar
snapshot_iter: 500
metric: VOC
map_type: 11point
pretrain_weights:
weights:
num_classes: 1
use_fine_grained_loss: true

TrainReader:
inputs_def:
image_shape: [1, -1, -1]
fields: ['image', 'gt_bbox', 'gt_class']
num_max_boxes: 50
use_fine_grained_loss: true

但是依然是这个错误。 在debug过程中发现是在如下语句中报错：

train-> train_stats = TrainingStats(cfg.log_smooth_window, train_keys)
	    train_loader.start()

我们在class Gt2YoloTarget（）类中做了如下修改：

      for sample in samples:
            # im, gt_bbox, gt_class, gt_score = sample
            im = sample['image']
            gt_bbox = sample['gt_bbox']
            gt_class = sample['gt_class']
            # gt_score = sample['gt_score']
            for i, (
                    mask, downsample_ratio
            ) in enumerate(zip(self.anchor_masks, self.downsample_ratios)):
                grid_h = int(h / downsample_ratio)
                grid_w = int(w / downsample_ratio)
                target = np.zeros(
                    (len(mask), 6 + self.num_classes, grid_h, grid_w),
                    dtype=np.float32)
                for b in range(gt_bbox.shape[0]):
                    gx, gy, gw, gh = gt_bbox[b, :]
                    cls = gt_class[b]
                    # score = gt_score[b]
                    if gw <= 0. or gh <= 0. :
                        continue

                    # find best match anchor index
                    best_iou = 0.
                    best_idx = -1
                    for an_idx in range(an_hw.shape[0]):
                        iou = jaccard_overlap(
                            [0., 0., gw, gh],
                            [0., 0., an_hw[an_idx, 0], an_hw[an_idx, 1]])
                        if iou > best_iou:
                            best_iou = iou
                            best_idx = an_idx

                    gi = int(gx * grid_w)
                    gj = int(gy * grid_h)

                    # gtbox should be regresed in this layes if best match 
                    # anchor index in anchor mask of this layer
                    if best_idx in mask:
                        best_n = mask.index(best_idx)

                        # x, y, w, h, scale
                        target[best_n, 0, gj, gi] = gx * grid_w - gi
                        target[best_n, 1, gj, gi] = gy * grid_h - gj
                        target[best_n, 2, gj, gi] = np.log(
                            gw * w / self.anchors[best_idx][0])
                        target[best_n, 3, gj, gi] = np.log(
                            gh * h / self.anchors[best_idx][1])
                        target[best_n, 4, gj, gi] = 2.0 - gw * gh

                        # objectness record gt_score
                        # target[best_n, 5, gj, gi] = score

                        # classification
                        target[best_n, 4 + cls, gj, gi] = 1.

                    # For non-matched anchors, calculate the target if the iou 
                    # between anchor and gt is larger than iou_thresh
                    if self.iou_thresh < 1:
                        for idx, mask_i in enumerate(mask):
                            if mask_i == best_idx: continue
                            iou = jaccard_overlap(
                                [0., 0., gw, gh],
                                [0., 0., an_hw[mask_i, 0], an_hw[mask_i, 1]])
                            if iou > self.iou_thresh:
                                # x, y, w, h, scale
                                target[idx, 0, gj, gi] = gx * grid_w - gi
                                target[idx, 1, gj, gi] = gy * grid_h - gj
                                target[idx, 2, gj, gi] = np.log(
                                    gw * w / self.anchors[mask_i][0])
                                target[idx, 3, gj, gi] = np.log(
                                    gh * h / self.anchors[mask_i][1])
                                target[idx, 4, gj, gi] = 2.0 - gw * gh

                                # objectness record gt_score
                                # target[idx, 5, gj, gi] = score

                                # classification
                                target[idx, 4 + cls, gj, gi] = 1.
                sample['target{}'.format(i)] = target

但是依然没有解决这个问题