[Bug] How to properly use RRandomCrop? I could not get it.

[unique-chan]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

I tried to use dict(type='RRandomCrop') in the list named train_pipeline of my configuration *py file, but it occurs bbox mismatch error, if only one valid bounding box exists in the cropped image!

** Error message:
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/assigners/max_iou_assigner.py", line 111, in assign
overlaps = self.iou_calculator(gt_bboxes, bboxes)
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/iou_calculators/iou2d_calculator.py", line 65, in call
return bbox_overlaps(bboxes1, bboxes2, mode, is_aligned)
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/iou_calculators/iou2d_calculator.py", line 198, in bbox_overlaps
assert bboxes1.shape[:-2] == bboxes2.shape[:-2], f'{bboxes1.shape} != {bboxes2.shape}'
AssertionError: torch.Size([1, 1, 4]) != torch.Size([142110, 4])

** I cannot find the usage of RRandomCrop online. How to properly use RRandomCrop?

bboxes1.shape[:-2] == bboxes2.shape[:-2],

Reproduces the problem - code sample

Case 1)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='RResize',
img_scale=[(1536, 1152), (2340, 1728)], # 0.8x - 1.2x (1x: 1920x1440)
multiscale_mode='range'),
dict(type='RRandomFlip',
flip_ratio=[0.25, 0.25, 0.25],
direction=['horizontal', 'vertical', 'diagonal'],
version=angle_version),
dict(type='Normalize', **img_norm_cfg),
dict(type='RRandomCrop', crop_size=(800, 800), allow_negative_crop=False,
crop_type='absolute', version=angle_version),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]

Case 2)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='RRandomCrop', crop_size=(800, 800), allow_negative_crop=False,
crop_type='absolute', version=angle_version),
dict(type='RResize',
img_scale=[(800, 800)]),
dict(type='RRandomFlip',
flip_ratio=[0.25, 0.25, 0.25],
direction=['horizontal', 'vertical', 'diagonal'],
version=angle_version),
dict(type='Normalize', **img_norm_cfg),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]

Reproduces the problem - command or script

** Error message:
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/assigners/max_iou_assigner.py", line 111, in assign
overlaps = self.iou_calculator(gt_bboxes, bboxes)
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/iou_calculators/iou2d_calculator.py", line 65, in call
return bbox_overlaps(bboxes1, bboxes2, mode, is_aligned)
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/iou_calculators/iou2d_calculator.py", line 198, in bbox_overlaps
assert bboxes1.shape[:-2] == bboxes2.shape[:-2], f'{bboxes1.shape} != {bboxes2.shape}'
AssertionError: torch.Size([1, 1, 4]) != torch.Size([142110, 4])

Reproduces the problem - error message

** Error message:
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/assigners/max_iou_assigner.py", line 111, in assign
overlaps = self.iou_calculator(gt_bboxes, bboxes)
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/iou_calculators/iou2d_calculator.py", line 65, in call
return bbox_overlaps(bboxes1, bboxes2, mode, is_aligned)
File "/home/yechani9/PycharmProjects/AMOD-ExpKit/mmdetection/mmdet/core/bbox/iou_calculators/iou2d_calculator.py", line 198, in bbox_overlaps
assert bboxes1.shape[:-2] == bboxes2.shape[:-2], f'{bboxes1.shape} != {bboxes2.shape}'
AssertionError: torch.Size([1, 1, 4]) != torch.Size([142110, 4])

Additional information

No response

[unique-chan]

I got the solution! You have to fix the code of the class named RRandomCrop in mmrotate/mmrotate/datasets/pipelines/transforms.py as follows:

What to solve: I tried to use dict(type='RRandomCrop') in the list named train_pipeline of my configuration *py file, but it occurs bbox mismatch error, if only one valid bounding box exists in the cropped image!

How to solve: In RRandomCrop/_crop_data(), valid_inds should be 1-dimensional, but sometimes valid_inds become 0-dimensional, when only one valid bounding box exists in the cropped image, which occurs bbox shape mismatch error! Therefore, I added "valid_inds = np.atleast_1d(valid_inds)" in _crop_data(), in order to avoid such error!

@ROTATED_PIPELINES.register_module()
class RRandomCrop(RandomCrop):
    """Random crop the image & bboxes.

    The absolute `crop_size` is sampled based on `crop_type` and `image_size`,
    then the cropped results are generated.

    Args:
        crop_size (tuple): The relative ratio or absolute pixels of
            height and width.
        crop_type (str, optional): one of "relative_range", "relative",
            "absolute", "absolute_range". "relative" randomly crops
            (h * crop_size[0], w * crop_size[1]) part from an input of size
            (h, w). "relative_range" uniformly samples relative crop size from
            range [crop_size[0], 1] and [crop_size[1], 1] for height and width
            respectively. "absolute" crops from an input with absolute size
            (crop_size[0], crop_size[1]). "absolute_range" uniformly samples
            crop_h in range [crop_size[0], min(h, crop_size[1])] and crop_w
            in range [crop_size[0], min(w, crop_size[1])]. Default "absolute".
        allow_negative_crop (bool, optional): Whether to allow a crop that does
            not contain any bbox area. Default False.
        iof_thr (float): The minimal iof between a object and window.
            Defaults to 0.7.

    Note:
        - If the image is smaller than the absolute crop size, return the
            original image.
        - The keys for bboxes, labels must be aligned. That is, `gt_bboxes`
          corresponds to `gt_labels`, and `gt_bboxes_ignore` corresponds to
          `gt_labels_ignore`.
        - If the crop does not contain any gt-bbox region and
          `allow_negative_crop` is set to False, skip this image.
    """

    def __init__(self,
                 crop_size,
                 crop_type='absolute',
                 allow_negative_crop=False,
                 iof_thr=0.7,
                 version='oc'):
        self.version = version
        self.iof_thr = iof_thr
        super(RRandomCrop, self).__init__(crop_size, crop_type,
                                          allow_negative_crop)

    def _crop_data(self, results, crop_size, allow_negative_crop):
        """Function to randomly crop images, bounding boxes.

        Args:
            results (dict): Result dict from loading pipeline.
            crop_size (tuple): Expected absolute size after cropping, (h, w).
            allow_negative_crop (bool): Whether to allow a crop that does not
                contain any bbox area. Default to False.

        Returns:
            dict: Randomly cropped results, 'img_shape' key in result dict is
                updated according to crop size.
        """
        assert crop_size[0] > 0 and crop_size[1] > 0
        for key in results.get('bbox_fields', []):
            assert results[key].shape[-1] % 5 == 0

        for key in results.get('img_fields', ['img']):
            img = results[key] #img
            margin_h = max(img.shape[0] - crop_size[0], 0)
            margin_w = max(img.shape[1] - crop_size[1], 0)
            offset_h = np.random.randint(0, margin_h + 1)
            offset_w = np.random.randint(0, margin_w + 1)
            crop_y1, crop_y2 = offset_h, offset_h + crop_size[0]
            crop_x1, crop_x2 = offset_w, offset_w + crop_size[1]

            # crop the image
            img = img[crop_y1:crop_y2, crop_x1:crop_x2, ...]
            img_shape = img.shape
            results[key] = img
        # results['img_shape'] = img_shape # e.g. (800, 800, 3)
        # I added ->
        results['img_shape'] = img_shape[:2] # e.g. (800, 800)

        height, width, _ = img_shape

        # crop bboxes accordingly and clip to the image boundary
        for key in results.get('bbox_fields', []):
            # e.g. gt_bboxes and gt_bboxes_ignore
            bbox_offset = np.array([offset_w, offset_h, 0, 0, 0],
                                   dtype=np.float32)
            bboxes = results[key] - bbox_offset
            windows = np.array([width / 2, height / 2, width, height, 0],
                               dtype=np.float32).reshape(-1, 5)

            valid_inds = box_iou_rotated(
                torch.tensor(bboxes), torch.tensor(windows),
                mode='iof').numpy().squeeze() > self.iof_thr

            # If the crop does not contain any gt-bbox area and
            # allow_negative_crop is False, skip this image.
            if (key == 'gt_bboxes' and not valid_inds.any()
                    and not allow_negative_crop):
                return None

            # I added! ->
            valid_inds = np.atleast_1d(valid_inds)

            results[key] = bboxes[valid_inds, :]
            # label fields. e.g. gt_labels and gt_labels_ignore
            label_key = self.bbox2label.get(key)
            if label_key in results:
                results[label_key] = results[label_key][valid_inds]
        return results

I've made a pull request about this! Please merge the fixed code.

[unique-chan]

#1083