[Bug] RLE annotation will be ignored in training, which make loss be 0

[CescMessi]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (master) or latest version (3.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmdetection

Environment

sys.platform: linux
Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
CUDA available: True
numpy_random_seed: 2147483648
GPU 0,1: Tesla V100-PCIE-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.12.0
PyTorch compiling details: PyTorch built with:

• GCC 9.3
• C++ Version: 201402
• Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.3
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.3.2 (built against CUDA 11.5)
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.13.0
OpenCV: 4.6.0
MMEngine: 0.1.0
MMDetection: 3.0.0rc1+e634592

Reproduces the problem - code sample

Code in mmdet/datasets/transforms/loading.py#L354

          elif isinstance(gt_mask, dict) and \
                  not (gt_mask.get('counts') is not None and
                       gt_mask.get('size') is not None and
                       isinstance(gt_mask['counts'], (list, str))):
              # if gt_mask is a dict, it should include `counts` and `size`,
              # so that `BitmapMasks` can uncompressed RLE
              instance['ignore_flag'] = 1
              gt_mask = [np.zeros(6)]

The code will make RLE gt be ignored. In my dataset, all annotation are labeled in RLE, so there is no gt instance in training. And the log shows that all the loss are 0.

When delete the line 354, loss will be ok.

Reproduces the problem - command or script

A placeholder for the command.

Reproduces the problem - error message

A placeholder for trackback.

Additional information

No response

[RangiLyu]

Thanks for your bug report! We are going to fix this in the next release version.

[BIGWangYuDong]

What does your RLE read structure look like?

[BIGWangYuDong]

The logic of these lines:

mmdetection/mmdet/datasets/transforms/loading.py

Lines 348 to 355 in 6e9cfd4

	elif isinstance(gt_mask, dict) and \
	not (gt_mask.get('counts') is not None and
	gt_mask.get('size') is not None and
	isinstance(gt_mask['counts'], (list, str))):
	# if gt_mask is a dict, it should include `counts` and `size`,
	# so that `BitmapMasks` can uncompressed RLE
	instance['ignore_flag'] = 1
	gt_mask = [np.zeros(6)]

Is to make sure the mask is in a format that can be handled.

If your mask is directly an RLE, it will not not enter this conditional code.

[CescMessi]

The logic of these lines:

mmdetection/mmdet/datasets/transforms/loading.py

Lines 348 to 355 in 6e9cfd4

	elif isinstance(gt_mask, dict) and \
	not (gt_mask.get('counts') is not None and
	gt_mask.get('size') is not None and
	isinstance(gt_mask['counts'], (list, str))):
	# if gt_mask is a dict, it should include `counts` and `size`,
	# so that `BitmapMasks` can uncompressed RLE
	instance['ignore_flag'] = 1
	gt_mask = [np.zeros(6)]

Is to make sure the mask is in a format that can be handled.

If your mask is directly an RLE, it will not not enter this conditional code.

The RLE annotation is like that:

{"id": 0, "image_id": 0, "category_id": 1, "segmentation": {"counts": [362609, 21, 939, 61, 899, 101, 859, 141, 819, 181, 779, 221, 739, 261, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 735, 265, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 736, 264, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 737, 263, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 263, 737, 263, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 738, 262, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 739, 261, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 260, 740, 237, 763, 201, 799, 166, 834, 130, 870, 94, 906, 58, 942, 23, 361604], "size": [1000, 1000]}, "area": 70905.71102418478, "bbox": [362.0, 365.0, 277.1, 267.9], "iscrowd": 0},

And this is my vscode screenshot:

[BIGWangYuDong]

You are right, it seems this is a logic error, if you have already fixed this error, PR is welcomed, else, we will fix ASAP

[hansaskov]

I am having the same issue. Any update?

[hansaskov]

After some investigation i can confirm that you must enable the poly2mask flag to convert the RLE format.

Despite it's misleading name, it will convert RLE to a BitmapMasks.

The below code is also correct, but slightly confusing.

elif isinstance(gt_mask, dict) and \
        not (gt_mask.get('counts') is not None and
                gt_mask.get('size') is not None and
                isinstance(gt_mask['counts'], (list, str))):
    # if gt_mask is a dict, it should include `counts` and `size`,
    # so that `BitmapMasks` can uncompressed RLE
    instance['ignore_flag'] = 1
    gt_mask = [np.zeros(6)]

It says that if the input is a dictionary and not in the RLE format, set the ignore flag to 1

[maritum]

@hansaskov thanks for your comments, I also fixed loading GT in RLE format. I am curious if the rest of training was fine for you. My model is struggling with mask predictions (bbox are alright). The prediction masks are all False

[GewelsJI]

Hey, guys,

Any updates on supporting RLE-style training with mmdetection?