About MMDetWandbHook

[chunxt]

when i use WandbLoggerHook, it's ok! Replace by MMDetWandbHook, the error likes this.
Traceback (most recent call last):
File "E:/mm250/tools/trains/train.py", line 237, in
main()
File "E:/mm250/tools/trains/train.py", line 226, in main
train_detector(
File "E:\mm250\mmdet\apis\train.py", line 244, in train_detector
runner.run(data_loaders, cfg.workflow)
File "E:\ProgramData\Anaconda3\envs\mmdet\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 111, in run
self.call_hook('before_run')
File "E:\ProgramData\Anaconda3\envs\mmdet\lib\site-packages\mmcv\runner\base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "E:\ProgramData\Anaconda3\envs\mmdet\lib\site-packages\mmcv\runner\dist_utils.py", line 135, in wrapper
return func(*args, **kwargs)
File "E:\mm250\mmdet\core\hook\wandblogger_hook.py", line 201, in before_run
self._add_ground_truth(runner)
File "E:\mm250\mmdet\core\hook\wandblogger_hook.py", line 380, in _add_ground_truth
wandb_masks = self._get_wandb_masks(
File "E:\mm250\mmdet\core\hook\wandblogger_hook.py", line 536, in _get_wandb_masks
mask = polygon_to_bitmap(mask, height, width)
File "E:\mm250\mmdet\core\mask\structures.py", line 1069, in polygon_to_bitmap
rles = maskUtils.frPyObjects(polygons, height, width)
File "pycocotools/_mask.pyx", line 293, in pycocotools._mask.frPyObjects
IndexError: list index out of range
wandb: Waiting for W&B process to finish... (failed 1). Press Ctrl-C to abort syncing.

[BIGWangYuDong]

what command and config did you use? Please use issue template

[chunxt]

what command and config did you use? Please use issue template

1. I have searched related issues but cannot get the expected help.
2. I have read the FAQ documentation but cannot get the expected help.
3. The bug has not been fixed in the latest version.

Describe the bug
WandbLoggerHook is used in log_config, but it has some errors.
rles = maskUtils.frPyObjects(polygons, height, width)
File "pycocotools/_mask.pyx", line 293, in pycocotools._mask.frPyObjects
IndexError: list index out of range

Reproduction

1. What command or script did you run?
   python train.py --config /public/home/cx/mm251/tools/config/csp1-ms-cos.py --work-dir exp/ms-cos

A placeholder for the command.

2. Did you make any modifications on the code or config? Did you understand what you have modified?
3. What dataset did you use?

Environment

1. Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here.

sys.platform: linux
Python: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0]
CUDA available: True
GPU 0: Tesla V100-PCIE-32GB
CUDA_HOME: /usr/local/cuda-10.1/
GCC: gcc (GCC) 5.5.0
PyTorch: 1.8.1+cu101
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 10.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
• CuDNN 7.6.3
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.1, CUDNN_VERSION=7.6.3, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.1+cu101
OpenCV: 4.6.0
MMCV: 1.5.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMDetection: 2.25.1+3b72b12
3. You may add addition that may be helpful for locating the problem, such as

• How you installed PyTorch [e.g., pip, conda, source]
• Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)
  myconfig is here!
  conf.txt

Error traceback
If applicable, paste the error trackback here.
Errors are shown above.

A placeholder for trackback.

Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

[BIGWangYuDong]

Seems this is a pycocotools error, have you checked your data annotations?

[chunxt]

Seems this is a pycocotools error, have you checked your data annotations?

yeah,n I have checked it, and everything is ok when i remove the mmdetWandbHook in log_config, or replace it by 'WandbLoogerHook'.
log_config = dict(interval=50, hooks=[
dict(type='TextLoggerHook'),
dict(
type='MMDetWandbHook', #WandbLoogerHook is ok
init_kwargs=dict(entity='cxt5', project='dw2ms'),
interval=50,
log_checkpoint=True,
log_checkpoint_metadata=True
)
])

[ayulockin]

Hey @chunxt will you be able to share a colab or a GitHub gist to reproduce the issue?

[cornzyblack]

I also experience the same issue

[echonax07]

I am also facing a similar issue, which got solved by replacing 'MMDetWandbHook' with 'WandbLoogerHook'. Here is the error log

##################
"
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run desert-leaf-12
wandb: ⭐️ View project at https://wandb.ai/mmwhale/mmwhale
wandb: 🚀 View run at https://wandb.ai/mmwhale/mmwhale/runs/1isubsvx
2022-09-30 15:29:27,819 - mmdet - WARNING - The num_eval_images (74) is greater than the total number of validation samples (74). The complete validation dataset will be logged.
Traceback (most recent call last):
File "tools/train.py", line 253, in
main()
File "tools/train.py", line 242, in main
train_detector(
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmdet/apis/train.py", line 244, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 108, in run
self.call_hook('before_run')
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmcv/runner/dist_utils.py", line 94, in wrapper
return func(*args, **kwargs)
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmdet/core/hook/wandblogger_hook.py", line 201, in before_run
self._add_ground_truth(runner)
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmdet/core/hook/wandblogger_hook.py", line 380, in _add_ground_truth
wandb_masks = self._get_wandb_masks(
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmdet/core/hook/wandblogger_hook.py", line 536, in _get_wandb_masks
mask = polygon_to_bitmap(mask, height, width)
File "/home/m32patel/mmdet/lib/python3.8/site-packages/mmdet/core/mask/structures.py", line 1069, in polygon_to_bitmap
rles = maskUtils.frPyObjects(polygons, height, width)
File "pycocotools/_mask.pyx", line 308, in pycocotools._mask.frPyObjects
Exception: input type is not supported.
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.

"

[echonax07]

@ayulockin
@BIGWangYuDong

I see that when your annotation doesn't have any segmentation masks, this error is produced.

I think this has something to do with 'MMDetWandbHook' hook is trying to get segmentation masks, but as the masks are not available in the annotation; pycocotools._mask.frPyObjects throws the error Exception: input type is not supported. (see here)

It's not supposed to go inside the if loop (line 379) but still it goes in somehow (i checked it using print statement)

I request the wandb and mmdetection to solve this issue.