在使用BDD100K数据集进行训练的时候，单卡训练没有问题，多卡训练出现了 assert (boxes1[:, 2:] >= boxes1[:, :2]).all()错误，请问是什么原因啊？

[damengyua]

这是报错：“Traceback (most recent call last):
File "/data/jsj-faster/MeMOTR-main/main.py", line 124, in
main(config=merged_config)
File "/data/jsj-faster/MeMOTR-main/main.py", line 107, in main
train(config=config)
File "/data/jsj-faster/MeMOTR-main/train_engine.py", line 129, in train
train_one_epoch(
File "/data/jsj-faster/MeMOTR-main/train_engine.py", line 214, in train_one_epoch
previous_tracks, new_tracks, unmatched_dets = criterion.process_single_frame(
File "/data/jsj-faster/MeMOTR-main/models/criterion.py", line 197, in process_single_frame
matcher_res = self.matcher(outputs=detection_res, targets=untracked_gt_trackinstances, use_focal=True)
File "/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/data/jsj-faster/MeMOTR-main/models/matcher.py", line 117, in forward
cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox),
File "/data/jsj-faster/MeMOTR-main/utils/box_ops.py", line 91, in generalized_box_iou
assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
AssertionError
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 55800 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 55801 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 55802 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 55803 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 55804 closing signal SIGTERM
”

[HELLORPG]

这个是在训练一开始出现的还是在一定时间之后出现的？这个错误是因为出现了非法的 bounding boxes 坐标，应该和单卡/多卡没有关系。

[damengyua]

这个是在训练一开始出现的还是在一定时间之后出现的？这个错误是因为出现了非法的 bounding boxes 坐标，应该和单卡/多卡没有关系。
在训练的一开始。我首先是使用单卡看看能不能跑起来，发现能运行后，重新使用多卡来运行。在我使用6张V100的时候，报上面的那个错误，但是我使用5张卡的时候，没有报错。像这样：“(memotr) jsj-faster@GPU-v100:~/MeMOTR-main$ python -m torch.distributed.run --nproc_per_node=5 main.py --use-distributed --config-path ./configs/train_bdd100k.yaml --data-root /data/jsj-faster/MeMOTR-main/DATADIR
WARNING:main:

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

Configs: {'GIT_VERSION': None, 'MODE': 'train', 'CONFIG_PATH': './configs/train_bdd100k.yaml', 'VISUALIZE': False, 'AVAILABLE_GPUS': '0,1,2,3,4,5', 'DEVICE': 'cuda', 'OUTPUTS_DIR': './outputs/memotr_bdd100k/', 'USE_DISTRIBUTED': True, 'USE_CHECKPOINT': False, 'CHECKPOINT_LEVEL': 2, 'RESUME': None, 'RESUME_SCHEDULER': False, 'MULTI_CHECKPOINT': False, 'SUBMIT_DIR': None, 'SUBMIT_MODEL': None, 'SUBMIT_DATA_SPLIT': 'val', 'DET_SCORE_THRESH': 0.5, 'TRACK_SCORE_THRESH': 0.5, 'RESULT_SCORE_THRESH': 0.5, 'MISS_TOLERANCE': 10, 'USE_MOTION': False, 'MOTION_MIN_LENGTH': 3, 'MOTION_MAX_LENGTH': 5, 'MOTION_LAMBDA': 0.5, 'EVAL_DIR': None, 'EVAL_MODE': 'specific', 'EVAL_MODEL': None, 'EVAL_PORT': None, 'EVAL_THREADS': 1, 'EVAL_DATA_SPLIT': 'val', 'DATASET': 'BDD100K', 'DATA_ROOT': '/data/jsj-faster/MeMOTR-main/DATADIR', 'DATA_PATH': None, 'NUM_WORKERS': 8, 'BATCH_SIZE': 1, 'ACCUMULATION_STEPS': 1, 'BACKBONE': 'resnet50', 'HIDDEN_DIM': 256, 'FFN_DIM': 2048, 'NUM_FEATURE_LEVELS': 4, 'NUM_HEADS': 8, 'NUM_ENC_POINTS': 4, 'NUM_DEC_POINTS': 4, 'NUM_ENC_LAYERS': 6, 'NUM_DEC_LAYERS': 6, 'MERGE_DET_TRACK_LAYER': 1, 'ACTIVATION': 'ReLU', 'RETURN_INTER_DEC': True, 'EXTRA_TRACK_ATTN': False, 'AUX_LOSS': True, 'USE_DAB': True, 'UPDATE_THRESH': 0.5, 'LONG_MEMORY_LAMBDA': 0.01, 'PRETRAINED_MODEL': 'dab_deformable_detr.pth', 'SAMPLE_STEPS': [6, 10], 'SAMPLE_LENGTHS': [2, 3, 4], 'SAMPLE_MODES': ['random_interval'], 'SAMPLE_INTERVALS': [4, 4, 4], 'SEED': 42, 'EPOCHS': 1, 'ONLY_TRAIN_QUERY_UPDATER_AFTER': 1000, 'DROPOUT': 0.0, 'NUM_DET_QUERIES': 300, 'TP_DROP_RATE': 0.0, 'FP_INSERT_RATE': 0.0, 'LR': 2e-05, 'LR_BACKBONE': 2e-06, 'LR_POINTS': 1e-06, 'WEIGHT_DECAY': 0.0002, 'CLIP_MAX_NORM': 0.1, 'LR_SCHEDULER': 'MultiStep', 'LR_DROP_RATE': 0.1, 'LR_DROP_MILESTONES': [12], 'MATCH_COST_CLASS': 2, 'MATCH_COST_BBOX': 5, 'MATCH_COST_GIOU': 2, 'LOSS_WEIGHT_FOCAL': 2, 'LOSS_WEIGHT_L1': 5, 'LOSS_WEIGHT_GIOU': 2, 'AUX_LOSS_WEIGHT': [1.0, 1.0, 1.0, 1.0, 1.0]}
Pretrained model is loaded, there are 0 parameters droped and 32 parameters unloaded, set 'show details' True to see more details.
[Epoch 0] lr=[{'lr_backbone': 2e-06}, {'lr_points': 1e-06}, {'lr_query_updater': 2e-05}, {'lr': 2e-05}]
/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[Epoch=0, Iter=0, 2.41s/iter, 0/54344 iters, rest time: 2186 min, Max Memory=4797MB] loss = 8.6317 (8.6317); frame0_box_l1_loss = 0.0471 (0.0471); frame0_box_giou_loss = 0.3033 (0.3033); frame0_label_focal_loss = 0.1368 (0.1368); frame1_box_l1_loss = 0.0953 (0.0953); frame1_box_giou_loss = 0.5505 (0.5505); frame1_label_focal_loss = 0.1331 (0.1331);
[Epoch=0, Iter=100, 0.60s/iter, 100/54344 iters, rest time: 546 min, Max Memory=7083MB] loss = 8.2187 (8.2228); frame0_box_l1_loss = 0.0356 (0.0357); frame0_box_giou_loss = 0.2931 (0.2932); frame0_label_focal_loss = 0.1248 (0.1249); frame1_box_l1_loss = 0.0801 (0.0803); frame1_box_giou_loss = 0.5445 (0.5446); frame1_label_focal_loss = 0.1264 (0.1265);
[Epoch=0, Iter=200, 0.59s/iter, 200/54344 iters, rest time: 534 min, Max Memory=7083MB] loss = 7.4809 (7.8537); frame0_box_l1_loss = 0.0369 (0.0363); frame0_box_giou_loss = 0.2760 (0.2847); frame0_label_focal_loss = 0.0908 (0.1079); frame1_box_l1_loss = 0.0765 (0.0784); frame1_box_giou_loss = 0.4879 (0.5164); frame1_label_focal_loss = 0.0937 (0.1102);
[Epoch=0, Iter=300, 0.59s/iter, 300/54344 iters, rest time: 530 min, Max Memory=7083MB] loss = 7.1223 (7.6107); frame0_box_l1_loss = 0.0347 (0.0358); frame0_box_giou_loss = 0.2832 (0.2842); frame0_label_focal_loss = 0.0608 (0.0923); frame1_box_l1_loss = 0.0731 (0.0766); frame1_box_giou_loss = 0.4868 (0.5066); frame1_label_focal_loss = 0.0641 (0.0948);
[Epoch=0, Iter=400, 0.59s/iter, 400/54344 iters, rest time: 528 min, Max Memory=7083MB] loss = 6.8760 (7.4275); frame0_box_l1_loss = 0.0368 (0.0360); frame0_box_giou_loss = 0.2751 (0.2819); frame0_label_focal_loss = 0.0532 (0.0825); frame1_box_l1_loss = 0.0693 (0.0748); frame1_box_giou_loss = 0.4837 (0.5009); frame1_label_focal_loss = 0.0562 (0.0852);
[Epoch=0, Iter=500, 0.58s/iter, 500/54344 iters, rest time: 522 min, Max Memory=7083MB] loss = 6.9273 (7.3276); frame0_box_l1_loss = 0.0415 (0.0371); frame0_box_giou_loss = 0.2844 (0.2824); frame0_label_focal_loss = 0.0493 (0.0759); frame1_box_l1_loss = 0.0730 (0.0745); frame1_box_giou_loss = 0.4762 (0.4959); frame1_label_focal_loss = 0.0524 (0.0787);
”

[HELLORPG]

如果不是一开始就出现错误的话，那和多卡运行时没有关系。应该是某一步得到了一个非法的数据。

我看报错的是 boxes1，如果没记错的话，这个应该是模型预测的结果。也就是说模型预测出了一个非法的 bounding boxes。可以尝试在这个断言语句前面打印一下输出的 boxes，大概率是 nan。

[damengyua]

该是模型预

是的，之前打印出来的结果就是nan，不知道是什么原因导致的？这是打印结果:“[Epoch=0, Iter=0, 2.46s/iter, 0/45287 iters, rest time: 1854 min, Max Memory=4797MB] loss = 9.0460 (9.0460); frame0_box_l1_loss = 0.0480 (0.0480); frame0_box_giou_loss = 0.3086 (0.3086); frame0_label_focal_loss = 0.1374 (0.1374); frame1_box_l1_loss = 0.1030 (0.1030); frame1_box_giou_loss = 0.5797 (0.5797); frame1_label_focal_loss = 0.1341 (0.1341);
[DEBUG] boxes1 有 300 个非法 box
索引: tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,
42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69,
70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83,
84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97,
98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111,
112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139,
140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153,
154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167,
168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195,
196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209,
210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223,
224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237,
238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251,
252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265,
266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279,
280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293,
294, 295, 296, 297, 298, 299], device='cuda:5')
值: tensor([[nan, nan, nan, nan],
[nan, nan, nan, nan],
[nan, nan, nan, nan],
...,
[nan, nan, nan, nan],
[nan, nan, nan, nan],
[nan, nan, nan, nan]], device='cuda:5')
Traceback (most recent call last):
File "/data/jsj-faster/MeMOTR-main/main.py", line 124, in
main(config=merged_config)
File "/data/jsj-faster/MeMOTR-main/main.py", line 107, in main
train(config=config)
File "/data/jsj-faster/MeMOTR-main/train_engine.py", line 129, in train
train_one_epoch(
File "/data/jsj-faster/MeMOTR-main/train_engine.py", line 214, in train_one_epoch
previous_tracks, new_tracks, unmatched_dets = criterion.process_single_frame(
File "/data/jsj-faster/MeMOTR-main/models/criterion.py", line 197, in process_single_frame
matcher_res = self.matcher(outputs=detection_res, targets=untracked_gt_trackinstances, use_focal=True)
File "/data/jsj-faster/anaconda3/envs/memotr/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/data/jsj-faster/MeMOTR-main/models/matcher.py", line 117, in forward
cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox),
File "/data/jsj-faster/MeMOTR-main/utils/box_ops.py", line 91, in generalized_box_iou
assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
AssertionError
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 70577 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 70578 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 70579 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 70580 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 70581 closing signal SIGTERM
”

[HELLORPG]

这个主要考虑是训练问题，可能训练崩溃了，也有可能机器（例如显卡）有一定的问题。如果你更少的 GPU 都不会报错的话，优先考虑靠后的 GPU 有问题；如果不是 GPU 的问题，应该就是训练的问题。但是如果和我们训练配置都一样，并且这个错误稳定复现的话，我觉得后者的概率应该不大才对。