CRM-Siam训练问题

[linananananana]

您好！请问在解冻backbone后，开始第11个epoch，会出现以下情况：
[2025-04-02 20:17:47,978-rk0-train.py#247] epoch: 11
[2025-04-02 20:17:47,978-rk0-train.py#251] epoch 11 lr 4.826744322208125e-06
[2025-04-02 20:17:47,978-rk0-train.py#251] epoch 11 lr 0.0004826744322208124
[2025-04-02 20:17:47,979-rk0-train.py#251] epoch 11 lr 0.0004826744322208124
[2025-04-02 20:17:47,979-rk0-train.py#251] epoch 11 lr 0.0004826744322208124
[2025-04-02 20:18:11,803-rk0-train.py#300] Epoch: [11][20/642] lr: 0.000483
batch_time: 0.543413 (1.253038) data_time: 0.082757 (0.569095)
cls_loss: 1581062.750000 (89252849.141115) loc_loss: 0.999936 (0.962544)
total_loss: 1581063.750000 (89252850.067277)
[2025-04-02 20:18:11,803-rk0-log_helper.py#105] Progress: 6440 / 12840 [50%], Speed: 1.253 s/iter, ETA 0:02:13 (D:H:M)

[2025-04-02 20:18:34,711-rk0-train.py#300] Epoch: [11][40/642] lr: 0.000483
batch_time: 0.464469 (1.305685) data_time: 0.000000 (0.521625)
cls_loss: 4303667200.000000 (219855600.983749) loc_loss: 0.998354 (0.957559)
total_loss: 4303667200.000000 (219855601.863008)
[2025-04-02 20:18:34,711-rk0-log_helper.py#105] Progress: 6460 / 12840 [50%], Speed: 1.306 s/iter, ETA 0:02:18 (D:H:M)

[2025-04-02 20:18:59,179-rk0-train.py#300] Epoch: [11][60/642] lr: 0.000483
batch_time: 0.462952 (1.358881) data_time: 0.000000 (0.453902)
cls_loss: 1192478.000000 (530384016.806605) loc_loss: 0.999114 (0.956494)
total_loss: 1192479.000000 (530384017.654972)
[2025-04-02 20:18:59,180-rk0-log_helper.py#105] Progress: 6480 / 12840 [50%], Speed: 1.359 s/iter, ETA 0:02:24 (D:H:M)

WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.

请问应该怎么解决呢

[xyl-507]

应该是你的训练图片有问题

…

---- 回复的原邮件 ---- | 发件人 | ***@***.***> | | 日期 | 2025年04月03日 11:07 | | 收件人 | ***@***.***> | | 抄送至 | ***@***.***> | | 主题 | [xyl-507/CRM] CRM-Siam训练问题 (Issue #12) | 您好！请问在解冻backbone后，开始第11个epoch，会出现以下情况： [2025-04-02 20:17:47,978-rk0-train.py#247] epoch: 11 [2025-04-02 20:17:47,978-rk0-train.py#251] epoch 11 lr 4.826744322208125e-06 [2025-04-02 20:17:47,978-rk0-train.py#251] epoch 11 lr 0.0004826744322208124 [2025-04-02 20:17:47,979-rk0-train.py#251] epoch 11 lr 0.0004826744322208124 [2025-04-02 20:17:47,979-rk0-train.py#251] epoch 11 lr 0.0004826744322208124 [2025-04-02 20:18:11,803-rk0-train.py#300] Epoch: [11][20/642] lr: 0.000483 batch_time: 0.543413 (1.253038) data_time: 0.082757 (0.569095) cls_loss: 1581062.750000 (89252849.141115) loc_loss: 0.999936 (0.962544) total_loss: 1581063.750000 (89252850.067277) [2025-04-02 20:18:11,803-rk0-log_helper.py#105] Progress: 6440 / 12840 [50%], Speed: 1.253 s/iter, ETA 0:02:13 (D:H:M) [2025-04-02 20:18:34,711-rk0-train.py#300] Epoch: [11][40/642] lr: 0.000483 batch_time: 0.464469 (1.305685) data_time: 0.000000 (0.521625) cls_loss: 4303667200.000000 (219855600.983749) loc_loss: 0.998354 (0.957559) total_loss: 4303667200.000000 (219855601.863008) [2025-04-02 20:18:34,711-rk0-log_helper.py#105] Progress: 6460 / 12840 [50%], Speed: 1.306 s/iter, ETA 0:02:18 (D:H:M) [2025-04-02 20:18:59,179-rk0-train.py#300] Epoch: [11][60/642] lr: 0.000483 batch_time: 0.462952 (1.358881) data_time: 0.000000 (0.453902) cls_loss: 1192478.000000 (530384016.806605) loc_loss: 0.999114 (0.956494) total_loss: 1192479.000000 (530384017.654972) [2025-04-02 20:18:59,180-rk0-log_helper.py#105] Progress: 6480 / 12840 [50%], Speed: 1.359 s/iter, ETA 0:02:24 (D:H:M) WARNING:root:NaN or Inf found in input tensor. WARNING:root:NaN or Inf found in input tensor. WARNING:root:NaN or Inf found in input tensor. 请问应该怎么解决呢 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> linananananana created an issue (xyl-507/CRM#12) 您好！请问在解冻backbone后，开始第11个epoch，会出现以下情况： [2025-04-02 20:17:47,978-rk0-train.py#247] epoch: 11 [2025-04-02 20:17:47,978-rk0-train.py#251] epoch 11 lr 4.826744322208125e-06 [2025-04-02 20:17:47,978-rk0-train.py#251] epoch 11 lr 0.0004826744322208124 [2025-04-02 20:17:47,979-rk0-train.py#251] epoch 11 lr 0.0004826744322208124 [2025-04-02 20:17:47,979-rk0-train.py#251] epoch 11 lr 0.0004826744322208124 [2025-04-02 20:18:11,803-rk0-train.py#300] Epoch: [11][20/642] lr: 0.000483 batch_time: 0.543413 (1.253038) data_time: 0.082757 (0.569095) cls_loss: 1581062.750000 (89252849.141115) loc_loss: 0.999936 (0.962544) total_loss: 1581063.750000 (89252850.067277) [2025-04-02 20:18:11,803-rk0-log_helper.py#105] Progress: 6440 / 12840 [50%], Speed: 1.253 s/iter, ETA 0:02:13 (D:H:M) [2025-04-02 20:18:34,711-rk0-train.py#300] Epoch: [11][40/642] lr: 0.000483 batch_time: 0.464469 (1.305685) data_time: 0.000000 (0.521625) cls_loss: 4303667200.000000 (219855600.983749) loc_loss: 0.998354 (0.957559) total_loss: 4303667200.000000 (219855601.863008) [2025-04-02 20:18:34,711-rk0-log_helper.py#105] Progress: 6460 / 12840 [50%], Speed: 1.306 s/iter, ETA 0:02:18 (D:H:M) [2025-04-02 20:18:59,179-rk0-train.py#300] Epoch: [11][60/642] lr: 0.000483 batch_time: 0.462952 (1.358881) data_time: 0.000000 (0.453902) cls_loss: 1192478.000000 (530384016.806605) loc_loss: 0.999114 (0.956494) total_loss: 1192479.000000 (530384017.654972) [2025-04-02 20:18:59,180-rk0-log_helper.py#105] Progress: 6480 / 12840 [50%], Speed: 1.359 s/iter, ETA 0:02:24 (D:H:M) WARNING:root:NaN or Inf found in input tensor. WARNING:root:NaN or Inf found in input tensor. WARNING:root:NaN or Inf found in input tensor. 请问应该怎么解决呢 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[linananananana]

应该是你的训练图片有问题
…

谢谢您的回答！我用的got数据集，并且也进行了裁剪，训练时前10个epoch都没有问题，解冻backbone后就会这样

[xyl-507]

可以换个训练集试试，我没有遇见这种情况

…

---- 回复的原邮件 ---- | 发件人 | ***@***.***> | | 日期 | 2025年04月03日 15:40 | | 收件人 | ***@***.***> | | 抄送至 | ***@***.***>***@***.***> | | 主题 | Re: [xyl-507/CRM] CRM-Siam训练问题 (Issue #12) | 应该是你的训练图片有问题 … 谢谢您的回答！我用的got数据集，并且也进行了裁剪，训练时前10个epoch都没有问题，解冻backbone后就会这样 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***> linananananana left a comment (xyl-507/CRM#12) 应该是你的训练图片有问题 … 谢谢您的回答！我用的got数据集，并且也进行了裁剪，训练时前10个epoch都没有问题，解冻backbone后就会这样 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[linananananana]

可以换个训练集试试，我没有遇见这种情况
…

好的，谢谢回答！

[whjzsy]

请问默认是单卡训练吗？

[xyl-507]

@whjzsy 我们用的是2张卡，训练命令在：https://github.com/xyl-507/CRM/blob/main/CRM-Siam/TRAIN.md

[whjzsy]

我在训练的时候，遇到“no module named 'siamban'”,请问是哪里的问题呢？

[xyl-507]

@whjzsy
你得加整个项目添加进python路径，我在readme中写了：https://github.com/xyl-507/CRM/tree/main/CRM-Siam

export PYTHONPATH=/path/to/CRM-Siam:$PYTHONPATH (for linux)

set PYTHONPATH=%PYTHONPATH%;/path/to/CRM-Siam (for windows)

[whjzsy]

非常感谢，训练跑起来了

[linananananana]

请问除了backbone，需要配置预训练模型吗

[xyl-507]

不需要了，只需要加载主干网络的预训练参数。

[linananananana]

谢谢

…

---原始邮件--- 发件人: ***@***.***> 发送时间: 2025年4月7日(周一) 晚上8:04 收件人: ***@***.***>; 抄送: ***@***.******@***.***>; 主题: Re: [xyl-507/CRM] CRM-Siam训练问题 (Issue #12) 不需要了，只需要加载主干网络的预训练参数。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***> xyl-507 left a comment (xyl-507/CRM#12) 不需要了，只需要加载主干网络的预训练参数。 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[linananananana]

__C.DATASET.GOT10K.FRAME_RANGE = 100
__C.DATASET.GOT10K.NUM_USE = 200000
请问这两个参数为什么要这样设置呀