Seeking Guidance on Reproduction

[eunicehon]

Thank you very much for open-sourcing this excellent work!

We encountered some discrepancies while attempting to reproduce your results and would appreciate your guidance. Following the default configuration and code provided in the repository, we conducted complete training (10 epochs) on an 8-GPU machine, but obtained a PLCC of around 0.8 on the KonIQ test set.

To verify our evaluation pipeline, we first tested using your released weights and achieved a PLCC of approximately 0.93, which aligns with the performance reported in the paper. This suggests that our evaluation code should be correct.

When analyzing the training process, we noticed a potential difference: during our reproduction, the format reward eventually converged around 0.5, while the trainer_state.json corresponding to your released weights shows that the format reward quickly converged to 1.0 by step 29. We are uncertain whether this discrepancy directly contributes to the final performance gap.

We've included our training curves for reference. If there are any training details or configuration adjustments we might have overlooked, we would be very grateful for your guidance. Thank you!

[ZachL1]

I encountered a similar issue during reproduction: on the KonIQ test set the PLCC is below 0.8.

It is worth mentioning that even when training with 80G × 16 GPUs, I still run into OOM and have to set gradient_checkpointing to true in order to train. I’m not sure whether this affects performance.

I would greatly appreciate any comments or discussion!

---

In addition, I also observed that during training there is one step where the gradient becomes extremely large:

[yinwen2019]

I encountered a similar issue during reproduction: on the KonIQ test set the PLCC is below 0.8.

It is worth mentioning that even when training with 80G × 16 GPUs, I still run into OOM and have to set gradient_checkpointing to true in order to train. I’m not sure whether this affects performance.

I would greatly appreciate any comments or discussion!

In addition, I also observed that during training there is one step where the gradient becomes extremely large:

我也遇到了类似的问题一致报错（RuntimeError: CUDA driver error: invalid argument），然后我试着改成--gradient_checkpointing true ，发现就能在8张a100(80gb)上运行，每张占用50-60%显存。

[yinwen2019]

Thank you very much for open-sourcing this excellent work!非常感谢你开源这部优秀的作品！

We encountered some discrepancies while attempting to reproduce your results and would appreciate your guidance. Following the default configuration and code provided in the repository, we conducted complete training (10 epochs) on an 8-GPU machine, but obtained a PLCC of around 0.8 on the KonIQ test set.我们在尝试复现您的结果时遇到了一些不一致，非常感谢您的指导。按照仓库中默认**配置和代码 **，我们在一台 8GPU 机器上进行了完整的训练（10 个时代），但在 KonIQ 测试集上获得了约 0.8 的 PLCC 分数。

To verify our evaluation pipeline, we first tested using your released weights and achieved a PLCC of approximately 0.93, which aligns with the performance reported in the paper. This suggests that our evaluation code should be correct.为了验证我们的评估流程，我们首先使用您发布的权重进行测试，PLCC 约为 0.93，与论文中报告的表现一致。 这表明我们的评估代码应该是正确的。

When analyzing the training process, we noticed a potential difference: during our reproduction, the format reward eventually converged around 0.5, while the trainer_state.json corresponding to your released weights shows that the format reward quickly converged to 1.0 by step 29. We are uncertain whether this discrepancy directly contributes to the final performance gap.在分析训练过程时，我们注意到一个潜在差异 ：在我们的重现过程中， 格式奖励最终收敛到大约 0.5，而你释放的权重对trainer_state.json应的指标显示，到第 29 步时格式奖励很快收敛到 1.0。我们尚不确定这种差异是否直接导致了最终性能差距。

We've included our training curves for reference. If there are any training details or configuration adjustments we might have overlooked, we would be very grateful for your guidance. Thank you!我们附上了训练曲线供参考。如果有任何培训细节或配置调整我们可能忽略了，非常感谢您的指导。谢谢！

我刚刚复现了训练过程，我的format_reward确实在30步左右收敛至1，如图：

[eunicehon]

Thank you very much for open-sourcing this excellent work!非常感谢你开源这部优秀的作品！
We encountered some discrepancies while attempting to reproduce your results and would appreciate your guidance. Following the default configuration and code provided in the repository, we conducted complete training (10 epochs) on an 8-GPU machine, but obtained a PLCC of around 0.8 on the KonIQ test set.我们在尝试复现您的结果时遇到了一些不一致，非常感谢您的指导。按照仓库中默认配置和代码 ，我们在一台 8GPU 机器上进行了完整的训练（10 个时代），但在 KonIQ 测试集上获得了约 0.8 的 PLCC 分数。
To verify our evaluation pipeline, we first tested using your released weights and achieved a PLCC of approximately 0.93, which aligns with the performance reported in the paper. This suggests that our evaluation code should be correct.为了验证我们的评估流程，我们首先使用您发布的权重进行测试，PLCC 约为 0.93，与论文中报告的表现一致。 这表明我们的评估代码应该是正确的。
When analyzing the training process, we noticed a potential difference: during our reproduction, the format reward eventually converged around 0.5, while the trainer_state.json corresponding to your released weights shows that the format reward quickly converged to 1.0 by step 29. We are uncertain whether this discrepancy directly contributes to the final performance gap.在分析训练过程时，我们注意到一个潜在差异 ：在我们的重现过程中， 格式奖励最终收敛到大约 0.5，而你释放的权重对trainer_state.json应的指标显示，到第 29 步时格式奖励很快收敛到 1.0。我们尚不确定这种差异是否直接导致了最终性能差距。
We've included our training curves for reference. If there are any training details or configuration adjustments we might have overlooked, we would be very grateful for your guidance. Thank you!我们附上了训练曲线供参考。如果有任何培训细节或配置调整我们可能忽略了，非常感谢您的指导。谢谢！

我刚刚复现了训练过程，我的format_reward确实在30步左右收敛至1，如图：

但是看起来@ZachL1 的训练曲线中和我相近，似乎也遇到了format_reward只有0.5~0.6的问题。您是否可以分享一下您复现用的训练script呢？您除了修改--gradient_checkpointing true还有其他部分的修改么？谢谢

[yinwen2019]

Thank you very much for open-sourcing this excellent work!非常感谢你开源这部优秀的作品！
We encountered some discrepancies while attempting to reproduce your results and would appreciate your guidance. Following the default configuration and code provided in the repository, we conducted complete training (10 epochs) on an 8-GPU machine, but obtained a PLCC of around 0.8 on the KonIQ test set.我们在尝试复现您的结果时遇到了一些不一致，非常感谢您的指导。按照仓库中默认配置和代码 ，我们在一台 8GPU 机器上进行了完整的训练（10 个时代），但在 KonIQ 测试集上获得了约 0.8 的 PLCC 分数。
To verify our evaluation pipeline, we first tested using your released weights and achieved a PLCC of approximately 0.93, which aligns with the performance reported in the paper. This suggests that our evaluation code should be correct.为了验证我们的评估流程，我们首先使用您发布的权重进行测试，PLCC 约为 0.93，与论文中报告的表现一致。 这表明我们的评估代码应该是正确的。
When analyzing the training process, we noticed a potential difference: during our reproduction, the format reward eventually converged around 0.5, while the trainer_state.json corresponding to your released weights shows that the format reward quickly converged to 1.0 by step 29. We are uncertain whether this discrepancy directly contributes to the final performance gap.在分析训练过程时，我们注意到一个潜在差异 ：在我们的重现过程中， 格式奖励最终收敛到大约 0.5，而你释放的权重对trainer_state.json应的指标显示，到第 29 步时格式奖励很快收敛到 1.0。我们尚不确定这种差异是否直接导致了最终性能差距。
We've included our training curves for reference. If there are any training details or configuration adjustments we might have overlooked, we would be very grateful for your guidance. Thank you!我们附上了训练曲线供参考。如果有任何培训细节或配置调整我们可能忽略了，非常感谢您的指导。谢谢！

我刚刚复现了训练过程，我的format_reward确实在30步左右收敛至1，如图：

但是看起来@ZachL1 的训练曲线中和我相近，似乎也遇到了format_reward只有0.5~0.6的问题。您是否可以分享一下您复现用的训练script呢？您除了修改--gradient_checkpointing true还有其他部分的修改么？谢谢

是的，我只改了这一行--gradient_checkpointing true