[Reproduction] Performance gap on Swap Blocks & Rearrange Blocks

[JYounngS]

Description

Hi, thanks for the great work! I strictly followed the paper and official code settings to reproduce three tasks, but observed significant gaps on two of them:

Task	Paper	Reproduced
Swap Blocks	67%	37%
Observe and Pick Up	4%	4% ✅
Rearrange Blocks	89%	59%

The correct reproduction on Observe and Pick Up suggests my environment is set up properly. The ~30% gaps on the other two tasks seem beyond normal seed variance.

Could you kindly clarify if there are any undocumented hyperparameters, preprocessing/evaluation details, or known hardware/library sensitivities for these two tasks? Training logs or checkpoints for cross-checking would also be greatly appreciated.

Thank you! 🙏

[wayrise]

Sorry for the late reply. I have just noticed the issues, and I will provide the corresponding evaluation logs and checkpoints in the next few days to justify the score.

By the way, during the actual evaluation process, we tried different action horizon values (e.g., 15 and 20) to achieve the best performance, and we found that the action horizon has a significant impact on performance. We recommend changing the original action horizon (possibly 30) to 20 or 15 and giving it a try.

Sorry again for the late reply and for missing the statement regarding the action horizon. The actual checkpoints will be provided soon.

[d-finite]

Hello! We have conducted a comprehensive evaluation of the m1 tasks. Due to limited computational resources, we trained all m1 tasks simultaneously for 50,000 steps to produce a multi-task m1_mix model and verified the performance of the Mem-0 model through evaluation.

All relevant files can be found here:

• https://huggingface.co/qiuly/Mem-0-m1mix-RMBench
• https://huggingface.co/datasets/qiuly/Mem-0-m1mix-dataset-RMBench

These include, but are not limited to:

• The processed m1_mix dataset (a combination of the raw datasets from all m1 tasks).
• The complete model.
• Training and inference configuration files.
• All evaluation logs and videos.
• Detailed documentation.

[JYounngS]

Hello! We have conducted a comprehensive evaluation of the m1 tasks. Due to limited computational resources, we trained all m1 tasks simultaneously for 50,000 steps to produce a multi-task m1_mix model and verified the performance of the Mem-0 model through evaluation.

All relevant files can be found here:

• https://huggingface.co/qiuly/Mem-0-m1mix-RMBench
• https://huggingface.co/datasets/qiuly/Mem-0-m1mix-dataset-RMBench

These include, but are not limited to:

• The processed m1_mix dataset (a combination of the raw datasets from all m1 tasks).
• The complete model.
• Training and inference configuration files.
• All evaluation logs and videos.
• Detailed documentation.

Thank you very much for your response. However, I am still encountering significant difficulties in reproducing the M(n) task. Would it be possible for you to release the specific checkpoints/weights and additional implementation details (such as the action horizon) used to obtain the results reported in the paper? This would be greatly appreciated.

[Sweethyh]

I also want to ask when you will publish your checkpoints corresponding to your paper? Especially on m(n) tasks? Thank you.