Question about single-task vs multi-task training for the Execution Module

[SizheZhao]

Hi, thanks for releasing the valuable code and simulation benchmark.

I have a question about the training setup of the Execution Module. In the paper, it is mentioned that:

“The Execution Module of Mem-0 utilizes a single-task training strategy.”

In my own experiments, I first trained and evaluated my model on a single task, and the execution performance looked reasonable. However, when I trained one shared executor jointly across all 9 tasks, the model's manipulation ability became much worse. The issue is mainly related to the manipulation capability rather than the memory component.

I would like to better understand the expected behavior and the experimental setup:

1. Is training one shared Execution Module across multiple tasks generally expected to be difficult in this environment?
2. Did you try multi-task training for the Execution Module of MEM-0, and if so, did you observe similar degradation in action execution?
3. For the comparison methods in the paper, such as pi0.5 and X-VLA, were their execution policies also trained separately for each task?

This would help me understand whether the degradation I observe is likely caused by my model, the training setup, the number of expert demonstrations (50) , or whether multi-task executor training is generally challenging in this benchmark.

Thanks!

---

Single-task training:

episode0_singletask.mp4

Multi-task training:

episode1.mp4

[wayrise]

Due to the special memory accumulation mechanism, when training on multiple tasks, we need to assign different episode IDs to all episodes, including episodes from different task categories. In other words, assuming you have 10 tasks with 100 trajectories per task, for the total of 1,000 data samples, it is better to assign episode IDs ranging from 0 to 999 to avoid potential confusion in the memory aggregation mechanism.

We have previously tried multi-task training, and the overall performance was fairly good. However, we observed some performance degradation on certain single tasks (especially tasks with relatively short execution trajectories), while some relatively longer-horizon tasks showed improvements. Of course, this phenomenon may not always hold, but in theory, multi-task training should work properly.