About Training

[Nidhoggof]

This is a great piece of work. Could I ask about some specific details regarding the training?

Based on the parameter configurations for each stage in the scripts, it seems that the stages described in the README, the scripts themselves, and the stages mentioned in the paper do not fully align. Which one should be followed in practice?

Also, is there any dependency or progressive relationship between the stages? From the scripts, it appears that each stage might be trained independently—could you clarify this?

Finally, regarding training time: I am currently using 32 A800 GPUs (80GB each) to train Stage 3 (Image + Video + Ref-Video), but the training time is far longer than the expected 3–5 days. Could this be due to a mismatch in configuration, such as steps versus epochs, or some other setting?

[linyq17]

Hi, thanks for your interest in our work.

In practice, we recommend following the README settings as training setup.

The training is progressive and stage-by-stage, rather than training each stage independently from scratch. In particular, all Stage 3 runs should be initialized from a Stage 2 checkpoint using:

--checkpoint path_to_checkpoint

Regarding training time, the step number refers to model updates, meaning the count is measured after gradient accumulation, not by the number of raw forward/backward iterations. One possible reason for longer training time is the --num_epochs setting. This parameter is mainly used to make sure training can run long enough when needed; in our actual runs, we use the step number as the main training metric.

Also, for the training time estimate, our tests were mainly conducted on H-series GPUs, so hardware differences may also affect the actual runtime.

[Nidhoggof]

Thank you for your explanation. At present, the training time has been aligned with the schedule you provided.

I would like to further clarify whether the first phase of Stage 2 needs to use the weights from Stage 1. It seems that, from both the training iteration setup and the script design perspective, Stage 1 does not appear to be a strict prerequisite.

In addition, for Stage 3, there is a separate training process for Ref-Video. Should this part be initialized based on the weights from the first phase of Stage 3, where the three tasks are jointly trained?

I am also wondering why this setup is not kept consistent with instruction-based video editing, where the training is achieved through joint optimization of the image editing task and the reference image editing task.