[Question] --max-length parameter actually determine when training a draft model

[xinjiahui]

What does the --max-length parameter actually determine when training a draft model? If I set --max-length 2048, does it mean that the maximum context length for the draft model (including during speculative decoding/acceleration) is always 2048 tokens? In other words, even if the target model supports longer sequences, the draft model’s context window for acceleration will still be capped at 2048? I want to confirm whether this understanding is correct. Thanks

[jiapingW]

The max-length parameter determines the maximum length of sequences that the Eagle3 model can see during training. It is not equal to the maximum length that Eagle3 can infer because ropes can perform length extrapolation. You can train with a 2k length and infer with a 4k length. However, using a larger length generally means that your Eagle3 model needs more training data on the same dataset, resulting in better performance.

What does the --max-length parameter actually determine when training a draft model? If I set --max-length 2048, does it mean that the maximum context length for the draft model (including during speculative decoding/acceleration) is always 2048 tokens? In other words, even if the target model supports longer sequences, the draft model’s context window for acceleration will still be capped at 2048? I want to confirm whether this understanding is correct. Thanks