During Track 2 baseline verification, run 3's wall-clock came in at 953.4s vs ~525s for runs 1–2. The training loop was identical (157 steps @ ~3.5s/it in all three runs) — the difference was ~6 minutes of pre-training I/O (model/dataset load from the cache volume) on that container. Details in the verification report.
Per the frozen rules, wall-clock includes loading, so the run counts and the official mean (668.0s) stands. This issue exists to track the question for any future track spec (v2+): should the official time be median-of-3 instead of mean, or should the clock start after a standardized warm-up? Changing it for existing tracks would break time comparability, so v1 tracks keep the current rule regardless.
Opinions welcome — especially from anyone who's hit similar variance on Modal volumes.
During Track 2 baseline verification, run 3's wall-clock came in at 953.4s vs ~525s for runs 1–2. The training loop was identical (157 steps @ ~3.5s/it in all three runs) — the difference was ~6 minutes of pre-training I/O (model/dataset load from the cache volume) on that container. Details in the verification report.
Per the frozen rules, wall-clock includes loading, so the run counts and the official mean (668.0s) stands. This issue exists to track the question for any future track spec (v2+): should the official time be median-of-3 instead of mean, or should the clock start after a standardized warm-up? Changing it for existing tracks would break time comparability, so v1 tracks keep the current rule regardless.
Opinions welcome — especially from anyone who's hit similar variance on Modal volumes.