CI: Add warmup requests before benchmark to avoid JIT compilation during timed run

[gyohuangxin]

Summary

• Add --num-warmups=$CONC to benchmark_serving.py invocation in atom_test.sh
• This sends warmup requests before the timed benchmark run, triggering JIT compilation of prefill-path kernels (e.g. mha_varlen_fwd) so they don't compile during the actual measurement
• Without this, the first batch of benchmark requests triggers ~24s of aiter JIT compilation, skewing TTFT and throughput numbers

Test plan

• Run a benchmark CI job and verify Warming up with N requests... appears before Starting main benchmark run...
• Confirm no [aiter] start build messages appear after Starting main benchmark run...

[Copilot]

Pull request overview

Updates the benchmark CI script to run warmup requests before the timed benchmark phase, aiming to avoid JIT compilation skewing TTFT/throughput measurements.

Changes:

• Add --num-warmups=$CONC to the benchmark_serving.py invocation in the benchmark path of atom_test.sh.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

--num-warmups=$CONC relies on a CLI option that may not exist in the cloned https://github.com/kimbochen/bench_serving.git (the script is cloned from HEAD, unpinned). If upstream changes or lacks this flag, the benchmark step will fail at runtime. Consider pinning bench_serving to a known commit/tag that supports --num-warmups (or switching to the in-repo atom/benchmarks/benchmark_serving.py), and/or adding a quick guard that checks the flag is supported before invoking the benchmark. Also note this adds CONC extra requests per run (up to 256 in the nightly matrix); you may want to cap warmups to a smaller fixed number if the goal is only to trigger JIT once.