[Bugfix][V1] Warm up slot mapping before JIT monitor#42165
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a warmup mechanism for the V1 slot mapping kernel to ensure it is compiled before the JIT monitor is enabled. Key changes include the implementation of warmup_v1_slot_mapping_kernel, its integration into the GPUWorker warmup sequence, and a Triton JIT optimization to prevent specialization on num_tokens. New unit tests verify the warmup process and its error handling. I have no feedback to provide.
|
@ZJY0516 @qiching @tdoublep @vadiklyutiy Hi, this is follow-up fix for #40137. I found V1 path still triggers JIT warning on first real request. Fix is small — You all reviewed #40137 so your feedback would be very helpful. Thanks. |
cad2699 to
877e619
Compare
76dfd60 to
01f5c6a
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
…tions The Triton _topk_topp_kernel specializes on (1) which of k/p are present (TOPK_ENABLED/TOPP_ENABLED constexprs) and (2) implicit integer specialization of the non-constexpr BATCH_SIZE argument (==1 / divisible-by-16 classes). Sampler warmup only exercised the both-set combination at batch=max_num_seqs, so the first real top-k-only or top-p-only request (batch>=8), or any batch-size class change (e.g. 16 -> 15 under continuous batching), JIT-compiled during inference and caused a latency spike flagged by jit_monitor. Fix: warm all three k/p combinations in _dummy_sampler_run, and mark BATCH_SIZE do_not_specialize (it is only the grid-stride loop bound, same approach as vllm-project#42165 for num_tokens). Adds tests/v1/sample/test_topk_topp_sampler.py:: TestTritonTopkToppWarmupSpecializations covering both axes via the JITFunction device_caches count (reverting either fix makes them fail). Scope: V1 model runner. The V2 SamplingParams.for_sampler_warmup helper has the same both-set-only gap (left as follow-up); the kernel-side do_not_specialize change benefits V2 as well. processed_logits/ processed_logprobs logprobs modes keep the existing single warmup call. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Fang Zihao <fzh075@gmail.com>
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com> (cherry picked from commit 877e619)
01f5c6a to
de3ac98
Compare
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
|
Closing as superseded by the newer slot-mapping warmup branch tracked in lesj0610#85. |
Problem
V1 startup warmup activates before the JIT monitor, but the normal dummy/profile path does not reliably cover
BlockTable.compute_slot_mapping(). As a result,_compute_slot_mapping_kernelcan still compile on the first real request after the monitor is active.There is also a specialization issue. The slot-mapping Triton kernel was specializing on runtime request size, so warming one token count did not necessarily cover the next real request shape.
Approach
This PR keeps the scope limited to the V1 slot-mapping path.
It adds
do_not_specialize=["num_tokens"]to_compute_slot_mapping_kernelwhile keepingmax_num_tokensspecialized, sincemax_num_tokensis engine-lifetime configuration and still useful for Triton optimization.It also adds
warmup_v1_slot_mapping_kernel(), which callscompute_slot_mapping()directly before the JIT monitor is activated. The warmup uses block id 1 instead of the null block and clears temporary state in afinallyblock.A follow-up commit warms request-shaped slot-mapping variants, because the first real serving request can use small prompt/decode shapes that differ from the original synthetic warmup. This is still slot-mapping only; it does not fold attention, MoE, TurboQuant, or hybrid model kernels into this PR.
I searched open PRs for this exact V1 slot-mapping JIT warmup issue and did not find another one.
Test Plan
Test Result
tests/v1/worker/test_gpu_model_runner.py: 34 passed, 16 warningsruff-format/ruff-check: passedmypy-3.10: passedgit diff --check: passedLocal smoke on V1 runner with Qwen3-8B text-only: HTTP 200, no
_compute_slot_mapping_kernelwarning on first request.Combined warmup smoke on the integration branch also showed
_compute_slot_mapping_kernel: 0 warnings after the request-shaped slot-mapping warmup was included. Other warmup buckets are handled by separate PRs.Checklist
AI assistance: Codex, Claude.