Skip to content

[Bugfix][V1] Warm up slot mapping before JIT monitor#42165

Closed
lesj0610 wants to merge 2 commits into
vllm-project:mainfrom
lesj0610:lesj/v1-slot-mapping-jit-warmup-upstream-20260509
Closed

[Bugfix][V1] Warm up slot mapping before JIT monitor#42165
lesj0610 wants to merge 2 commits into
vllm-project:mainfrom
lesj0610:lesj/v1-slot-mapping-jit-warmup-upstream-20260509

Conversation

@lesj0610

@lesj0610 lesj0610 commented May 9, 2026

Copy link
Copy Markdown
Contributor

Problem

V1 startup warmup activates before the JIT monitor, but the normal dummy/profile path does not reliably cover BlockTable.compute_slot_mapping(). As a result, _compute_slot_mapping_kernel can still compile on the first real request after the monitor is active.

There is also a specialization issue. The slot-mapping Triton kernel was specializing on runtime request size, so warming one token count did not necessarily cover the next real request shape.

Approach

This PR keeps the scope limited to the V1 slot-mapping path.

It adds do_not_specialize=["num_tokens"] to _compute_slot_mapping_kernel while keeping max_num_tokens specialized, since max_num_tokens is engine-lifetime configuration and still useful for Triton optimization.

It also adds warmup_v1_slot_mapping_kernel(), which calls compute_slot_mapping() directly before the JIT monitor is activated. The warmup uses block id 1 instead of the null block and clears temporary state in a finally block.

A follow-up commit warms request-shaped slot-mapping variants, because the first real serving request can use small prompt/decode shapes that differ from the original synthetic warmup. This is still slot-mapping only; it does not fold attention, MoE, TurboQuant, or hybrid model kernels into this PR.

I searched open PRs for this exact V1 slot-mapping JIT warmup issue and did not find another one.

Test Plan

.venv/bin/python -m pytest tests/v1/worker/test_gpu_model_runner.py -v

pre-commit run ruff-format --files \
  vllm/v1/worker/block_table.py \
  vllm/v1/worker/gpu/warmup.py \
  vllm/v1/worker/gpu_worker.py \
  tests/v1/worker/test_gpu_model_runner.py

pre-commit run ruff-check --files \
  vllm/v1/worker/block_table.py \
  vllm/v1/worker/gpu/warmup.py \
  vllm/v1/worker/gpu_worker.py \
  tests/v1/worker/test_gpu_model_runner.py

pre-commit run mypy-3.10 --files \
  vllm/v1/worker/block_table.py \
  vllm/v1/worker/gpu/warmup.py \
  vllm/v1/worker/gpu_worker.py \
  tests/v1/worker/test_gpu_model_runner.py \
  --hook-stage manual

git diff --check

Test Result

  • tests/v1/worker/test_gpu_model_runner.py: 34 passed, 16 warnings
  • ruff-format / ruff-check: passed
  • mypy-3.10: passed
  • git diff --check: passed

Local smoke on V1 runner with Qwen3-8B text-only: HTTP 200, no _compute_slot_mapping_kernel warning on first request.

Combined warmup smoke on the integration branch also showed _compute_slot_mapping_kernel: 0 warnings after the request-shaped slot-mapping warmup was included. Other warmup buckets are handled by separate PRs.

Checklist
  • Purpose
  • Test plan and results
  • AI assistance disclosed

AI assistance: Codex, Claude.

@lesj0610 lesj0610 requested review from WoosukKwon and njhill as code owners May 9, 2026 13:13

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added v1 bug Something isn't working labels May 9, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a warmup mechanism for the V1 slot mapping kernel to ensure it is compiled before the JIT monitor is enabled. Key changes include the implementation of warmup_v1_slot_mapping_kernel, its integration into the GPUWorker warmup sequence, and a Triton JIT optimization to prevent specialization on num_tokens. New unit tests verify the warmup process and its error handling. I have no feedback to provide.

@lesj0610

lesj0610 commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

@ZJY0516 @qiching @tdoublep @vadiklyutiy Hi, this is follow-up fix for #40137.

I found V1 path still triggers JIT warning on first real request. _dummy_run() does not call BlockTable.compute_slot_mapping(), and the kernel was specialized on num_tokens so each different request size recompiles.

Fix is small — do_not_specialize=["num_tokens"] and direct compute_slot_mapping() call during V1 warmup before monitor activates. No synthetic execute_model() warmup, that is model-specific so I keep scope to slot mapping only.

You all reviewed #40137 so your feedback would be very helpful. Thanks.

@lesj0610 lesj0610 force-pushed the lesj/v1-slot-mapping-jit-warmup-upstream-20260509 branch from cad2699 to 877e619 Compare May 9, 2026 13:24
@lesj0610 lesj0610 force-pushed the lesj/v1-slot-mapping-jit-warmup-upstream-20260509 branch from 76dfd60 to 01f5c6a Compare May 23, 2026 05:47
@mergify

mergify Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lesj0610.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 15, 2026
fzh075 added a commit to fzh075/vllm that referenced this pull request Jun 16, 2026
…tions

The Triton _topk_topp_kernel specializes on (1) which of k/p are present
(TOPK_ENABLED/TOPP_ENABLED constexprs) and (2) implicit integer
specialization of the non-constexpr BATCH_SIZE argument (==1 /
divisible-by-16 classes). Sampler warmup only exercised the both-set
combination at batch=max_num_seqs, so the first real top-k-only or
top-p-only request (batch>=8), or any batch-size class change (e.g.
16 -> 15 under continuous batching), JIT-compiled during inference and
caused a latency spike flagged by jit_monitor.

Fix: warm all three k/p combinations in _dummy_sampler_run, and mark
BATCH_SIZE do_not_specialize (it is only the grid-stride loop bound,
same approach as vllm-project#42165 for num_tokens).

Adds tests/v1/sample/test_topk_topp_sampler.py::
TestTritonTopkToppWarmupSpecializations covering both axes via the
JITFunction device_caches count (reverting either fix makes them fail).

Scope: V1 model runner. The V2 SamplingParams.for_sampler_warmup helper
has the same both-set-only gap (left as follow-up); the kernel-side
do_not_specialize change benefits V2 as well. processed_logits/
processed_logprobs logprobs modes keep the existing single warmup call.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Fang Zihao <fzh075@gmail.com>
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
(cherry picked from commit 877e619)
@lesj0610 lesj0610 force-pushed the lesj/v1-slot-mapping-jit-warmup-upstream-20260509 branch from 01f5c6a to de3ac98 Compare June 19, 2026 14:14
@lesj0610 lesj0610 requested a review from yewentao256 as a code owner June 19, 2026 14:14
@mergify mergify Bot removed the needs-rebase label Jun 19, 2026
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
@lesj0610

Copy link
Copy Markdown
Contributor Author

Closing as superseded by the newer slot-mapping warmup branch tracked in lesj0610#85.

@lesj0610 lesj0610 closed this Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant