Skip to content

Synchronize distributed FlashInfer autotuning and persist cache#48714

Draft
jiahanc wants to merge 2 commits into
vllm-project:mainfrom
jiahanc:autoune-cache
Draft

Synchronize distributed FlashInfer autotuning and persist cache#48714
jiahanc wants to merge 2 commits into
vllm-project:mainfrom
jiahanc:autoune-cache

Conversation

@jiahanc

@jiahanc jiahanc commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Need flashinfer version update

Purpose

Distributed FlashInfer autotuning currently disables the persistent cache and reruns tuning on every server startup. Ranks also profile tactics independently, so GPU timing noise can cause them to select different tactics.

flashinfer-ai/flashinfer#3187 adds set_autotune_process_group, which averages tactic timings across ranks so all ranks select the same tactic.

This PR wires that API into vLLM:

  • Broadcast and load an existing cache before distributed autotuning.
  • Run autotuning on every rank with synchronized timings over the world CPU process group.
  • Save the resulting cache on rank 0.
  • Broadcast and load the final cache on every rank.
  • Clear the autotune process group in a finally block.

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

jiahanc added 2 commits July 14, 2026 23:37
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Assisted-by: OpenAI Codex
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant