Skip to content

[Feature][NPU] Add FP8 KV-slice dense FIA path for packed FP8 attention (MINDIESD_FP8_KV_SLICE) - #6800

Open
brandneway wants to merge 3 commits into
vllm-project:mainfrom
brandneway:feature/npu-fp8-kv-slice
Open

[Feature][NPU] Add FP8 KV-slice dense FIA path for packed FP8 attention (MINDIESD_FP8_KV_SLICE)#6800
brandneway wants to merge 3 commits into
vllm-project:mainfrom
brandneway:feature/npu-fp8-kv-slice

Conversation

@brandneway

Copy link
Copy Markdown
Contributor

What

Adds an opt-in FP8 attention path for NPU where K/V are sliced to the
valid prefix outside the operator, so packed [real, pad] attention
runs as a single dense MindIE-SD FIA call instead of relying on the
operator's varlen feature:

  • vllm_omni/platforms/npu/quant/kv_quant_npu.py
    • New entry point fp8_rotate_quant_kv_slice(query, key, value, kv_len):
      K/V are sliced to the valid prefix (zero-copy narrow) before
      rotation and quantization — padding rows are neither attended nor
      quantized — then one dense block-FP8 FIA call runs with the varlen
      feature (actual_seq_* / NTD_TND) entirely off.
    • fp8_kv_slice_enabled() reads MINDIESD_FP8_KV_SLICE per call
      (no cache), matching the MINDIE_SD_FA_TYPE dispatch convention.
    • The MindIE-SD FIA wrapper is loaded optionally in
      _load_quant_ops, so the existing dense fp8_rotate_quant_fa path
      keeps working against MindIE-SD builds that do not provide it; the
      kv-slice entry point raises a clear ImportError in that case.
  • vllm_omni/diffusion/attention/backends/flash_attn.py
    • forward_npu: when kv_cache_dtype=fp8 rides the
      npu_attn_varlen packed route and MINDIESD_FP8_KV_SLICE is set,
      dispatch to _forward_prefix_kv_slice_quant_npu, which resolves the
      packed boundaries and hands the backend unsliced BSND q/k/v plus the
      real document length. If the packed contract does not hold, falls
      back to unquantized attention — dense FP8 would ignore document
      boundaries. With the env unset the dispatch is unchanged.

Why

  • The FIA dispatch always uses input_layout="BNSD" regardless of the
    caller-facing layout: fa_block_quant_preprocess transposes BSND
    inputs before the quant kernel, so its output (and the FIA output) is
    always BNSD-logical; the wrapper transposes back for BSND callers.
  • Query keeps its full length; outputs on padding rows are never consumed
    downstream (same contract as the unquantized prefix-K/V-slice path).
    Per-block scales can mix pad rows in the boundary block — exact when
    the packing pads with zeros, which is the MiniMax-H3 packing this path
    is built for.
  • With kv_cache_dtype=fp8, K/V slices shrink to the valid prefix, so
    quantization and attention both skip the padding document entirely.

Usage

export MINDIESD_FP8_KV_SLICE=1
vllm serve ... \
  --diffusion-attention-backend FLASH_ATTN \
  --diffusion-kv-cache-dtype fp8

Test Plan
 CPU unit tests (tests/platforms/npu/quant/test_kv_quant_npu.py): fake quant ops mirror the real kernel contract (BNSD-logical quant output, per-(head, row-block) scales); cover env parsing, the dense single-call dispatch for BSND/BNSD callers (K/V sliced before quant, BNSD input_layout, varlen feature off), full-length K/V as a no-op slice, and kv_len/layout validation.
 Routing tests (tests/diffusion/attention/test_flash_attn.py): env opt-in selects the kv-slice path; env-off keeps the dense quant dispatch unchanged; a failed packed contract falls back to unquantized attention, never dense FP8; the quant wrapper receives unsliced BSND q/k/v with the real document length.
 NPU-only smoke test for the real shape contract (test_fp8_rotate_quant_kv_slice_real_npu_shape_contract).
 End-to-end on 4×950PR with MiniMax-H3 (--task-type fl2va) using the usage config above.

changetheway and others added 3 commits August 30, 2026 00:58
Co-Authored-By: brandneway <gyuan4892@gmail.com>
Co-Authored-By: brandneway <gyuan4892@gmail.com>
Co-Authored-By: brandneway <gyuan4892@gmail.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to belong to: docs/design/module/quantization.md.

Module owners: @david6666666 @Isotr0py @lishunyang12

@brandneway, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants