Skip to content

perf(qwen): reduce generation overhead - #1024

Open
chaofengw-nv wants to merge 2 commits into
NVIDIA:mainfrom
chaofengw-nv:fix/qwen3-generation-performance-901
Open

perf(qwen): reduce generation overhead#1024
chaofengw-nv wants to merge 2 commits into
NVIDIA:mainfrom
chaofengw-nv:fix/qwen3-generation-performance-901

Conversation

@chaofengw-nv

Copy link
Copy Markdown
Collaborator

Background

qwen3-4b-instruct-2507 preserved exact FP16 output but measured 39.7% slower than its reference on Thor X. This change removes repeated CUDA-graph setup from the generation path and keeps the release comparison precision aligned.

Exit Criteria

  • Preserve exact-token Accuracy behavior on Thor X and GB300.
  • Bring the Thor X release workload back within the existing 5% performance policy without weakening the gate.
  • Keep the fixed native KV-cache boundary valid for full-capacity prompts.

Implementation

  • Enable CUDA graphs for the Qwen3-4B release workload and compare against an FP16 reference.
  • Skip decoder graph priming once the graph is already captured.
  • Account for the first generated token coming from prefill so fixed KV capacity only reserves rows consumed by later decoder steps.
  • Add source and native-contract regressions for graph priming, release policy, and KV-cache capacity boundaries.

Validation

  • python3 -m pytest -q tests/builder/test_qwen_runtime_performance_contract.py tests/e2e/models/qwen/test_qwen_performance_contract.py: 4 passed.
  • python3 -m tools.community_ci source-quality --base github/main: passed all complexity, lint, architecture, and source-quality checks; 158 tests passed.
  • test_qwen_native_kv_cache native executable: passed on GB300 at exact head 500ca807556eaa5a5b8d59717af559dfe41889dc.
  • GB300 model checks at the exact head: Accuracy passed 20/20 with exact agreement and 0 failed samples; Perf passed green with 9.34 ms TRTMC p50 versus 71.31 ms reference p50.
  • Thor X model checks at the exact head: Accuracy passed 20/20 with exact agreement and 0 failed samples; Perf passed green with 102.80 ms TRTMC p50 versus 109.88 ms reference p50. Both 10-sample timing series were stable and output tokens matched exactly.
  • A broader local Qwen collection reached 298 passed and 4 skipped; 11 GPU/E2E cases were not runnable in the local no-CUDA/read-only-storage environment and were covered by the target-hardware checks above.

Notes For Future Readers

  • Review the runtime changes first, then the Qwen-specific release profile, followed by the shared native KV-cache contract tests.
  • The existing 5% performance policy is unchanged.

Closes #901

Enable CUDA graph measurement and avoid re-priming an already captured decoder graph. Correct fixed KV capacity for tokens produced by prefill.

Refs: NVIDIA#901
Format the shared native KV contract test with the clang-format version pinned by Community CPU so local and CI checks agree.
@chaofengw-nv chaofengw-nv added the run-internal-ci Maintainer-approved dispatch to internal CI label Aug 25, 2026
@github-actions github-actions Bot removed the run-internal-ci Maintainer-approved dispatch to internal CI label Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] qwen3-4b-instruct-2507 latency is 39.7% above reference

1 participant