Skip to content

[CI] Shard Kernels Quantization Test to 6 shards, pin AMD mirror at 2 (<20 min) - #52341

Closed
khluu wants to merge 1 commit into
vllm-project:mainfrom
khluu:amr/shard-kernels-quantization
Closed

[CI] Shard Kernels Quantization Test to 6 shards, pin AMD mirror at 2 (<20 min)#52341
khluu wants to merge 1 commit into
vllm-project:mainfrom
khluu:amr/shard-kernels-quantization

Conversation

@khluu

@khluu khluu commented Aug 14, 2026

Copy link
Copy Markdown
Member

What

kernels-quantization-test runs 2 NVIDIA shards at ~40–45 min each (~85 min logical) — a Phase-2 long pole. Raise NVIDIA parallelism 2 → 6 to land each shard ~14 min (<20 min target), keeping the AMD mirror at its current 2 shards.

How

  • NVIDIA parallelism: 2 → 6 (the pytest -v -s kernels/quantization --shard-id/--num-shards command already shards). Per-shard timeout_in_minutes 60 → 30. Coverage/selection unchanged.
  • Pin mirror.amd.parallelism: 2 so the higher NVIDIA count does not multiply the AMD mirror. Depends on the ci-infra AMD-mirror parallelism override — Honor mirror.amd.parallelism so NVIDIA sharding cannot multiply AMD mirrors ci-infra#473 (amd-mirror-parallelism-override); without it the field is ignored (AMD would inherit 6), with it AMD stays 2. AMD command/coverage unchanged.

Sizing

~85 min logical ÷ 6 ≈ 14 min/shard; headroom under 20 min for nodeid-hash imbalance. Provisional — confirm/trim from the targeted run.

Validation — build #83916 (targeted kernels-quantization-test on #473, 6/6 green)

  • Per-shard NVIDIA wall (min): 10.27 / 11.72 / 11.85 / 10.47 / 16.15 / 13.55 — max 16.15, all <20.
  • Exact node union: per-shard "Running N" = 1032/992/1027/1004/1002/1026, Σ = 6083 = collected 6083 (disjoint partition).
  • Fixed setup: 4.30–4.36 min/shard.
  • Total GPU-min: 74.0 vs the 2-shard baseline 85 (−11) (within run variance; added per-shard setup offset by lower test time this run); wall dropped ~48→16.15 min (~3×).
  • AMD2 selector-off render (real indexSelectLargeIndex: block: [308,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed. #473 generator, head e006ba64): NVIDIA renders 6, AMD mirror pinned to exactly 2 (un-pinned it would inherit 6); AMD command byte-identical.

Part of the Phase-2 CI-overhaul sharding pass (kernels/basic-models workstream), one PR per key.

…or at 2

kernels-quantization-test currently runs 2 NVIDIA shards at ~40-45m each (~85m
logical, Phase 2 45+ min long pole). Raise NVIDIA parallelism 2 -> 6 to land each
shard ~14m (<20m target). Pin mirror.amd.parallelism: 2 so the higher NVIDIA
count does not multiply the AMD mirror; relies on ci-infra amd-mirror-parallelism
-override (PR vllm-project#473). Per-shard timeout 60 -> 30.
@khluu

khluu commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Superseded by consolidated kernels PR #54754 on current main.

@khluu khluu closed this Sep 1, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Sep 1, 2026
@khluu
khluu deleted the amr/shard-kernels-quantization branch September 1, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant