Skip to content

[CI] Shard long kernel test groups - #54754

Merged
khluu merged 1 commit into
vllm-project:mainfrom
khluu:ci-shard-kernels-p90
Sep 1, 2026
Merged

[CI] Shard long kernel test groups#54754
khluu merged 1 commit into
vllm-project:mainfrom
khluu:ci-shard-kernels-p90

Conversation

@khluu

@khluu khluu commented Sep 1, 2026

Copy link
Copy Markdown
Member

Why

In the 24-hour CI dashboard window ending 2026-09-01 10:31 UTC, these kernel jobs exceeded 45 minutes at P90:

  • AMD Attention Kernels shard 1: 3,877s (17 runs)
  • AMD Attention Kernels shard 2: 3,044s (17 runs)
  • NVIDIA B200 Kernels: 2,994s (8 runs)
  • NVIDIA Quantization Kernels shard 1: 2,821s (11 runs)

What changed

  • expand Attention Kernels from 2 to 7 pytest shards; the AMD mirror deliberately inherits the same fan-out because both existing AMD shards are over the threshold
  • expand NVIDIA Quantization Kernels from 2 to 6 pytest shards; retain the AMD mirror's two-shard declaration for when ci-infra indexSelectLargeIndex: block: [308,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed. #473 starts honoring mirror-local parallelism
  • split the B200 command list into three timing-balanced static buckets with fail-closed shard dispatch
  • set 30-minute shard timeouts and shard-qualified labels

This consolidates and replaces draft PRs #52340, #52341, and #52349. No merged or active non-draft PR covers these jobs.

Validation

  • YAML parse
  • current ci-infra Buildkite step schema
  • exact B200 command coverage (every original command appears once)
  • POSIX shell syntax for the B200 dispatcher
  • git diff --check

Targeted CI

Build #86579 passed all targeted jobs. Attention shards: 6:43, 6:48, 6:58, 7:19, 7:35, 7:40, 7:45; B200 shards: 15:24, 16:35, 17:17; Quantization shards: 7:20, 7:25, 8:49, 8:55, 10:46, 13:34.

This is a draft pending human review. AI assistance was used.

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@yewentao256 yewentao256 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the work!

@khluu
khluu merged commit 3b6c0bd into vllm-project:main Sep 1, 2026
35 checks passed
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants