Skip to content

Add minimal test-selection trace hooks - #51982

Closed
khluu wants to merge 2 commits into
mainfrom
ci-test-selection-trace-mvp
Closed

Add minimal test-selection trace hooks#51982
khluu wants to merge 2 commits into
mainfrom
ci-test-selection-trace-mvp

Conversation

@khluu

@khluu khluu commented Aug 12, 2026

Copy link
Copy Markdown
Member

What changed

  • add trace_mode: kernel-set to the existing FlashMLA/H100 test definition
  • add one generic optional post-image-build hook callout at the normal build completion point and the existing-image fast path
  • pass only the exact image tag, Bake inputs, target, and repository root to the version-pinned collector supplied by ci-infra
  • keep hook failure best-effort so test-image production behavior and result are preserved

This final vLLM diff is exactly 2 files and 12 inserted lines. Collector code, artifact schemas, Nsight parsing, static kernel-map export, graph construction, snapshot policy, and selection all live in the companion ci-infra draft.

Why

The required evidence is deliberately small: unordered per-test executed repository line sets and unordered per-test launched mangled CUDA kernel identity sets. Ordered call/return traces, Python launch stacks, sampling, deep modes, and retained Nsight timelines are out of scope.

The thin callout lets ci-infra load one exact collector version into both runtime jobs and the completed image build without vendoring CI implementation into vLLM.

Companion ci-infra draft: vllm-project/ci-infra#469

Impact and safety

  • production test commands remain unchanged
  • the hook is inert unless ci-infra explicitly injects VLLM_CI_IMAGE_POST_BUILD_HOOK
  • hook failure does not fail the image build; missing static provenance causes the selector to quarantine GPU evidence as always-run
  • this remains advisory and does not skip production jobs
  • a PR-consumable snapshot is collected only after both drafts land, from the exact resulting vLLM main tree

Validation

  • all applicable repository pre-commit hooks passed on both changed files
  • shell syntax validation passed for .buildkite/image_build/image_build.sh
  • the companion full production render accounts for all 307 unique production keys: 89 traced + 218 always-run
  • the targeted render contains exactly collector, image-build, FlashMLA/H100, and snapshot with exact bundle/hash injection and dependency closure

The rewritten image-build hook has unit, render, packaging, and shell validation through the companion draft but has not yet had a live Buildkite run. This PR remains a draft for human line-by-line review and that live gate.

Contribution notes

AI assistance was used in implementation and validation. The human submitter must review every changed line and understand the design before moving this PR out of draft.

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
@khluu
khluu force-pushed the ci-test-selection-trace-mvp branch from 56a07a9 to 27d8e11 Compare August 19, 2026 06:56
@khluu khluu changed the title Add generic CI trace collectors Add minimal test-selection trace hooks Aug 19, 2026
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
@khluu

khluu commented Aug 19, 2026

Copy link
Copy Markdown
Member Author

Superseded by the automatic ci-infra-only design in vllm-project/ci-infra#469. The branch now removes the explicit trace_mode and image post-build hook, leaving no net vLLM source change.

@khluu khluu closed this Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant