Automatically trace vLLM nightly pytest jobs

[khluu]

What changed

• Automatically trace pytest jobs declared under vLLM's .buildkite/test_areas on trusted main nightlies.
• Collect per-test repository Python file/line presence for CPU jobs and the same Python evidence plus mangled CUDA kernel identities for eligible NVIDIA jobs.
• Preserve evidence-backed exact-key firewalls: 46 collector-compatibility jobs and 5 prohibitive Python-coverage-overhead jobs stay visible but uninstrumented/always-run.
• Publish one checksum-pinned collector bundle without making production jobs depend on it; collector failures preserve the original command result and mark evidence unhealthy.
• Build and publish a compact immutable SQLite graph, deterministic gzip-v2 object, checksum sidecar, and conditional S3 index.
• Keep missing, unhealthy, incompatible, stale, non-ancestral, and unmapped coverage conservatively always-run/fallback.
• Expose an advisory selector with file → test → job why-chains.
• Add a fail-closed, operator-only recovery gate that renders one CPU postmerge republisher instead of rerunning the fleet.

There are no per-job tracing declarations, retained Nsight timelines, AST symbol indexes, static source-to-kernel export steps, or production PR skip behavior in this draft.

Why

The repository layout is the enrollment contract: pytest jobs under .buildkite/test_areas should produce evidence automatically. Keeping policy in ci-infra removes duplicated YAML knobs and lets an unchanged vLLM main tree collect trace evidence.

The first selector is deliberately conservative. Python changes may use observed file/line presence. Native/kernel source changes and any unmapped file fall back until an independently validated source-to-kernel mapping exists.

Safety properties

• Tracing activates only for vllm-project/vllm, NIGHTLY=1, branch main, and a non-PR build.
• The production S3 bucket default exists only inside that trusted gate.
• The collector step is soft-failing and has no hard production dependents.
• Traced jobs preserve their original command order and exit status; missing/invalid/import-failed collectors run the original production command exactly once.
• The publisher polls terminal step state with an eight-hour bound; missing/timeout evidence becomes always-run.
• Snapshot identity binds the exact vLLM SHA, ci-infra revision, collector SHA, inventory, shard count, artifacts, and checksums.
• Readers verify compressed bytes before bounded decompression, then verify the logical SQLite checksum.
• Recovery inputs require a canonical inventory bound to the build commit, complete wait accounting, numeric source-build provenance, and the Buildkite source UUID. Any recovery env replaces the full generated fleet with one recovery step; never set it on a normal nightly.
• The selector is shadow-only. This PR does not make production skip decisions.

Validation

Local and GitHub

• Final exact head: 605214b63275ba4ebdbe0ee81861e0f1d6e5dc39.
• 175 passed across buildkite/tests.
• Applicable pre-commit hooks and git diff --check pass.
• GitHub Pipeline generator tests and DCO pass.
• Exact vLLM seed render at eac636a7fa476983cdae34b45a984e9852aad375: 307 unique production command keys = 121 traced + 186 always-run.
• Trace modes: 25 Python-only + 96 NVIDIA kernel-set, 144 expected shards.
• Exactly 46 compatibility-policy + 5 CPU-overhead-policy rows remain always-run.
• All 2,101 rendered shell commands parse after Buildkite dollar normalization; no duplicate command keys or collector dependencies.

Production evidence

• Seed build #84585 produced the frozen fleet evidence.
• Final inventory: 36,052 bytes, SHA-256 8b22e49c058651312e90703807d3714eaa8555999f95039040b1967352e2889c.
• Current jobs: 307 keys, SHA-256 93e823241c21935490c615f0cb98348090b116b533486ab0e512e6dd9bf22eff.
• Fan-in accounting: 121 traced = 80 healthy + 8 missing + 33 unhealthy; 186 additional jobs remain always-run.
• Recovery build #84672 downloaded all 1,136 artifacts (28.05 GiB), materialized and published gzip-v2, CAS-promoted the production index, and completed a fresh read-back.
• Manifest SHA-256: 4c09961b5b5a5fa16f3a3a280a35dd6f5f5108309c27a933cf34f0ee5a1d460c.
• Read-back SQLite SHA-256: df68c3f2efca9fdf982d01df3a21be0fd20bdc35ee5d164996a3139f0bcd31e9.
• Exact-seed shadow trial PR #50227 (one llama4.py change): selected 235/307 = 227 uncovered/fail-closed + 8 evidence-backed; safely skipped 72 jobs (~23%) with 96 inspectable why-chain rows.
• Exact-seed shadow trial PR #48939: tests/quantization/test_quark.py was unmapped, so selection fell back to run-all exactly as designed.

The #84672 child reports failed only because the dedicated trial gate deliberately exits nonzero when any shadow trial falls back; publication, read-back, both trial outputs, and result artifact upload completed first.

Known limits

This validates the collector/publisher/selector MVP for shadow-only use, not enforcement. On the representative #50227 trial, 227/307 jobs remained uncovered or fail-closed. Coverage must expand and false-negative audits must remain clean before any production skipping is considered.

Contribution notes

AI assistance was used in implementation and validation. The human submitter must review every changed line and understand the design before moving this PR out of draft.