Readiness gate keys off non-monotonic Skyhook status, not the tuning taint → flaky post-gate failures

[njhensley]

Summary

The GPU-stack readiness gate can certify a cluster "ready" and then have skyhook/nodewright node tuning re-open, producing intermittent downstream failures in the conformance phase. Root cause: the gate keys off a non-monotonic signal (Skyhook status.status) rather than the durable terminal signal (the tuning NoSchedule taint being removed). This is the RCA of the two UAT-AWS expected-resources failures that motivated the debug bundle (#1764).

Evidence

Run 29393800033, one expected-resources run, the same Skyhook/tuning CR:

• 07:03:53 — Go pre-flight printed Nodewright tuning: complete (stable ≥1m0s) (i.e. status.status == complete continuously for 60s) → passed.
• 07:08:53 — the chainsaw step validate-skyhook-cr-complete failed reading status.status: in_progress (persisted through its 5m assert budget).

So the same CR went complete → in_progress within one check. Run 29402956254 is the same race with the tuning reboot evicting the check Job's pod (pod for job not found).

Why the signal flaps

The tuning Skyhook (recipes/components/nodewright-customizations/manifests/tuning.yaml) is a multi-package chain where nvidia-tuned (L105) and nvidia-setup-full (L126) both carry interrupt: reboot. status.status is an aggregate over (packages × matching GPU nodes), and the cluster runs 2 GPU nodes — so it re-opens to in_progress on each package reboot and per newly-joined node.

Why the current gate doesn't close it

Both gate layers watch status.status, not the taint:

• Runner gate — tests/uat/aws/run#L307: validate --phase deployment until it passes READINESS_CONSECUTIVE_PASSES (=2) times consecutively.
• Go pre-flight — verifyNodewrightReady / pollUntilStable polls status.status (read at L505) until "complete" stable for GPUReadinessStabilityWindow = 60s.

Both are probabilistic settling heuristics — "quiet at N sampled instants." They cannot guarantee no further reboot, and the runner gate closes in phase_install minutes before phase_conformance opens a fresh run; a reboot in that gap is invisible. The code even acknowledges the non-monotonicity (expected_resources.go#L442).

Proposed direction

Gate on the durable terminal signal instead of the flapping status:

1. Gate on taint-absence being stable — skyhook applies a skyhook.nvidia.com NoSchedule taint during tuning and removes it on completion. If that removal is skyhook's monotone terminal step (i.e. it does not re-taint for a later package/reboot in the same plan), "taint absent + stable" is a stronger "done, won't reboot again" signal than status.status.
2. And/or give the chainsaw validate-skyhook-cr-complete assert (recipes/checks/nodewright-customizations/health-check.yaml#L31) the same poll-until-stable ride-through budget the Go barrier has, so it doesn't fail on a transient re-open.
3. Possibly lengthen GPUReadinessStabilityWindow so "stable" better predicts no re-tuning.

Open questions (needs data)

• Is skyhook's taint removal monotone-terminal within a plan? If skyhook re-taints between packages (nvidia-setup-kernel → nvidia-tuned → nvidia-setup-full, each a reboot), taint-absence would flap too and (1) buys nothing. This must be confirmed against skyhook's behavior before changing the gate.
• The debug bundle (feat(ci): collect UAT cluster debug bundle on failure #1764) now captures the per-node/per-package Skyhook status timeline + node reboot fingerprint + taint state at failure, which is exactly the data needed to answer this on the next occurrence.

References

• Diagnostic tooling: feat(ci): collect UAT cluster debug bundle on failure #1764 (debug bundle). Runs: 29393800033, 29402956254.
• Related prior work: commit d47ce33 (poll GPU readiness to ride through reboots) added the current pollUntilStable mitigation.