Skip to content

Commit a8c5ed7

Browse files
njhensleynjhensley
committed
feat(recipe-health): deep-link Evidence to dashboard + link-check bot
RQ1 (#1283): render the recipe-health Evidence column as a deterministic, hermetic deep-link into the evidence dashboard (validation.aicr.run) for each recipe with a committed dashboard presence; keep an honest `pending` otherwise. Link construction reuses the shared pkg/recipe.CoordinateFor mapping and carries no k8s version, so links are stable across version rolls. New pkg/testgrid owns the dashboard origin, link scheme, and the committed presence manifest (presence.yaml). Evidence-column prose in docs/user/recipe-health.md is rewritten to explain the link, the author-asserted-provenance caveat, and its coexistence with ADR-009's deferred in-cell freshness state. RQ2 (#1284): add tools/testgrid-link-check and a weekly, warning-only workflow (clone of bom-refresh minus the PR step) that fetches the live dashboard data with an origin-pinned, redirect-hardened, bounded HTTP client and reports any linked coordinate the dashboard no longer serves. Never blocks merges, never edits the doc; a coordinate not yet linked is expected, not a warning. Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
1 parent 6f9050b commit a8c5ed7

14 files changed

Lines changed: 1650 additions & 14 deletions

File tree

‎.github/workflows/testgrid-link-check.yaml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Weekly link-integrity check for the recipe-health Evidence deep-links
16+
# (RQ2 / #1284).
17+
#
18+
# Structurally a clone of bom-refresh.yaml (weekly schedule + workflow_dispatch,
19+
# read-only GITHUB_TOKEN, ubuntu-latest, timeout-minutes) with one deliberate
20+
# divergence: it never opens a PR and never edits a doc. Instead it runs
21+
# tools/testgrid-link-check, which reads the committed dashboard presence
22+
# manifest (pkg/testgrid/presence.yaml — the coordinates RQ1 deep-links) and
23+
# fetches the live dashboard data with a bounded, origin-pinned HTTP client
24+
# (off-origin redirects refused), then emits a Markdown report to
25+
# $GITHUB_STEP_SUMMARY. A linked coordinate the live dashboard no longer serves
26+
# is reported as a dead-link warning; a coordinate not yet linked is expected
27+
# and not a warning.
28+
#
29+
# Advisory only, mirroring the warning-only recipe-evidence gate: it never
30+
# blocks a merge (the job exits 0 even when warnings exist). Coordinate
31+
# resolution against the live SPA is necessarily separate from lychee, which
32+
# runs --offline and excludes design/ (fern-docs-ci.yaml) and so resolves no
33+
# dynamic URLs at all — this is the right home, not a lychee tweak.
34+
35+
name: TestGrid Link Check
36+
37+
on:
38+
schedule:
39+
# Mondays 07:00 UTC, offset from bom-refresh (06:00) and recipe-health-refresh
40+
# (06:30) so the three weekly jobs do not contend for runners at the same minute.
41+
- cron: "0 7 * * 1"
42+
workflow_dispatch: {}
43+
44+
# Serialize runs so a manual workflow_dispatch cannot race a scheduled run.
45+
concurrency:
46+
group: ${{ github.workflow }}
47+
48+
permissions:
49+
contents: read
50+
51+
jobs:
52+
check:
53+
name: Verify recipe-health Evidence deep-links against the live dashboard
54+
runs-on: ubuntu-latest
55+
timeout-minutes: 10
56+
steps:
57+
- name: Checkout Code
58+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
59+
with:
60+
persist-credentials: false
61+
62+
- name: Load versions
63+
id: versions
64+
uses: ./.github/actions/load-versions
65+
66+
- name: Set up Go
67+
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
68+
with:
69+
go-version: ${{ steps.versions.outputs.go }}
70+
cache: false
71+
72+
# No Helm setup: this job only reads an embedded manifest and fetches one
73+
# JSON document — it templates no chart and touches no cluster.
74+
75+
- name: Check Evidence deep-links and write report to step summary
76+
env:
77+
GOFLAGS: -mod=vendor
78+
# The tool exits 0 even when it finds dead links (warning-only), so the
79+
# job never fails the weekly run; the report is the signal.
80+
run: |
81+
go run ./tools/testgrid-link-check -report-out "$GITHUB_STEP_SUMMARY"

‎docs/user/recipe-health.md‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,13 @@ The matrix is computed **hermetically and offline**: every signal is a pure read
2929

3030
**Coverage** is a descriptor — it is *never* graded, so a deliberately minimal recipe is never penalized for declaring fewer checks. It is a compact per-phase summary of the **declared** validation checks, in the form `R:n D:n P:n C:n` — the count of named checks declared for the readiness, deployment, performance, and conformance phases respectively.
3131

32-
**Evidence** is a literal `pending` for every recipe today. No conformance attestations exist yet, so the column is honestly uniform: it reports the absence of evidence rather than overstating what is known. A differentiated, evidence-derived column lands once the first signed attestation does.
32+
**Evidence** deep-links each recipe that has a live dashboard presence to its coordinate on the AICR evidence dashboard at [validation.aicr.run](https://validation.aicr.run) (`/#/<group>/<dashboard>/<tab>`, where `group` is the service, `dashboard` is `<accelerator>-<os>`, and `tab` is `<intent>[-<platform>]`). A recipe without a presence stays a literal `pending` — the column reports the absence of live posture rather than overstating what is known. Today the dashboard covers only the UAT-driven coordinates, so most rows are `pending` until real-hardware coverage broadens; this matrix does not gate on that.
33+
34+
The link is constructed **hermetically and deterministically** from the recipe's resolved criteria using the shared `pkg/recipe.CoordinateFor` mapping — the generator makes no network call, and the coordinate carries **no Kubernetes version**, so a link is stable across k8s-version rolls. The Evidence cell is a link and nothing more: it points at the live board and carries **no** pass/fail state or count. Which recipes are linked is driven by a committed presence manifest (`pkg/testgrid/presence.yaml`); a weekly, warning-only bot keeps it honest by reporting any link that no longer resolves against the live dashboard data (it never blocks merges or edits this doc).
35+
36+
> **Community-source posture is author-asserted provenance, not independently-verified correctness.** A coordinate's tab placement and any community-sourced results on the dashboard are declared by the signing author; corroboration counts agreement at the same AICR version but does not itself certify runtime correctness. Read the linked board accordingly.
37+
38+
The deep-link is the current Evidence rendering. It is distinct from — and coexists with — [ADR-009](../design/009-recipe-health-tracking.md)'s deferred, verify-gated *in-cell freshness* state (`unattested` vs aged, derived from a signed attestation's `AttestedAt`): the link points at the live board, and an optional freshness token can later annotate the same cell without replacing the link. See also the [coverage matrix](coverage-matrix.md) for the complementary breadth view.
3339

3440
{/* BEGIN AICR-HEALTH */}
3541
## Summary
@@ -50,18 +56,18 @@ The matrix is computed **hermetically and offline**: every signal is a pure read
5056
| rtx-pro-6000-any | — | rtx-pro-6000 | — | — | — | pass | R:0 D:4 P:0 C:0 | pending |
5157
| monitoring-hpa | — | — | — | — | — | pass | R:0 D:0 P:0 C:0 | pending |
5258
| a100-aks-ubuntu-training-kubeflow | aks | a100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |
53-
| h100-aks-ubuntu-inference-dynamo | aks | h100 | ubuntu | inference | dynamo | pass | R:0 D:4 P:1 C:11 | pending |
54-
| h100-aks-ubuntu-training-kubeflow | aks | h100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:1 C:10 | pending |
59+
| h100-aks-ubuntu-inference-dynamo | aks | h100 | ubuntu | inference | dynamo | pass | R:0 D:4 P:1 C:11 | [aks/h100-ubuntu/inference-dynamo](https://validation.aicr.run/#/aks/h100-ubuntu/inference-dynamo) |
60+
| h100-aks-ubuntu-training-kubeflow | aks | h100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:1 C:10 | [aks/h100-ubuntu/training-kubeflow](https://validation.aicr.run/#/aks/h100-ubuntu/training-kubeflow) |
5561
| h100-aks-ubuntu-training-slurm | aks | h100 | ubuntu | training | slurm | pass | R:0 D:4 P:0 C:11 | pending |
5662
| bcm-inference | bcm | — | — | inference | — | pass | R:0 D:0 P:0 C:5 | pending |
5763
| h100-bcm-ubuntu-training | bcm | h100 | ubuntu | training | — | pass | R:0 D:4 P:0 C:5 | pending |
5864
| a100-eks-ubuntu-training-kubeflow | eks | a100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |
5965
| gb200-eks-ubuntu-inference-dynamo | eks | gb200 | ubuntu | inference | dynamo | pass | R:0 D:4 P:1 C:10 | pending |
6066
| gb200-eks-ubuntu-training-kubeflow | eks | gb200 | ubuntu | training | kubeflow | pass | R:0 D:4 P:2 C:8 | pending |
6167
| gb200-eks-ubuntu-training-slurm | eks | gb200 | ubuntu | training | slurm | pass | R:0 D:4 P:0 C:10 | pending |
62-
| h100-eks-ubuntu-inference-dynamo | eks | h100 | ubuntu | inference | dynamo | pass | R:0 D:4 P:1 C:11 | pending |
68+
| h100-eks-ubuntu-inference-dynamo | eks | h100 | ubuntu | inference | dynamo | pass | R:0 D:4 P:1 C:11 | [eks/h100-ubuntu/inference-dynamo](https://validation.aicr.run/#/eks/h100-ubuntu/inference-dynamo) |
6369
| h100-eks-ubuntu-inference-nim | eks | h100 | ubuntu | inference | nim | pass | R:0 D:4 P:0 C:11 | pending |
64-
| h100-eks-ubuntu-training-kubeflow | eks | h100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:1 C:10 | pending |
70+
| h100-eks-ubuntu-training-kubeflow | eks | h100 | ubuntu | training | kubeflow | pass | R:0 D:4 P:1 C:10 | [eks/h100-ubuntu/training-kubeflow](https://validation.aicr.run/#/eks/h100-ubuntu/training-kubeflow) |
6571
| h100-eks-ubuntu-training-slurm | eks | h100 | ubuntu | training | slurm | pass | R:0 D:4 P:0 C:11 | pending |
6672
| h200-eks-inference | eks | h200 | — | inference | — | pass | R:0 D:4 P:0 C:5 | pending |
6773
| h200-eks-training | eks | h200 | — | training | — | pass | R:0 D:4 P:1 C:10 | pending |
@@ -70,8 +76,8 @@ The matrix is computed **hermetically and offline**: every signal is a pure read
7076
| a100-gke-cos-training-kubeflow | gke | a100 | cos | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |
7177
| b200-gke-cos-inference-dynamo | gke | b200 | cos | inference | dynamo | pass | R:0 D:4 P:0 C:11 | pending |
7278
| b200-gke-cos-training-kubeflow | gke | b200 | cos | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |
73-
| h100-gke-cos-inference-dynamo | gke | h100 | cos | inference | dynamo | pass | R:0 D:4 P:1 C:11 | pending |
74-
| h100-gke-cos-training-kubeflow | gke | h100 | cos | training | kubeflow | pass | R:0 D:4 P:1 C:10 | pending |
79+
| h100-gke-cos-inference-dynamo | gke | h100 | cos | inference | dynamo | pass | R:0 D:4 P:1 C:11 | [gke/h100-cos/inference-dynamo](https://validation.aicr.run/#/gke/h100-cos/inference-dynamo) |
80+
| h100-gke-cos-training-kubeflow | gke | h100 | cos | training | kubeflow | pass | R:0 D:4 P:1 C:10 | [gke/h100-cos/training-kubeflow](https://validation.aicr.run/#/gke/h100-cos/training-kubeflow) |
7581
| h100-gke-cos-training-slurm | gke | h100 | cos | training | slurm | pass | R:0 D:4 P:0 C:11 | pending |
7682
| h100-kind-inference-dynamo | kind | h100 | — | inference | dynamo | pass | R:0 D:4 P:0 C:11 | pending |
7783
| h100-kind-training-kubeflow | kind | h100 | — | training | kubeflow | pass | R:0 D:4 P:0 C:10 | pending |

‎pkg/testgrid/doc.go‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
// Package testgrid maps a recipe's canonical coordinate to the AICR evidence
16+
// dashboard (validation.aicr.run) and answers "does this coordinate have a
17+
// dashboard presence?" — the two facts RQ1 (#1283) and RQ2 (#1284) share.
18+
//
19+
// It is deliberately thin and owns no mapping of its own: the recipe →
20+
// coordinate mapping is pkg/recipe.CoordinateFor (the single shared function
21+
// consumed by GP4/GP5/TG2/RQ1). testgrid only adds the dashboard host, the
22+
// hash-routed deep-link scheme built around Coordinate.Path, and the committed
23+
// presence manifest.
24+
//
25+
// Two presence sources, one spelling of a coordinate path:
26+
//
27+
// - Committed (hermetic): presence.yaml, embedded here and read offline by
28+
// the recipe-health generator so link *construction* never touches the
29+
// network. LoadPresence exposes it as a Presence set.
30+
// - Live: the published data/index.json the dashboard renderer boots from.
31+
// LivePaths extracts the present coordinate paths from a parsed index so
32+
// the warning-only link-check bot can compare the committed links against
33+
// what the dashboard actually serves.
34+
package testgrid

‎pkg/testgrid/presence.yaml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Dashboard presence manifest (RQ1 / #1283).
16+
#
17+
# The committed set of TestGrid/validation.aicr.run coordinate paths that
18+
# currently carry at least one real presence on the evidence dashboard. The
19+
# recipe-health generator (tools/health) reads this list to decide which
20+
# Evidence cells become a deterministic deep-link into the dashboard and which
21+
# stay an honest `pending` — it stays hermetic (no network) because presence is
22+
# read from this committed file, never fetched.
23+
#
24+
# Each entry is a "<group>/<dashboard>/<tab>" coordinate path, exactly as
25+
# pkg/recipe.CoordinateFor(criteria).Path() renders it: group=service,
26+
# dashboard=<accelerator>-<os>, tab=<intent>[-<platform>].
27+
#
28+
# Today the dashboard only covers the UAT-driven recipes; most recipes stay
29+
# `pending` until real-hardware coverage broadens. This file is the single
30+
# source of truth for "linked" — the weekly, warning-only testgrid-link-check
31+
# bot (RQ2 / #1284) fetches the live dashboard data and warns when an entry
32+
# here no longer resolves, keeping the manifest honest without blocking merges.
33+
#
34+
# To add a coordinate: confirm https://validation.aicr.run/#/<path> shows real
35+
# data, then add the path below (keep the list sorted).
36+
coordinates:
37+
- aks/h100-ubuntu/inference-dynamo
38+
- aks/h100-ubuntu/training-kubeflow
39+
- eks/h100-ubuntu/inference-dynamo
40+
- eks/h100-ubuntu/training-kubeflow
41+
- gke/h100-cos/inference-dynamo
42+
- gke/h100-cos/training-kubeflow

0 commit comments

Comments
 (0)