Recipe-aware cleanup as a standalone Go test-automation binary

[lockwobr]

Summary

Replace the tools/cleanup bash script with a standalone Go test-automation binary that reuses AICR's own libraries (pkg/recipe, pkg/k8s) to clean up a cluster in a recipe-aware way.

This is explicitly not an aicr subcommand: cluster teardown is a CI / test-harness concern, not something the product CLI (snapshot → recipe → validate → bundle) should surface. The binary lives under tools/ (or cmd/) and is invoked by test automation and by developers resetting a dev cluster.

Follow-up to #1672 / PR #1773, which fixed the immediate data-loss bugs in the bash script (missed check-created namespaces; no exclusion for out-of-band installs). The bash fix stands on its own; this issue captures the better long-term design surfaced during that review.

Motivation

The current bash script hardcodes two lists that duplicate knowledge elsewhere and drift over time:

• AICR_NAMESPACES — a static copy of defaultNamespace entries from recipes/registry.yaml. Add a component, forget to update cleanup, and cleanup silently misses it.
• AICR_CRD_PATTERNS — a hand-maintained list of CRD API groups that lives nowhere structured today.

A Go tool can do better on three axes:

1. Recipe-scoped cleanup (the main prize). Resolve a recipe through pkg/recipe and clean up exactly what that recipe installs (namespaces + release names derived from the resolved component set). This solves tools/cleanup: misses check-created namespaces; deletes out-of-band installs #1672's part 2 at the root: a component the recipe excludes is never in the cleanup set, so there is nothing to fence out. The --exclude-* flags become a fallback rather than the primary safety mechanism.
2. Convention alignment. Use pkg/k8s/pod watch-based waits (the repo forbids poll-sleep), pkg/errors with codes, context timeouts, and errgroup fan-out for the independent read-only probes. No new distributed artifact beyond the Go toolchain we already build with.
3. Testability. Unit-test with a fake clientset + table-driven cases instead of the bash-stub harness (tools/cleanup_test.sh) added in PR fix(cleanup): backstop check namespaces; fence out-of-band installs #1773.

Design notes / open questions

• Registry gap: CRD API groups are not modeled. registry.yaml records helm repo/chart/version and defaultNamespace, but not the CRD groups a chart installs. To make CRD cleanup registry-driven rather than a hand-maintained list, add a crdGroups: [...] field per component in registry.yaml. This is a prerequisite enhancement.
• Helm uninstall. Reimplementing uninstall in Go means vendoring the Helm SDK (heavy) or still shelling out to the helm binary. Shelling to helm uninstall is likely the pragmatic choice ("boring first"); client-go handles CRDs, namespaces, and the finalizer-rescue.
• Preserve current behavior: driver-managed-cluster guidance (tools/cleanup leaves nvidia_uvm wedged on driver-managed GPU nodes (EKS) #1553), stuck-finalizer namespace rescue, --dry-run, --keep-crds/--keep-namespaces, and the exclusion semantics from tools/cleanup: misses check-created namespaces; deletes out-of-band installs #1672.
• Distribution: decide tools/ vs cmd/ and how test automation invokes it (Make target, direct go run, or a built binary in the e2e/kwok harnesses).

Acceptance criteria

• Recipe-aware cleanup: given a resolved recipe, clean up only its namespaces/releases.
• crdGroups field added to registry.yaml and consumed for CRD cleanup.
• Feature parity with the bash script (dry-run, keep-crds/keep-namespaces, exclusions, finalizer rescue, driver-managed guidance).
• Unit tests with a fake clientset; retire tools/cleanup_test.sh and the bash script.
• Wired into the test-automation paths that currently call tools/cleanup / make cleanup.