GitOps drift detection + Argo CD API deep diff

Two coordinated capabilities for the "drift between deployments and GitOps"
ask, credential-free detection plus a hardened connected deep-diff view.

## Detection (no credentials)

- Manual-sync Applications continuously OutOfSync for >24h reach the issues
  stream (gitops_out_of_sync, warning). Drift onset is tracked Radar-side per
  app UID on the ResourceCache (Argo records no sync-status transition time);
  a restart resets the clock — deliberately conservative.
- New gitops_stale category: a down application-controller emits one critical
  rollup ("sync verdicts are frozen"), a majority-stale fleet (>=3 apps) one
  warning rollup, isolated stale apps per-app warnings. Threshold max(30m,
  10x argocd-cm timeout.reconciliation); controller pods found by label across
  all namespaces.
- Cluster Audit: gitops-unmanaged-workload / gitops-helm-only-workload flag
  top-level workloads outside GitOps via the pkg/subject overlay tiers. Gated
  on real GitOps root objects existing (not just CRDs); ownerRef'd workloads
  skipped; Argo label-tracking recognized by cross-referencing app names.
- GitOps detail page discloses spec.ignoreDifferences as comparison coverage
  (rule/jq/managed-fields counts + kinds).

## Argo CD API deep diff (connected)

- pkg/argoapi: pure REST client for argocd-server (filtered managed-resources,
  session/userinfo probe, version reachability, CLI-config token extraction);
  no internal/ imports, so radar-hub can reuse it.
- internal/argocd: discovery by service label, argocd-owned port-forward for
  out-of-cluster runs, 15s TTL cache, lazy self-reconnect, context-switch reset.
- GET /api/argo/applications/{ns}/{name}/resource-diff: dual RBAC gate
  (Application namespace access + per-kind get SAR) before any upstream call;
  Git-rendered desired vs normalized live YAML + field entries.
- UI: Settings "Argo CD" card (URL/token/insecure-TLS/CLI-session), inline
  unified diff on the expanded change row with a full-screen maximize, and a
  "Connect Argo CD" hint when disconnected.

## Security model (3 review rounds, all findings fixed)

- Token bound to its origin AND kubeconfig context: changing the URL or
  switching context requires re-entering it; the probe runs off an immutable
  snapshot (URL + token + TLS + frozen k8s client/config), fails closed if a
  switch races the capture, and commits only under a generation check — the
  token never reaches a different cluster's argocd-server.
- Secret redaction is fail-closed: data/stringData/binaryData masked in any
  shape, annotation values masked, last-applied/managedFields/status stripped
  before serialization. No un-redact.
- No credential or secret in responses or logs: token redacted from GET
  /api/config and probe logs; upstream Argo error bodies never reach responses.

Verified: full Go suite + tsc green both modules; real-cluster validation
against nonprod GKE Argo (discovery/port-forward/token-verify round-trip);
diff render + redaction + UI verified on the gitops-demo kind cluster.

Author: nadaverell

.codex-sky1100-precision-plan.md

@@ -0,0 +1,76 @@
+# SKY-1100 follow-up precision plan
+
+Task: continue PR #1067 by implementing the high-priority precision fixes identified by the orphan ConfigMap/Secret competitor comparison and product review.
+
+Premise and scope:
+
+- This is detector precision work for the `orphanConfigMapSecret` audit check. The dangerous failure mode is a false-positive cleanup recommendation for a ConfigMap/Secret that is actually consumed by convention or platform/controller behavior.
+- Keep this follow-up intentionally narrow. Implement the three live-supported, high-value suppressions:
+  1. Dynamic-loader ConfigMaps labeled for Grafana/Prometheus/fluentd/K8sGPT dynamic consumption.
+  2. ConfigMap leader-election state with the canonical `control-plane.alpha.kubernetes.io/leader` annotation.
+  3. Narrow GKE/GMP managed add-on Secret suppression for the sampled `gmp-system/webhook-tls` class.
+- Do not build low-evidence/niche handlers in this PR: SecretProviderClass, Volcano command-line refs, Argo Workflows/CNPG controller config, or cert-manager `Certificate.status.nextPrivateKeySecretName`. Those were either source-derived only, 1-2 sampled objects, or likely label-covered in real cert-manager Secrets.
+
+Current-code evidence:
+
+- `pkg/audit/checks.go:808-829` emits orphan ConfigMap findings after checking referenced refs, `kube-root-ca`, `isKnownPlatformConfigMap`, and controller owner references.
+- `pkg/audit/checks.go:831-858` emits orphan Secret findings after checking references, service-account tokens, Helm release Secrets, cert-manager certificate labels, `isKnownPlatformSecret`, and owner refs.
+- `pkg/audit/checks.go:864-935` contains the ConfigMap platform suppressor. It has narrow GKE/GMP ConfigMap names and controller platform cases, but no dynamic-loader label or leader-election annotation handling.
+- `pkg/audit/checks.go:937-975` contains the Secret platform suppressor. It has SealedSecrets, Argo CD, cert-manager webhook/account-key, `gmp-public/alertmanager`, and Crossplane root CA suppressions, but no `gmp-system/webhook-tls` managed add-on case.
+- `pkg/audit/checks_test.go:1287-1480` is the existing platform-artifact positive/negative corpus for `orphanConfigMapSecret`; this is the right place to add the new suppressions and same-looking negative cases.
+- `pkg/audit/checks_test.go:1523-1802` already pins precision paths for workload refs, terminal jobs/pods, ServiceAccount imagePullSecrets, ephemeral containers, and CRD refs.
+- The comparison report confirmed the live evidence:
+  - `/private/tmp/radar-orphan-compare/reports/final-orphan-tool-comparison-report-v2.md:104-121` identifies 31 dashboard-looking Radar findings, `datadog-operator-lock`, and `gmp-system/webhook-tls` as concrete precision gaps.
+  - `/private/tmp/radar-orphan-compare/reports/final-orphan-tool-comparison-report-v2.md:146-148` ranks dynamic-loader labels, leader-election annotations, and narrow GKE/GMP managed add-on suppression as the first recommended changes.
+  - `/private/tmp/radar-orphan-compare/reports/source-logic-report.md:150` names K8sGPT's dynamic-loader labels: `grafana_dashboard`, `grafana_datasource`, `prometheus_rule`, `fluentd_config`, and `k8sgpt.ai/dynamically-loaded=true`.
+
+Certain:
+
+- Add a small helper in `pkg/audit/checks.go`, near `labelsValue`, for label/annotation predicates rather than scattering string checks through the suppressors.
+- For ConfigMaps, extend `isKnownPlatformConfigMap` to suppress:
+  - label keys `grafana_dashboard`, `grafana_datasource`, `prometheus_rule`, `fluentd_config` when present with a non-empty value other than clear false values (`false`, `0`, `no`).
+  - label `k8sgpt.ai/dynamically-loaded` only when the value is truthy (`true`, `1`, `yes`), because that label is explicit state rather than a generic loader selector.
+  - annotation `control-plane.alpha.kubernetes.io/leader` when non-empty.
+- For Secrets, extend `isKnownPlatformSecret` to suppress only the GKE/GMP sampled shape: namespace `gmp-system`, name `webhook-tls`, and strong managed add-on evidence via either `addonmanager.kubernetes.io/mode=Reconcile` or a `components.gke.io/` metadata key. Do not suppress every `webhook-tls` or every `gmp-system` Secret.
+- Add tests to `TestOrphanConfigMapSecretSkipsKnownPlatformArtifacts` for representative dynamic-loader labels, leader-election annotation, and the GKE/GMP `webhook-tls` Secret.
+- Add negative tests to `TestOrphanConfigMapSecretKnownPlatformArtifactNegativeCases` for:
+  - a dashboard-looking name with no dynamic-loader label (e.g. `koala-grafana-dashboards`) remains flagged.
+  - a `grafana_dashboard=false` ConfigMap remains flagged.
+  - a `control-plane.alpha.kubernetes.io/leader` annotation with an empty value remains flagged.
+  - a `gmp-system/webhook-tls` Secret without managed add-on evidence remains flagged.
+
+Confident:
+
+- No registry copy change is needed. The current copy says "supported controller configuration" (`pkg/audit/registry.go:285-291`); these convention suppressions are part of the supported platform/controller carve-out, not a new user-facing concept.
+- No visual-test is needed. This is backend audit logic only; the UI surfaces fewer findings but no rendered behavior/layout changes.
+- No live mutation is needed. Live validation can be done by building/running the PR binary against already-authenticated clusters and comparing the orphan check output count/names before/after, but unit tests are the main preservation proof.
+
+Needs-input:
+
+- None expected. The product decision to keep this precise and not add broad generic scanning was already settled in the product review. If Claude disagrees and argues for adding a wider mechanism or a user-facing opt-out annotation now, treat that as Discuss and stop.
+
+Implementation sequence:
+
+1. Edit `pkg/audit/checks.go`:
+   - Add metadata helpers: `metadataValue`, `hasMetadataKeyPrefix`, `metadataValueEqualFold`, and a compact dynamic-loader predicate. Keep names local and explicit.
+   - Update `isKnownPlatformConfigMap` with dynamic-loader and leader-election checks after nil/explicit name checks but before controller-specific name allowlists.
+   - Update `isKnownPlatformSecret` with the narrow `gmp-system/webhook-tls` managed add-on check.
+2. Edit `pkg/audit/checks_test.go`:
+   - Extend the positive platform fixture list and expected suppressed keys.
+   - Extend negative cases for false-looking labels and insufficient GKE/GMP evidence.
+3. Run focused tests:
+   - `cd pkg && go test ./audit -run 'TestOrphanConfigMapSecret'`
+4. Run broader repo checks before PR update:
+   - `make test`
+   - `make build`
+   - `make tsc` is redundant if `make build` includes frontend build, but run it if the repo command expects it separately in the QA ledger.
+   - `visual-test: skipped (backend audit logic only)`.
+5. Optional live validation after tests:
+   - Build/run PR binary and query `/api/audit?raw=true` on a GKE cluster that previously had dashboard and `gmp-system/webhook-tls` findings.
+   - Verify the specific dynamic-loader/leader/GMP findings are absent and ordinary unlabeled dashboard-ish rows remain if present.
+
+Risk / blast radius:
+
+- Affected surface: only the `orphanConfigMapSecret` audit check.
+- Main risk: over-suppression, especially if a user labels an actually-unused ConfigMap with a dynamic-loader key by accident. Mitigation: require precise known keys and false-value negative tests; do not suppress by name pattern alone.
+- Residual risk: unlabeled dynamic dashboard ConfigMaps remain possible false positives; this is intentional because suppressing by name would hide real stale ConfigMaps.

AUDIT-FINDINGS.md

@@ -0,0 +1,52 @@
+# Radar UX/diagnosis audit — findings (with repro locations)
+
+Severity: B=blocker H=high M=medium N=nit | [+]=highlight. Screenshots: `.playwright-mcp/vt-audit/`.
+Clusters: KIND = kind-radar-gitops-demo · GCP = gke_koalabackend_us-east1-b_nonprod-cluster-us-east1 · AWS = (below)
+
+| # | Sev | Finding | Where (cluster / resource / URL) | Fix area |
+|---|-----|---------|----------------------------------|----------|
+| A | B | PVC drawer stacks a CRITICAL "StorageClass doesn't exist" issue above a blue "Pending — this is normal, expected indefinitely" note that's factually wrong here AND visually louder. Reassurance dominates the diagnosis. | KIND · PVC radar-demo-pointers/data · /resources/persistentvolumeclaims?resource=radar-demo-pointers/data | FE: PVCRenderer.tsx:53-59 (suppress/soften note when a critical pending issue exists) |
+| B | H | On a ROOT resource the forward blast-radius block is labeled "CONTEXT · Possible cause" (reads as *its* cause). Service no-endpoints correctly says "Affected" — so it's specific to candidate-role facts. | KIND · same PVC drawer | FE: ResourceIssuesSection CausalContext / diagnostic.ts role label |
+| C | M | Remediation leads with the unlikely fix: "Create StorageClass 'does-not-exist'" — a typo'd name; should foreground "point at an existing class / remove". Same on Ingress ("Create Service 'ai-temp'"). | KIND PVC data · GCP Ingress dev/ai-temp | BE: detect_missing_refs.go action copy order |
+| 1 | H | Symptom fan-out, no grouping: ~21 rows = ~5 incidents (one bad Helm chart → rollout+readiness+no-endpoints+helm-failed ×3). | KIND · /issues | (incident grouping feature) |
+| 1b | N | "since deploy" age on 8+ rows collapses recency signal. | KIND · /issues | FE issues list |
+| 1c | N | Category chips (Control plane/Scaling/…) low-contrast grey; weak scan axis. | KIND+GCP · /issues | FE/theme |
+| 2 | H | HPA "Autoscaling limited" dumps raw 4-level k8s error, NO Next step, expanded "What's wrong" just echoes collapsed line. Real cause ("metrics-server down") buried. | KIND · HPA demo-flux/podinfo · /issues (expand) | BE: HPA diagnosis parser (hpadiag) + issue cause/action |
+| 3 | H | Argo diagnosis inconsistent: guestbook-drift = gold plain-language; guestbook-broken-path = raw gRPC dump "rpc error: code = Unknown desc =". | KIND · Application argocd/guestbook-broken-path & guestbook-drift | BE: gitops error parsing |
+| 4 | M | Helm failure leaks condition-speak: "status: InProgress… context deadline exceeded". | KIND · HelmRelease radar-helm-agent-demo/native-failed-demo | BE: helm diagnosis |
+| 5 | M | Over-reporting: needs-storage shows 3 stacked issues for one root (unschedulable→PVC, workload_degraded, volume_access_conflict); RWO-conflict moot while PVC can't bind. | KIND · Deployment radar-demo-pointers/needs-storage | BE: dedupe/suppression |
+| 6 | M | Missing-ref FALSE POSITIVE: flagged kindnet node hostname (env CONTROL_PLANE_ENDPOINT=…control-plane:6443) as a missing Service. | KIND · DaemonSet kube-system/kindnet · /issues | BE: detect_missing_refs env-Service heuristic (exclude host:port / non-Service) |
+| 7 | N | Resources browser: CRD group shown as raw "apiregistration.k8s.io" beside friendly "Flux"/"Argo"; node CPU/Mem cols empty "-" no "metrics unavailable" hint. | KIND+GCP · /resources | FE resource browser sidebar / table |
+| 8 | M | 3 JS console errors fire on opening the Node drawer (likely metrics fetch). | KIND · Node radar-gitops-demo-control-plane | FE: NodeRenderer metrics fetch error handling |
+| 9b | + | Service no-endpoints drawer correctly shows forward "Context · Affected · Backend pods · high confidence". | KIND · Service radar-helm-agent-demo/native-failed-demo-radar-agent-bad-chart | — (keep) |
+| 10 | + | Cluster picker: clean provider/account grouping (GKE·acct, EKS·acct, Other). Nit: EKS shows raw ARNs as subtitles. | any · header context switcher | FE (nit) |
+| 12 | H | Chronic criticals dominate: dangling-ref criticals aged 84d/93d/112d/537d/881d all flat "Critical", no aging-down/ack. Real-cluster Issues page = graveyard. | GCP · /issues (HTTPRoutes, Ingresses, webhooks) | (feature: acute-vs-chronic / ack / age-decay severity) |
+| 13 | M | Double-comma formatting bug + raw scheduler jargon: "…no new claims to deallocate,, 1 node(s) no free host ports". | GCP · DaemonSet prometheus-node-exporter/prometheus-node-exporter · /issues | BE: scheduler-reason join/format + plain-language |
+| 14 | M | DaemonSet partial-unschedulable flagged Critical (node-exporters can't place on some nodes by design). | GCP · DaemonSet prometheus-node-exporter & opencost/prometheus-prometheus-node-exporter | BE: scheduling detector (daemonset-by-design tolerance) |
+| 15 | H | Webhook-backend-down ignores failurePolicy: 881d "datadog webhook backend missing" = Critical; if failurePolicy=Ignore it's benign. | GCP · MutatingWebhookConfiguration datadog-webhook & sinkbindings.webhook.sources.knative.dev | BE: webhook detector (read failurePolicy → severity) |
+| 16 | + | Ingress drawer: good parsed cause + Next step. Nits: leads with "Create Service" (route likely obsolete); "1 certificate(s)". | GCP · Ingress dev/ai-temp | FE/BE copy |
+| 17 | + | Secrets LIST data-safe: TYPE/KEYS-count/EXPIRES/AGE, no values; inline TLS expiry is a highlight. (Verify many "Expired" are real.) | GCP · /resources/secrets (ns dev) | — (keep) |
+| 18 | + | Secret DRAWER: values redacted (security ✓), cert parsing nice. Verify possible inverted cert dates (Not Before > Expires). | GCP · Secret dev/app-tls-cert | FE: cert date display (verify) |
+| 19 | N | Deployments list good (READY/IMAGES-with-tags). CRD-group sidebar naming INCONSISTENT: friendly "CERT MANAGER/GATEWAY API/KARPENTER" beside raw "MONITORING.GOOGLEAPIS.COM/GATEWAY.ENVOYPROXY.IO/NETWORKING.GKE.IO". (extends #7) | GCP · /resources/deployments | FE resource browser group-label map |
+| 20 | M/+ | Deployment drawer rich+good (Status/Strategy/Conditions/Restart/Rollback/Logs). "Permissions via ServiceAccount default" renders a dense ~40-chip wall — verify risky perms are prioritized/highlighted, not just dumped. | GCP · Deployment dev/koala-backend | FE: RBAC permissions section density/prioritization |
+| 21 | H | Connection-error state visually good BUT mis-diagnoses cause: expired AWS SSO (exec-plugin auth fail) shown as "Connection Timed Out / cluster under heavy load / network latency / check cluster health" — points user at cluster/network, not re-auth. Doesn't distinguish auth failures from timeouts. | AWS us-east-1-nonprod (SSO expired) · /issues | BE/FE: connection error classification (auth vs timeout) |
+| 22 | +/M | Checks page well-designed (remediation queue, grouped by check, framework chips). But: "Unused ConfigMap or Secret — 137 resources" dominates + is classic noisy/false-positive check; "Potential secret in ConfigMap — 25" heuristic (verify precision); "Service has no matching pods" (Check) overlaps "No endpoints" (Issue) — conflation across the two axes CLAUDE.md says to keep separate. | GCP · /audit (ns dev) | BE audit checks (unused precision; dedupe vs issues) |
+| 23 | M | Topology default renders 723 nodes = hairball, low-insight without a focus/entry point. Error nodes do highlight red. | GCP · /topology (ns dev) | FE topology default scoping |
+| 24 | + | Pod drawer comprehensive+clean (Status/Containers/Env/Resource-Usage charts/Conditions + Terminal/Logs/Forward). Verify env-var secret redaction. | GCP · Pod dev/app-6dfc8f5cbb-sbsfj | — (keep; verify env redaction) |
+
+| 25 | + | Namespace drawer RBAC section (RoleBindings + ClusterRoleBindings touching ns, with subject chips) = useful security surface. Namespace LIST shows red 'Terminating' badges on several ns (argocd/automation/…) — verify real stuck-terminating vs noise. | GCP · Namespace dev · /resources/namespaces | FE (keep); verify terminating badges |
+
+| 26 | H | AWS EKS queue DOMINATED by "Webhook backend down" flood (8+ criticals: cert-manager/kube-system/validating webhooks → missing services). Real-cluster proof that #15 (webhook ignoring failurePolicy / stale uninstalled-operator webhooks) is a top-of-queue noise generator, not a nit. + PVC pending (StorageClass "low-cost"). | AWS us-east-1-nonprod · /issues | BE: webhook detector (failurePolicy + service-resolve) — HIGH PRIORITY |
+| 26b | M | Connection-error UX: (1) header shows GREEN connected dot while body says "Cannot Connect"; (2) after creds restored (API 200), the error state is STICKY — doesn't auto-recover, needs manual reload. | AWS us-east-1-nonprod (during reconnection) | FE: connection state sync + auto-recover |
+| 27 | N/+ | Dark mode parity good on Issues view. Theme toggle works (hover tooltip) but top-right header icon buttons (theme/terminal/help/debug) have NO accessible names in a11y tree (title-only); also ambiguous which does what (clicked theme, got Terminal). | any · header top-right | FE a11y: aria-labels on header icon buttons |
+
+| 28 | + | GitOps view clean: sync/health/automation filters, summary chips, source/dest/last-sync table; good dark-mode parity. Note several apps 'Pending: Nmo ago' last-sync (chronic-staleness, reinforces #12). | AWS · /gitops | — (keep) |
+
+| 29 | +/M | Helm view clean (Installed/Catalog, status badges, inline "7/7 ready"). karpenter & mcp-s show "failed" for 118-375d while running — stale failed-upgrade not distinguished from active failure (chronic; correctly NOT in Issues). | AWS · /helm | FE/BE: helm status staleness label |
+| 30 | M | crossplane-demo cluster DOWN (connection refused) — couldn't audit Crossplane CRDs. BUT its error state is CORRECTLY classified ("Network Unreachable / connection refused / confirm cluster running"). Refines #21: classifier works for connection-refused; only mis-buckets auth-expiry (which surfaces as context-deadline → "timeout/heavy load"). | KIND crossplane-demo (down) | (informs #21 fix) |
+| 31 | + | HTTPRoute drawer excellent: inline "Unresolved References" AlertBanner + unresolved backend flagged red in Rules + "Unresolved Refs" in Parent Status. Mild redundancy (same problem 3 places). | GCP · HTTPRoute staging/node-js-api-2 | — (keep) |
+| 32 | M | Gateway list good (STATUS Programmed/CLASS/LISTENERS/ROUTES/LB-addr). CONFIRMS #7/#19 at scale: sidebar mixes friendly CRD groups (KARPENTER/GATEWAY API/CERT MANAGER/EXTERNAL SECRETS/DYNAMIC RESOURCE ALLOCATION) with RAW (POLICIES.KYVERNO.IO/NETWORKING.GKE.IO/APIREGISTRATION.K8S.IO/WARDEN.GKE.IO/CLOUD.GOOGLE.COM). Friendly-name map ~half-covered. | GCP · /resources (sidebar) | FE: complete the CRD-group friendly-name map / fallback formatting |
+
+## Coverage notes
+- Clusters audited: KIND (gitops-demo, GitOps/Helm-heavy), GCP nonprod (real workloads, networking-heavy). AWS us-east-1-nonprod = AUDITED (webhook-down flood — see #26).
+- Not yet audited (offer to continue for literal 30-40): AWS (post-auth), crossplane-demo CRDs, HTTPRoute/Gateway drawer, Namespace drawer (RBAC), ConfigMap drawer (consumed-by), CronJob/Job, GitOps Application detail nested pages, Helm release detail, dark mode parity.