GitOps drift detection: manual-sync gate, stale comparisons, coverage audit, ignoreDifferences disclosure#1136
GitOps drift detection: manual-sync gate, stale comparisons, coverage audit, ignoreDifferences disclosure#1136nadaverell wants to merge 3 commits into
Conversation
… audit, ignoreDifferences disclosure
Four detection-completeness gaps for Argo CD drift, all credential-free:
- Manual-sync Applications continuously OutOfSync for >24h now reach the
issues stream (gitops_out_of_sync, warning). Drift onset is tracked
Radar-side per app UID on the ResourceCache (Argo records no sync-status
transition time); a restart resets the clock — deliberately conservative.
- New gitops_stale category: a down application-controller emits one
critical rollup on the controller workload ("sync verdicts are frozen"),
a majority-stale fleet (≥3 apps) one warning rollup, and isolated stale
apps per-app warnings — threshold max(30m, 10× argocd-cm
timeout.reconciliation). Controller pods found by label across all
namespaces, not a hard-coded "argocd".
- Cluster Audit: gitops-unmanaged-workload / gitops-helm-only-workload flag
top-level workloads outside GitOps via the pkg/subject overlay tiers.
Gated on a GitOps controller actually being installed; ownerRef'd
workloads skipped (tracking doesn't propagate to controller children).
- GitOps detail page disclosures spec.ignoreDifferences as comparison
coverage (rule/jq counts + kinds) — jq rules aren't evaluated by Radar,
so the drift panel can show entries Argo's UI suppresses.
- Coverage gate now requires actual GitOps root objects, not CRD registration — leftover CRDs after an Argo/Flux uninstall no longer flag every workload as unmanaged. - Argo label-tracking mode (application.resourceTrackingMethod: label) recognized: app.kubernetes.io/instance naming a real Application counts as GitOps-managed; the Application names ride CheckInput.ArgoAppNames. - Manual-drift tracker purge moved to the cluster-wide pass only — namespace-scoped polls no longer drop other namespaces' drift clocks. - ignoreDifferences disclosure counts managedFieldsManagers rules as unsupported alongside jq (jqRuleCount → unsupportedRuleCount).
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| argoAppNames = make(map[string]struct{}, len(apps)) | ||
| for _, app := range apps { | ||
| argoAppNames[app.GetName()] = struct{}{} | ||
| } |
There was a problem hiding this comment.
App names ignore namespace
Medium Severity
gitOpsRoots and argoAppTracksByLabel key Argo Applications by metadata.name only. Kubernetes allows the same Application name in different namespaces, so an unrelated app.kubernetes.io/instance value can match and skip the unmanaged-workload finding for workloads Argo does not manage.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2bb56bd. Configure here.
…tection # Conflicts: # packages/k8s-ui/src/components/gitops/insights/GitOpsInsightViews.tsx
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
There are 4 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8dfa00f. Configure here.
| "warning", "GitOpsComparisonsStale", | ||
| fmt.Sprintf("%d of %d Applications have stale sync/drift comparisons — the application-controller may be overloaded or wedged", len(stale), len(eligible)), 0) | ||
| d.Action = "Check the application-controller for reconcile backlog or throttling; individual Applications' verdicts are older than expected." | ||
| return []Detection{d} |
There was a problem hiding this comment.
Rollup issues reset issue age
Medium Severity
GitOpsControllerStalled and GitOpsComparisonsStale are built with zero DurationSeconds/AgeSeconds. fromProblem then sets FirstSeen to the compose time, so these rollups look newly opened on every refresh instead of reflecting how long comparisons have been frozen.
Reviewed by Cursor Bugbot for commit 8dfa00f. Configure here.
| if namespace == "" { | ||
| driftTracker(cache).purge(now, argoDriftRetain) | ||
| problems = append(problems, detectArgoStaleFromCache(cache, argoApps, now)...) | ||
| } |
There was a problem hiding this comment.
Namespace scope skips stale detection
Medium Severity
gitops_stale detection and drift-tracker purge run only when DetectGitOpsProblems is called with a cluster-wide namespace. The issues provider invokes it per selected namespace, so a typical namespace-filtered Issues view never emits controller-stale or fleet-stale rollups.
Reviewed by Cursor Bugbot for commit 8dfa00f. Configure here.
| // other namespaces' drift clocks and restart their 24h gate. | ||
| if namespace == "" { | ||
| driftTracker(cache).purge(now, argoDriftRetain) | ||
| problems = append(problems, detectArgoStaleFromCache(cache, argoApps, now)...) |
There was a problem hiding this comment.
Scoped compose never purges drift
Low Severity
Manual-sync drift tracking is purged only on the cluster-wide DetectGitOpsProblems pass. When issues compose stays namespace-scoped, deleted Applications can leave entries in argoDrift indefinitely because purge never runs.
Reviewed by Cursor Bugbot for commit 8dfa00f. Configure here.
|
Folded into #1138 — the two stacked branches were squashed into a single commit on |
1 participant
What & why
A customer asked for "drift between deployments and gitops". Radar already reads Argo's own per-resource comparison verdicts, but four detection gaps kept real drift invisible outside the detail page. This PR closes them — all credential-free (the Argo-API deep-diff view lands separately, stacked on this branch). Plan settled via cross-model plan review; part of the #601 arc.
1. Manual-sync drift reaches the issues stream
The issues engine deliberately skipped OutOfSync for non-auto-synced apps ("too chatty"). For a manual-sync shop that's the mainline drift case — it accumulates silently, fleet-wide. Now: a manual-sync Application observed continuously OutOfSync for >24h emits a
gitops_out_of_syncwarning (reasonOutOfSyncManual). Argo records no sync-status transition time, so drift onset is tracked Radar-side per app UID on the ResourceCache (context-switch drops it; restart resets the clock — deliberately conservative, documented). Fresh drift on a manual app stays a workflow state, not an issue.2. New
gitops_stalecategory: frozen sync verdictsArgo only re-compares on refresh — a down/wedged application-controller silently freezes every verdict ("Synced" lies). Two-level detection avoids the noise storm:
Threshold:
max(30m, 10× argocd-cm timeout.reconciliation). Controller pods found by label across all namespaces (not hard-codedargocd). Never-reconciled / terminating / mid-operation apps excluded.3. Cluster Audit: GitOps coverage
gitops-unmanaged-workload+gitops-helm-only-workload(independently toggleable) flag top-level workloads outside GitOps, classified via the existingpkg/subjectoverlay tiers — no new label heuristics. Gated on actual GitOps root objects existing (not CRD registration — leftover CRDs after uninstall must not flag the world). Argo's label-tracking mode is recognized by cross-referencingapp.kubernetes.io/instanceagainst real Application names. OwnerRef'd workloads skipped (tracking doesn't propagate to controller children). Native Helm = softer distinct finding.4. ignoreDifferences comparison-coverage disclosure
Detail page now discloses
spec.ignoreDifferences(rule count + kinds) as a neutral note, with an explicit warning when rules use jq expressions or managed-fields managers — Radar's drift filter doesn't evaluate those, so the panel can show entries Argo's UI suppresses.Wire/contract notes
gitops_staleis an additive issuesapi category (catalog + control_plane group + CEL-filterable). Consumers with exhaustive switches need awareness.issuesapiseverity contract unchanged (critical|warning; both new detections normalize into it).GitOpsInsightSummary.ignoredDifferencesis additive-optional on the k8s-ui types.Testing
Table-driven tests across all four features (drift-gate boundaries + state reset, stale rollup rules + threshold parsing + eligibility exclusions, coverage tier classification incl. label-tracking cross-reference + gate-closed, disclosure counts). Full suite + tsc green in both modules. Cross-model review ran pre-PR; its four findings (CRD-gate false positives, label-tracking miss, purge race, managedFieldsManagers disclosure) are fixed in the second commit.
Visual test: skipped — the only UI change is a small conditional text note in an existing disclaimer slot; the phase-2 PR (diff view) will run the full gitops-demo visual pass.
Note
Medium Risk
Changes GitOps issue and audit classification logic where mis-gating or overlay tiers could add noise or miss real drift; new additive
gitops_stalecategory affects issue consumers with exhaustive switches.Overview
Extends GitOps drift visibility beyond the detail page: issues, audit checks, and a small UI disclosure.
Issues — manual-sync drift: Manual Argo apps that stay OutOfSync for 24h+ now emit a warning (
OutOfSyncManual), with drift onset tracked per Application UID on the clusterResourceCache(resets on sync; conservative after restart).Issues — frozen comparisons: Adds
gitops_staledetection when Argo’s application-controller isn’t re-comparing: a critical rollup if the controller is visible but not ready, a warning rollup when a majority of a ≥3-app fleet is past threshold, otherwise per-appComparisonStalewarnings. Threshold usesmax(30m, 10× argocd-cm timeout.reconciliation).Audit — GitOps coverage: New checks
gitops-unmanaged-workloadandgitops-helm-only-workloadclassify top-level workloads viasubject.ResolveOverlay, only when real GitOps root objects exist (gitOpsRootsin the audit runner, not CRD-only). Argo label-tracking is recognized whenapp.kubernetes.io/instancematches a live Application name; owner-referenced children are skipped.GitOps UI: Drift view shows a neutral note for
spec.ignoreDifferences(counts + kinds), including when rules use jq/managed-fields features Radar doesn’t evaluate.Also wires new reasons into issue category classification and adds table-driven tests. The diff includes large audit/plan markdown captures (
AUDIT-*.md,gcp-issues.md, etc.) unrelated to runtime behavior.Reviewed by Cursor Bugbot for commit 8dfa00f. Bugbot is set up for automated code reviews on this repo. Configure here.