rollouts-plugin-metric-holmesgpt

An Argo Rollouts metric provider plugin that gates canary/blue-green analysis on a HolmesGPT HealthCheck — an LLM-backed natural-language health question ("is the checkout service healthy after this rollout?") evaluated by the HolmesGPT operator.

This is a standalone binary. It does not modify argo-rollouts or holmesgpt — it's installed into an existing argo-rollouts controller the same way any other out-of-tree metric plugin is.

How it works

1. Run() creates a HealthCheck custom resource (holmesgpt.dev/v1alpha1) per measurement, with spec.query set from your AnalysisTemplate's query, automatically prefixed with a context block so Holmes doesn't have to guess. The prefix includes:

   • the Rollout name, namespace, and revision;
   • the analysis phase (canary step index, background/pre-promotion/post-promotion);
   • whether the check gates the rollout or is a dry-run;
   • when the analysis started (so Holmes scopes to the rollout window);
   • the check attempt number;
   • ready-to-use kubectl label selectors for the new/canary vs old/stable pods — the app's own selector plus the rollouts-pod-template-hash clause, so Holmes can pull and compare new vs old pod logs/events directly;
   • generic verdict calibration — "use the details above directly, don't re-derive them", "fail only if the canary is materially worse than stable (a warming-up pod or one-off restart is not a failure)", and "end with a clear verdict + evidence". (The "materially worse" rule is omitted for dry-run checks, which don't gate.)

   All of this is derived entirely from the AnalysisRun's own owner reference, labels, and annotations — the same ones argo-rollouts always sets when it creates an AnalysisRun for a Rollout — so no extra RBAC and no AnalysisTemplate args are required. Because the framing and facts are injected for you, your query only needs the investigation specific to your service (see Writing a good query).

2. Resume() polls the HealthCheck's .status.phase/.status.result every pollIntervalSeconds (default 5s) until it reaches Completed or Failed, or exceeds its configured timeout.

3. The result maps directly onto the measurement's AnalysisPhase (pass -> Successful, fail -> Failed, error/operator failure -> Error) — same approach as argo-rollouts' built-in Job provider, just targeting the HealthCheck CRD instead of a batchv1.Job. No successCondition/failureCondition needed in the AnalysisTemplate.

4. GarbageCollect() prunes old HealthChecks for a given AnalysisRun beyond the configured history limit.

Prerequisites

• An existing argo-rollouts controller installation.
• The HolmesGPT operator installed and running in the same cluster, with the healthchecks.holmesgpt.dev CRD registered.

Installing

1. Get the plugin binary

Download a prebuilt binary from the releases page, or build it yourself:

make build-linux-amd64   # -> dist/holmesgpt-metric-plugin-linux-amd64
# or build a container image (see Dockerfile) to use as an init container
make image

2. Mount the binary into the rollouts-controller pod

Add an init container to the argo-rollouts controller Deployment that copies the binary onto a volume shared with the main container, at plugin-bin/argoproj-labs/rollouts-plugin-metric-holmesgpt relative to the controller's working directory (argo-rollouts resolves the plugin's install path from its configmap name, not from the location field — see plugins.md):

# strategic merge patch on the argo-rollouts Deployment
spec:
  template:
    spec:
      initContainers:
        - name: holmesgpt-plugin-init
          image: <your-registry>/holmesgpt-argo-rollouts-metric-plugin:latest
          command: ["cp", "/holmesgpt-metric-plugin", "/plugin-bin/argoproj-labs/rollouts-plugin-metric-holmesgpt"]
          volumeMounts:
            - name: plugin-bin
              mountPath: /plugin-bin/argoproj-labs
      containers:
        - name: argo-rollouts
          volumeMounts:
            - name: plugin-bin
              mountPath: /plugin-bin/argoproj-labs
      volumes:
        - name: plugin-bin
          emptyDir: {}

(Or host the binary over HTTP(S) and use a location: https://... URL instead — see examples/configmap-patch.yaml and the upstream plugin docs for that path.)

3. Register the plugin

Merge examples/configmap-patch.yaml into the argo-rollouts-config ConfigMap.

4. Grant RBAC

Apply examples/rbac.yaml (adjust the namespace to wherever your AnalysisRuns/HealthChecks live).

5. Reference it from an AnalysisTemplate

See examples/analysistemplate.yaml and examples/rollout-canary-gate.yaml for a full canary step wired up to a HolmesGPT gate.

Config reference

Set under spec.metrics[].provider.plugin["argoproj-labs/rollouts-plugin-metric-holmesgpt"]:

Field	Type	Default	Description
query	string	(required)	Natural language health question, passed to HealthCheck.spec.query
namespace	string	AnalysisRun's namespace	Namespace to create the HealthCheck in
timeout	int	300	HealthCheck.spec.timeout (seconds, capped at 300 by the CRD); also drives the plugin's own stuck-check safety valve (timeout + 30s)
mode	string	monitor	HealthCheck.spec.mode (alert or monitor)
model	string	-	Override LLM model for this check (HealthCheck.spec.model)
destinations	list	-	Alert destinations, only used when mode: alert
pollIntervalSeconds	int	5	How often Resume() re-checks the HealthCheck's status

Writing a good query

The plugin already injects, in front of every query, all the generic framing a canary check needs — rollout identity, phase, start time, the exact new/old pod selectors, and verdict calibration (see How it works). So your query should contain only what's specific to your service, and should not repeat any of the injected boilerplate.

Don't restate what's already injected:

# ❌ redundant — all of this is injected already
query: |
  You are checking rollout my-app. Find the canary and stable pods.
  The selectors are provided above; use them and don't re-derive them.
  Fail only if the canary is materially worse than stable; ignore
  transient startup noise. Report a verdict.

Do state just the investigation specific to your service:

# ✅ only the service-specific part
query: |
  Compare the new/canary pods against the old/stable pods to decide whether
  the new version introduced a regression:
  1. Compare their logs for new errors, panics, or unexpected behaviour.
  2. Check Kubernetes events for restarts, OOMKills, or failed probes.
  3. If Prometheus/Datadog is connected, compare error rate and latency
     of the canary vs the stable pods.

Tips:

• Reference the injected selectors ("the new/canary and old/stable pods") instead of telling Holmes to discover pods — it wastes tool calls and time.
• Keep it focused. Every extra step is more tool calls, a longer run, and a higher chance of hitting timeout. A leaner query is faster and cheaper.
• Only add service-specific knowledge Holmes can't infer — e.g. "the source repo is github.com/acme/foo", "healthy means the /readyz endpoint returns 200", "ignore the known-noisy reconcile skipped warning".
• The full example is in examples/analysistemplate.yaml.

Development

go mod tidy   # requires network access; already run, go.sum is checked in
go build ./...
go vet ./...
go test ./...

go build, go vet, and go test ./... (all 6 test cases across Run/Resume/GarbageCollect/name-truncation) pass as of this writing against github.com/argoproj/argo-rollouts v1.9.0 (go mod tidy resolved this automatically; v1.2.0 doesn't publish these plugin packages on the module proxy). go.mod also carries a copy of argo-rollouts' own replace block for the k8s.io/* staging modules — Go replace directives aren't transitive, so any consumer of argo-rollouts needs to repeat them or module resolution fails on k8s.io/cluster-bootstrap@v0.0.0-style pseudo-versions.

Limitations

• Verified end-to-end against a live argo-rollouts controller + HolmesGPT operator: the plugin creates the HealthCheck, HolmesGPT evaluates it, and the resulting pass/fail verdict correctly gates the canary rollout.
• No generated Go client exists for the HealthCheck CRD (the operator is Python/kopf), so the plugin talks to it via client-go's dynamic client and unstructured.Unstructured rather than typed objects.
• Deleting a HealthCheck (on Terminate) can't cancel an in-flight LLM call inside the operator — it only stops the plugin from polling it further.
• Cross-namespace HealthChecks (via the namespace config field) don't get an owner reference back to the AnalysisRun, so they aren't garbage-collected by Kubernetes if the AnalysisRun is deleted directly; rely on GarbageCollect's history limit instead.