Skip to content

ci: actually run conformance suite (export KUBECONFIG to kind) - #69

Merged
stubbi merged 5 commits into
mainfrom
ci/fix-conformance-kubeconfig
Jun 13, 2026
Merged

ci: actually run conformance suite (export KUBECONFIG to kind)#69
stubbi merged 5 commits into
mainfrom
ci/fix-conformance-kubeconfig

Conversation

@stubbi

@stubbi stubbi commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

The conformance suite's specs all silently SKIP in CI: helm/kind-action writes the kubeconfig to $HOME/.kube/config but never exports KUBECONFIG, and the suite's BeforeSuite skips the whole run when KUBECONFIG is unset (run 27262377802 reported Ran 0 of 39 Specs ... 39 Skipped while still going green). This exports KUBECONFIG in the workflow and hardens BeforeSuite, then fixes the real failures that were masked by the silent skip so the suite actually executes and passes.

Root-cause fix

  • .github/workflows/conformance.yaml: after each helm/kind-action step, export KUBECONFIG=$HOME/.kube/config to $GITHUB_ENV (4 kind-using jobs).
  • test/conformance/conformance_suite_test.go: BeforeSuite now resolves the kubeconfig via clientcmdPath() — the same $KUBECONFIG-or-~/.kube/config fallback the rest of the suite already uses — and only skips when no kubeconfig file exists. Defense-in-depth so a reachable cluster is never silently ignored again.

Pre-existing bugs surfaced once the suite ran (all masked by the silent skip)

  • Makefile: conformance-negative focused on "negative", which matches no spec text (webhook deny paths / deny: ...), so the job ran 0 specs. Fixed the focus to "webhook deny paths".
  • negative_test.go: the deny-path assertion checked err.Error() (only the process exit status, "exit status 1") instead of the kubectl output that carries the webhook denial message. Now asserts against the output. All 19 webhook-deny specs pass.
  • testdata/*.yaml: fixtures pinned image tag v1.0.0 / 1.0.0, which was never published to ghcr.io/paperclipinc/hermes-agent (only v2026.5.29.2 and latest exist) → ImagePullBackOff. Pinned to the real published tag v2026.5.29.2.

Left skipped (visibly, with reasons) — out of scope for this CI fix

Local verification

kind v1.31 cluster + cert-manager + operator installed, suite run with KUBECONFIG set:

[BeforeSuite] PASSED
Ran 19 of 39 Specs in 3.745 seconds
SUCCESS! -- 19 Passed | 0 Failed | 0 Pending | 20 Skipped

(vs the original Ran 0 of 39 ... 39 Skipped.) The 19 webhook-deny specs execute and pass; the 20 idempotency specs skip with documented reasons.

Test plan

  • Lint passes (make lint)
  • Tests pass (make test)
  • Reconcile-guard passes (bash hack/reconcile-guard.sh)
  • Helm RBAC sync passes (bash hack/check-helm-rbac.sh)
  • For CRD/API changes: make manifests + make generate regenerated — n/a (no API changes)
  • For RBAC changes: make sync-bundle-rbac if the bundle is affected — n/a
  • For behavior changes: conformance suite now executes against the kind cluster (verified locally; 19 webhook-deny specs pass, idempotency specs skip with reasons)

Related issues

Closes #64
Refs #68 (init-uv /opt/venv-template image-contract bug that blocks idempotency Ready-gating)

stubbi and others added 5 commits June 13, 2026 19:52
The conformance suite's specs all silently SKIP in CI. helm/kind-action
writes the kubeconfig to $HOME/.kube/config but never exports KUBECONFIG,
and the suite's BeforeSuite skips the entire run when KUBECONFIG is unset
(run 27262377802: "Ran 0 of 39 Specs ... 39 Skipped").

Root-cause fix:
- conformance.yaml: after each helm/kind-action step, export
  KUBECONFIG=$HOME/.kube/config to $GITHUB_ENV so the suite (and make
  targets) see the cluster.
- conformance_suite_test.go: BeforeSuite now resolves the kubeconfig via
  clientcmdPath() (the same $KUBECONFIG-or-~/.kube/config fallback the rest
  of the suite already uses) and only skips when no kubeconfig file exists.
  Defense-in-depth so a reachable cluster is never silently ignored again.

Surfaced once the suite actually ran (all pre-existing, masked by the
silent skip):
- Makefile: conformance-negative focused on "negative", which matches no
  spec text ("webhook deny paths" / "deny: ..."), so that job ran 0 specs.
  Fixed the focus to "webhook deny paths".
- negative_test.go: the deny-path assertion checked err.Error() (only the
  process exit status, "exit status 1") instead of the kubectl output that
  carries the webhook denial message. Assert against the output. All 19
  webhook-deny specs now pass.
- testdata/*.yaml: fixtures pinned image tag v1.0.0 / 1.0.0, which was
  never published to ghcr.io/paperclipinc/hermes-agent (only v2026.5.29.2
  and latest exist), so pods ImagePullBackOff. Pinned to v2026.5.29.2.

Left skipped (visibly, with reasons), out of scope for this CI fix:
- idempotency corpus "becomes Ready" specs are blocked by #68: the
  operator's init-uv container copies pyproject.toml/uv.lock from
  /opt/venv-template/, which the published hermes-agent image does not
  ship (its venv is at /opt/venv), so no HermesInstance reaches Ready.
- ollama-webterminal-tailscale additionally needs a real ephemeral
  tailnet auth key (the fixture ships a dummy key).

Verified locally on a kind v1.31 cluster with cert-manager + the operator
installed: BeforeSuite PASSED and "Ran 19 of 39 Specs ... 19 Passed |
0 Failed | 20 Skipped" (vs the original "Ran 0 of 39 ... 39 Skipped").

Closes #64

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first run of the now-executing suite showed the Negative job hit its
20m timeout while the (pre-existing) "Install cert-manager" step was still
running on a slow runner, so the spec step never ran. Now that the suite
actually executes, each kind job also builds the operator image and runs
specs, so the old timeouts are too tight.

- Raise kind-job timeouts: negative 20->35, gitops/failure 30->40
  (idempotency already 45, upgrade already 90).
- Drop the redundant `make docker-build` step: `conformance-install`
  already declares `docker-build` as a prerequisite, so the image was
  being built twice per job.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gitops-coexistence and failure-injection suites aren't implemented
(gitops_coexistence_test.go is a t.Skip placeholder; there is no
failure_injection_test.go), so `make conformance-gitops`/`-failure` match
zero Ginkgo specs. Yet each job was spinning up a kind cluster, installing
cert-manager, and building the operator image to run nothing.

Worse, those extra parallel kind clusters starve the shared runner: in
practice cert-manager's `helm install --wait` blows far past its 5m timeout
(observed ~30+ min wall-clock) in the jobs that DO run specs, timing the
Negative job out. The Idempotency job (45m budget) still passed and proves
the suite now executes against the cluster (see #64).

Run the two empty suites without a cluster (the focus matches nothing and
BeforeSuite skips cleanly), leaving only negative + idempotency to contend
for the runner. Restore the kind/cert-manager/install steps (mirroring the
negative job) when those specs are actually written.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cert-manager normally installs in ~15s (verified in a passing Idempotency
run: the step took 17s). But on GitHub's shared runners the install
intermittently wedges for 20-30+ min, and a single
`helm install --wait --timeout 5m` does not reliably bail out of that hang
within the conformance job budget, so the kind-based jobs time out (the
Idempotency job, with the largest budget, is the one that has been passing).

Replace the inline helm install in the negative and idempotency jobs with
hack/install-cert-manager.sh: each attempt is hard-capped with `timeout` and
`--wait --timeout`, and a stuck attempt is killed and retried (the retry
almost always gets the fast install). This turns a 30-min hang into a few
minutes worst case. Pin cert-manager to v1.20.2 and use upgrade --install so
retries are idempotent. Negative back to a 30m budget now that cert-manager
is bounded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@stubbi
stubbi merged commit 6989e6d into main Jun 13, 2026
20 checks passed
@stubbi
stubbi deleted the ci/fix-conformance-kubeconfig branch June 13, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: conformance suite never executes (kind-action does not export KUBECONFIG, all 39 specs skip)

1 participant