ci: actually run conformance suite (export KUBECONFIG to kind) - #69
Merged
Conversation
The conformance suite's specs all silently SKIP in CI. helm/kind-action
writes the kubeconfig to $HOME/.kube/config but never exports KUBECONFIG,
and the suite's BeforeSuite skips the entire run when KUBECONFIG is unset
(run 27262377802: "Ran 0 of 39 Specs ... 39 Skipped").
Root-cause fix:
- conformance.yaml: after each helm/kind-action step, export
KUBECONFIG=$HOME/.kube/config to $GITHUB_ENV so the suite (and make
targets) see the cluster.
- conformance_suite_test.go: BeforeSuite now resolves the kubeconfig via
clientcmdPath() (the same $KUBECONFIG-or-~/.kube/config fallback the rest
of the suite already uses) and only skips when no kubeconfig file exists.
Defense-in-depth so a reachable cluster is never silently ignored again.
Surfaced once the suite actually ran (all pre-existing, masked by the
silent skip):
- Makefile: conformance-negative focused on "negative", which matches no
spec text ("webhook deny paths" / "deny: ..."), so that job ran 0 specs.
Fixed the focus to "webhook deny paths".
- negative_test.go: the deny-path assertion checked err.Error() (only the
process exit status, "exit status 1") instead of the kubectl output that
carries the webhook denial message. Assert against the output. All 19
webhook-deny specs now pass.
- testdata/*.yaml: fixtures pinned image tag v1.0.0 / 1.0.0, which was
never published to ghcr.io/paperclipinc/hermes-agent (only v2026.5.29.2
and latest exist), so pods ImagePullBackOff. Pinned to v2026.5.29.2.
Left skipped (visibly, with reasons), out of scope for this CI fix:
- idempotency corpus "becomes Ready" specs are blocked by #68: the
operator's init-uv container copies pyproject.toml/uv.lock from
/opt/venv-template/, which the published hermes-agent image does not
ship (its venv is at /opt/venv), so no HermesInstance reaches Ready.
- ollama-webterminal-tailscale additionally needs a real ephemeral
tailnet auth key (the fixture ships a dummy key).
Verified locally on a kind v1.31 cluster with cert-manager + the operator
installed: BeforeSuite PASSED and "Ran 19 of 39 Specs ... 19 Passed |
0 Failed | 20 Skipped" (vs the original "Ran 0 of 39 ... 39 Skipped").
Closes #64
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first run of the now-executing suite showed the Negative job hit its 20m timeout while the (pre-existing) "Install cert-manager" step was still running on a slow runner, so the spec step never ran. Now that the suite actually executes, each kind job also builds the operator image and runs specs, so the old timeouts are too tight. - Raise kind-job timeouts: negative 20->35, gitops/failure 30->40 (idempotency already 45, upgrade already 90). - Drop the redundant `make docker-build` step: `conformance-install` already declares `docker-build` as a prerequisite, so the image was being built twice per job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gitops-coexistence and failure-injection suites aren't implemented (gitops_coexistence_test.go is a t.Skip placeholder; there is no failure_injection_test.go), so `make conformance-gitops`/`-failure` match zero Ginkgo specs. Yet each job was spinning up a kind cluster, installing cert-manager, and building the operator image to run nothing. Worse, those extra parallel kind clusters starve the shared runner: in practice cert-manager's `helm install --wait` blows far past its 5m timeout (observed ~30+ min wall-clock) in the jobs that DO run specs, timing the Negative job out. The Idempotency job (45m budget) still passed and proves the suite now executes against the cluster (see #64). Run the two empty suites without a cluster (the focus matches nothing and BeforeSuite skips cleanly), leaving only negative + idempotency to contend for the runner. Restore the kind/cert-manager/install steps (mirroring the negative job) when those specs are actually written. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cert-manager normally installs in ~15s (verified in a passing Idempotency run: the step took 17s). But on GitHub's shared runners the install intermittently wedges for 20-30+ min, and a single `helm install --wait --timeout 5m` does not reliably bail out of that hang within the conformance job budget, so the kind-based jobs time out (the Idempotency job, with the largest budget, is the one that has been passing). Replace the inline helm install in the negative and idempotency jobs with hack/install-cert-manager.sh: each attempt is hard-capped with `timeout` and `--wait --timeout`, and a stuck attempt is killed and retried (the retry almost always gets the fast install). This turns a 30-min hang into a few minutes worst case. Pin cert-manager to v1.20.2 and use upgrade --install so retries are idempotent. Negative back to a 30m budget now that cert-manager is bounded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The conformance suite's specs all silently SKIP in CI:
helm/kind-actionwrites the kubeconfig to$HOME/.kube/configbut never exportsKUBECONFIG, and the suite'sBeforeSuiteskips the whole run whenKUBECONFIGis unset (run 27262377802 reportedRan 0 of 39 Specs ... 39 Skippedwhile still going green). This exportsKUBECONFIGin the workflow and hardensBeforeSuite, then fixes the real failures that were masked by the silent skip so the suite actually executes and passes.Root-cause fix
.github/workflows/conformance.yaml: after eachhelm/kind-actionstep, exportKUBECONFIG=$HOME/.kube/configto$GITHUB_ENV(4 kind-using jobs).test/conformance/conformance_suite_test.go:BeforeSuitenow resolves the kubeconfig viaclientcmdPath()— the same$KUBECONFIG-or-~/.kube/configfallback the rest of the suite already uses — and only skips when no kubeconfig file exists. Defense-in-depth so a reachable cluster is never silently ignored again.Pre-existing bugs surfaced once the suite ran (all masked by the silent skip)
conformance-negativefocused on"negative", which matches no spec text (webhook deny paths/deny: ...), so the job ran 0 specs. Fixed the focus to"webhook deny paths".negative_test.go: the deny-path assertion checkederr.Error()(only the process exit status,"exit status 1") instead of the kubectl output that carries the webhook denial message. Now asserts against the output. All 19 webhook-deny specs pass.testdata/*.yaml: fixtures pinned image tagv1.0.0/1.0.0, which was never published toghcr.io/paperclipinc/hermes-agent(onlyv2026.5.29.2andlatestexist) →ImagePullBackOff. Pinned to the real published tagv2026.5.29.2.Left skipped (visibly, with reasons) — out of scope for this CI fix
becomes Readyspecs are blocked by operator: init-uv copies from /opt/venv-template (absent in published hermes-agent image) so no HermesInstance reaches Ready #68: the operator'sinit-uvcontainer copiespyproject.toml/uv.lockfrom/opt/venv-template/, which the published hermes-agent image does not ship (its venv is at/opt/venv), so noHermesInstancereaches Ready. Filed as operator: init-uv copies from /opt/venv-template (absent in published hermes-agent image) so no HermesInstance reaches Ready #68; specsSkipwith a reason pointing there rather than hanging until timeout.ollama-webterminal-tailscaleadditionally needs a real ephemeral tailnet auth key (the fixture ships a dummy key).Local verification
kind v1.31 cluster + cert-manager + operator installed, suite run with
KUBECONFIGset:(vs the original
Ran 0 of 39 ... 39 Skipped.) The 19 webhook-deny specs execute and pass; the 20 idempotency specs skip with documented reasons.Test plan
make lint)make test)bash hack/reconcile-guard.sh)bash hack/check-helm-rbac.sh)make manifests+make generateregenerated — n/a (no API changes)make sync-bundle-rbacif the bundle is affected — n/aRelated issues
Closes #64
Refs #68 (init-uv
/opt/venv-templateimage-contract bug that blocks idempotency Ready-gating)