test(conformance): refresh Ready-gated skip reasons (#68 fixed) + dump diagnostics on timeout - #87
Merged
Merged
Conversation
#68 is fixed (#85 ships pyproject.toml + uv.lock at /opt/venv-template/ in the agent image, and the v2026.5.29.2 image has been republished with the fix — verified: /opt/venv-template/{pyproject.toml,uv.lock} now present). Every idempotency corpus entry except the tailscale one was gated solely on #68, so un-skip them; the conformance suite pins tag v2026.5.29.2 which now reaches Ready. The ollama-webterminal-tailscale entry stays skipped — it needs a live ephemeral Tailscale auth key (unrelated to #68). Its reason is updated to drop the now-stale #68 reference. The unused idempotencyImageContractSkip const is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
waitForInstanceReady previously failed with a bare timeout message, giving no signal on *why* an instance didn't reach Ready. Add a best-effort kubectl dump (instance yaml, pods/sts, describe, events, init-container logs) before failing, so conformance CI failures are diagnosable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stubbi
added a commit
that referenced
this pull request
Jun 18, 2026
## Second half of the #68 fix #85 added `pyproject.toml`/`uv.lock` at `/opt/venv-template/`, which let `init-uv` get past its `cp` step — and immediately exposed a **second** latent defect in the same init container. The conformance suite (un-skipped in #87, now that #85 republished the image) caught it: ``` pod/conformance-minimal-0 0/1 Init:Error 4 restarts init-uv Terminated Exit Code: 127 logs init-uv: /bin/sh: 1: uv: not found ``` `init-uv` runs `uv sync --frozen` at pod start to materialise the env onto the per-instance PVC, but the runtime stage copies the resolved venv (`/opt/venv`) and **never the `uv` binary**. So `uv sync` → exit 127. Before #85, `init-uv` died at the earlier `cp`, masking this. ## Fix - `COPY --from=uv /uv /uvx /usr/local/bin/` in the runtime stage (mirrors the builder). - Deterministic UV_* defaults so the pod-start sync never downloads a Python or hardlinks across the image-layer → PVC boundary: `UV_PYTHON_DOWNLOADS=never`, `UV_PYTHON=/usr/local/bin/python`, `UV_LINK_MODE=copy`. ## Verification (local) Built the image and ran the **exact** init-uv command: ``` $ docker run --rm --entrypoint /bin/sh hermes-agent:uvtest -c \ 'set -eu; cd /home/hermes/.hermes; cp /opt/venv-template/{pyproject.toml,uv.lock} .; uv --version; uv sync --frozen' uv version: uv 0.11.7 ... === EXIT 0 ; .venv populated === ``` Exit 127 → exit 0; `.venv` is created on the (PVC-equivalent) workdir. ## After merge Republish `ghcr.io/paperclipinc/hermes-agent:v2026.5.29.2`, then #87's conformance Idempotency job should go green and the un-skip lands. Refs #68 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Un-skipping (earlier in this branch) proved the #68 init-uv contract is fixed (#85 lockfiles, #88 uv binary, image republished): init-uv now succeeds and the pod reaches its main hermes container. But the diagnostics dump showed the instance still can't reach Ready in CI, for reasons beyond #68: 1. hermes-agent exits immediately without an LLM provider+credentials ("No LLM provider configured"); CI fixtures have none -> CrashLoopBackOff. 2. The long-lived agent form is `hermes gateway run` (a chat-platform daemon binding no :8443 server), not the one-shot `hermes-agent run` the entrypoint invokes, so the operator's TCPSocket :8443 readiness probe can't pass as-is. 3. The published image is missing modules (websockets, hermes_cli.dashboard_auth). Reaching Ready in CI needs design decisions (inject LLM creds, rework entrypoint/readiness/port, fix image deps), not just the #68 image fix. So keep the corpus skipped, but refresh the reason (a new idempotencyReadyBlockedSkip const) to reflect that #68 is resolved and point at the real blockers (#64). The waitForInstanceReady diagnostics dump is retained so the next attempt is debuggable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stubbi
changed the title
stubbi
enabled auto-merge (squash)
June 18, 2026 17:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #85 and #88. Net effect: the Ready-gated idempotency corpus stays skipped, but for accurate reasons, plus a diagnostics dump. (This branch briefly un-skipped the corpus to find out what was actually blocking Ready — that investigation is summarized below.)
What the investigation found
#68 (the
init-uvcontract) is fixed: #85 ships the lockfiles at/opt/venv-template/, #88 ships theuvbinary, andv2026.5.29.2was republished + verified.init-uvnow succeeds and the pod reaches its mainhermescontainer.But un-skipping showed the instance still can't become Ready in CI — three deeper, separate blockers (surfaced by the new diagnostics dump):
Failed to initialize agent: No LLM provider configured). CI fixtures carry none → container exits 0 → CrashLoopBackOff.hermes gateway run(a chat-platform daemon that binds no:8443server), not the one-shothermes-agent runthe image entrypoint invokes. The operator'sTCPSocket :8443readiness probe can't pass as-is.websockets,hermes_cli.dashboard_auth.Reaching Ready in CI needs design-level decisions (inject LLM creds, rework the entrypoint/readiness/port model, fix image deps) — out of scope for the #68 image fix.
This PR
#68skip reason withidempotencyReadyBlockedSkipdescribing the real blockers (and noting operator: init-uv copies from /opt/venv-template (absent in published hermes-agent image) so no HermesInstance reaches Ready #68 is resolved). Tracked under ci: conformance suite never executes (kind-action does not export KUBECONFIG, all 39 specs skip) #64.dumpInstanceDiagnosticstowaitForInstanceReady(instance YAML, pods/sts, describe, events, init-container logs) so the next Ready attempt is debuggable from the CI log instead of a bare timeout.Conformance stays green (skips). Un-skip once an instance can actually reach Ready in CI.
🤖 Generated with Claude Code