fix(agent-image): ship pyproject.toml + uv.lock at /opt/venv-template/ (#68) by stubbi · Pull Request #85 · paperclipinc/hermes-operator

stubbi · 2026-06-18T10:03:26Z

Problem

Every HermesInstance fails to reach Ready because the operator's init-uv init container is incompatible with the published ghcr.io/paperclipinc/hermes-agent image (issue #68, confirmed by multiple users).

internal/resources/runtime_init.go renders:

set -eu; cd /home/hermes/.hermes; \
  cp /opt/venv-template/pyproject.toml /opt/venv-template/uv.lock .; \
  uv sync --frozen

…but images/hermes-agent/Dockerfile builds the venv at /opt/venv and copies nothing to /opt/venv-template/. So init-uv exits 1:

cp: cannot stat '/opt/venv-template/pyproject.toml': No such file or directory

…the hermes container never starts, the StatefulSet never reaches readyReplicas == replicas, and the instance never becomes Ready=True.

Fix

Add one COPY to the runtime stage shipping the lockfile + project metadata at /opt/venv-template/, exactly where the operator's contract expects them. The files already exist at /build/ in the builder stage.

Scope

Minimal, surgical fix — Dockerfile only. No change to the build/sign/SBOM/multi-arch pipeline.

Follow-up

Once a fixed agent image is republished, un-skip the #68-gated specs in test/conformance/idempotency_test.go (idempotencyImageContractSkip).

Fixes #68

🤖 Generated with Claude Code

The operator's `init-uv` init container copies pyproject.toml and uv.lock from /opt/venv-template/ before running `uv sync --frozen`, but the agent Dockerfile only copied the resolved venv to /opt/venv and shipped nothing at /opt/venv-template/. Every HermesInstance therefore failed at init with: cp: cannot stat '/opt/venv-template/pyproject.toml': No such file or directory so the StatefulSet never reached readyReplicas == replicas and no instance became Ready. Add a single COPY of the lockfile + project metadata from the builder stage to /opt/venv-template/ to satisfy the operator's contract. Once a fixed agent image is republished, the #68-gated conformance specs in test/conformance/idempotency_test.go can be un-skipped. Fixes #68 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

## Second half of the #68 fix #85 added `pyproject.toml`/`uv.lock` at `/opt/venv-template/`, which let `init-uv` get past its `cp` step — and immediately exposed a **second** latent defect in the same init container. The conformance suite (un-skipped in #87, now that #85 republished the image) caught it: ``` pod/conformance-minimal-0 0/1 Init:Error 4 restarts init-uv Terminated Exit Code: 127 logs init-uv: /bin/sh: 1: uv: not found ``` `init-uv` runs `uv sync --frozen` at pod start to materialise the env onto the per-instance PVC, but the runtime stage copies the resolved venv (`/opt/venv`) and **never the `uv` binary**. So `uv sync` → exit 127. Before #85, `init-uv` died at the earlier `cp`, masking this. ## Fix - `COPY --from=uv /uv /uvx /usr/local/bin/` in the runtime stage (mirrors the builder). - Deterministic UV_* defaults so the pod-start sync never downloads a Python or hardlinks across the image-layer → PVC boundary: `UV_PYTHON_DOWNLOADS=never`, `UV_PYTHON=/usr/local/bin/python`, `UV_LINK_MODE=copy`. ## Verification (local) Built the image and ran the **exact** init-uv command: ``` $ docker run --rm --entrypoint /bin/sh hermes-agent:uvtest -c \ 'set -eu; cd /home/hermes/.hermes; cp /opt/venv-template/{pyproject.toml,uv.lock} .; uv --version; uv sync --frozen' uv version: uv 0.11.7 ... === EXIT 0 ; .venv populated === ``` Exit 127 → exit 0; `.venv` is created on the (PVC-equivalent) workdir. ## After merge Republish `ghcr.io/paperclipinc/hermes-agent:v2026.5.29.2`, then #87's conformance Idempotency job should go green and the un-skip lands. Refs #68 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Un-skipping (earlier in this branch) proved the #68 init-uv contract is fixed (#85 lockfiles, #88 uv binary, image republished): init-uv now succeeds and the pod reaches its main hermes container. But the diagnostics dump showed the instance still can't reach Ready in CI, for reasons beyond #68: 1. hermes-agent exits immediately without an LLM provider+credentials ("No LLM provider configured"); CI fixtures have none -> CrashLoopBackOff. 2. The long-lived agent form is `hermes gateway run` (a chat-platform daemon binding no :8443 server), not the one-shot `hermes-agent run` the entrypoint invokes, so the operator's TCPSocket :8443 readiness probe can't pass as-is. 3. The published image is missing modules (websockets, hermes_cli.dashboard_auth). Reaching Ready in CI needs design decisions (inject LLM creds, rework entrypoint/readiness/port, fix image deps), not just the #68 image fix. So keep the corpus skipped, but refresh the reason (a new idempotencyReadyBlockedSkip const) to reflect that #68 is resolved and point at the real blockers (#64). The waitForInstanceReady diagnostics dump is retained so the next attempt is debuggable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…p diagnostics on timeout (#87) Follow-up to #85 and #88. **Net effect: the Ready-gated idempotency corpus stays skipped, but for accurate reasons, plus a diagnostics dump.** (This branch briefly un-skipped the corpus to find out what was actually blocking Ready — that investigation is summarized below.) ## What the investigation found #68 (the `init-uv` contract) is **fixed**: #85 ships the lockfiles at `/opt/venv-template/`, #88 ships the `uv` binary, and `v2026.5.29.2` was republished + verified. `init-uv` now succeeds and the pod reaches its main `hermes` container. But un-skipping showed the instance still can't become Ready in CI — three deeper, separate blockers (surfaced by the new diagnostics dump): 1. **No LLM provider** — the published hermes-agent exits immediately without a provider + credentials (`Failed to initialize agent: No LLM provider configured`). CI fixtures carry none → container exits 0 → CrashLoopBackOff. 2. **Wrong serve command / port model** — the long-lived agent is `hermes gateway run` (a chat-platform daemon that binds **no** `:8443` server), not the one-shot `hermes-agent run` the image entrypoint invokes. The operator's `TCPSocket :8443` readiness probe can't pass as-is. 3. **Missing image deps** — `websockets`, `hermes_cli.dashboard_auth`. Reaching Ready in CI needs design-level decisions (inject LLM creds, rework the entrypoint/readiness/port model, fix image deps) — out of scope for the #68 image fix. ## This PR - Keeps the corpus skipped, but replaces the stale `#68` skip reason with `idempotencyReadyBlockedSkip` describing the real blockers (and noting #68 is resolved). Tracked under #64. - Adds `dumpInstanceDiagnostics` to `waitForInstanceReady` (instance YAML, pods/sts, describe, events, init-container logs) so the next Ready attempt is debuggable from the CI log instead of a bare timeout. Conformance stays green (skips). Un-skip once an instance can actually reach Ready in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

🤖 I have created a release *beep* *boop* --- ## [0.1.18](v0.1.17...v0.1.18) (2026-06-19) ### Features * **runtime:** run agents on the upstream s6 image — instance reaches Ready (gateway API + /health) ([#90](#90)) ([12dccb4](12dccb4)) ### Bug Fixes * **agent-image:** ship pyproject.toml + uv.lock at /opt/venv-template/ ([#68](#68)) ([#85](#85)) ([39ef52c](39ef52c)) * **agent-image:** ship the uv binary in the runtime stage ([#68](#68)) ([#88](#88)) ([13fb1e5](13fb1e5)) * **backup:** bound the on-delete finalizer so a failing snapshot can't make an instance undeletable ([#94](#94)) ([40d6099](40d6099)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: paperclip-release-bot[bot] <288053502+paperclip-release-bot[bot]@users.noreply.github.com>

This was referenced Jun 18, 2026

fix: copy pyproject.toml + uv.lock to /opt/venv-template/ in agent image #84

Closed

operator: init-uv copies from /opt/venv-template (absent in published hermes-agent image) so no HermesInstance reaches Ready #68

Closed

stubbi enabled auto-merge (squash) June 18, 2026 10:06

Merge branch 'main' into fix/agent-venv-template-lockfiles

8b42b58

stubbi merged commit 39ef52c into main Jun 18, 2026
14 checks passed

stubbi deleted the fix/agent-venv-template-lockfiles branch June 18, 2026 10:51

paperclip-release-bot Bot mentioned this pull request Jun 18, 2026

chore(main): release 0.1.18 #86

Merged

This was referenced Jun 18, 2026

test(conformance): refresh Ready-gated skip reasons (#68 fixed) + dump diagnostics on timeout #87

Merged

fix(agent-image): ship the uv binary in the runtime stage (#68) #88

Merged

stubbi mentioned this pull request Jun 18, 2026

Operator runtime model (:8443 gateway server) doesn't match the real hermes-agent; no HermesInstance reaches Ready #89

Closed

Navigation Menu

Uh oh!

fix(agent-image): ship pyproject.toml + uv.lock at /opt/venv-template/ (#68) - #85

fix(agent-image): ship pyproject.toml + uv.lock at /opt/venv-template/ (#68)#85
stubbi merged 2 commits into
mainfrom
fix/agent-venv-template-lockfiles

stubbi commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

stubbi commented Jun 18, 2026

Problem

Fix

Scope

Follow-up

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant