fix(#341): pin dev by digest so pods converge; symmetric convergence check (#342)

* fix(#341): pin dev by digest (convergent) + symmetric convergence check

dev pinned the moving :main tag with imagePullPolicy: Always, which is
non-convergent: Always re-resolves :main -> digest independently per pod
at each (re)start, so pods started seconds apart or restarted later
(liveness/eviction/cron) silently drift onto different builds. This is
the exact cross-pod skew dev's >=2 replicas exist to surface, yet it was
invisible: AGENTS.md documented convergence verification for prod only.

- k8s/dev-deployment.yaml: pin main@sha256:<digest>, IfNotPresent (like prod)
- AGENTS.md: dev redeploy is now a promote-by-digest flow; convergence
  check is stated once for both dev and prod, with the Terminating-zombie
  caveat (never force-delete).

* fix(#341): refresh the dev pin to current :main; document a docker-less digest lookup

The pin in this PR was read 18 days ago. :main has moved five builds since, and
dev has been hand-pinned twice in the meantime to keep up — most recently to
sha256:f9be5dae (git_sha 19b3c70, current main tip), which is what the live
deployment and both serving pods run right now.

Merging the stale sha256:1b0f133f would have made git disagree with the cluster
and silently rolled dev back 18 days on the next `kubectl apply` — the same
class of drift this PR exists to prevent, just slower. Re-read the digest from
the registry and pin what dev actually runs, so the merge is a no-op against the
live deployment.

Also document the registry-v2 fallback for reading a digest. The `imagetools
inspect` line assumes a docker CLI, which a JupyterLab session doesn't have, and
the GHCR packages API needs a read:packages scope our gh logins generally lack —
so the documented step was not executable from the environment the rollout often
gets driven from. The curl form works anywhere, with a note to validate it
against prod's existing pin before trusting it.

Author: cboettig

AGENTS.md

@@ -82,14 +82,24 @@ release tags. The image is the unit of release.
 - `:<git-sha>` — immutable; one per commit.
 - `:vX.Y.Z` — immutable; built on release tags. **prod** pins this (by digest, below).
 
-**Merge to `main` → redeploy dev:**
-```
-kubectl apply -f k8s/dev-deployment.yaml
-kubectl rollout restart deployment/dev-duckdb-mcp -n biodiversity
-```
-dev pins `:main` with `imagePullPolicy: Always`, so the restart pulls the freshest build.
-**Wait for the `docker.yml` run on your merge to go green first** — rolling before the
-image is pushed gives `ImagePullBackOff`.
+**Merge to `main` → redeploy dev (promote by digest — same discipline as prod):**
+Dev pins an immutable `:main@sha256:…` (tag for humans, digest enforced), **not** the
+bare moving `:main` tag. `:main` + `imagePullPolicy: Always` is *non-convergent*: `Always`
+re-resolves the mutable `:main` → digest independently per pod at each (re)start, and
+`:main` moves on every push to `main` plus the weekly cron. So pods brought up seconds
+apart, or any single later liveness/eviction/crash/cron restart, silently drift onto
+different builds — the exact cross-pod skew dev's ≥2 replicas exist to catch (issue #341).
+Pinning a digest makes every dev pod run one reproducible build.
+
+1. **Wait for the `docker.yml` run on your merge to go green first** — rolling before the
+   image is pushed gives `ImagePullBackOff`.
+2. Read the freshly-built `:main` digest from the build run's job summary, or:
+   `docker buildx imagetools inspect ghcr.io/boettiger-lab/mcp-data-server:main --format '{{.Manifest.Digest}}'`
+3. Set `image: ghcr.io/boettiger-lab/mcp-data-server:main@sha256:<digest>` in
+   `k8s/dev-deployment.yaml` (keep `imagePullPolicy: IfNotPresent`).
+4. `kubectl apply -f k8s/dev-deployment.yaml`
+5. `kubectl rollout restart deployment/dev-duckdb-mcp -n biodiversity`
+6. Verify convergence (below) — this is dev's canary job; do not skip it.
 
 **Tag a release → redeploy prod (promote by digest):**
 1. `git tag vX.Y.Z && git push origin vX.Y.Z`, then wait for `docker.yml` to build `:vX.Y.Z`.
@@ -105,11 +115,33 @@ disagree, the digest wins). **Never apply prod while the manifest points at an i
 hasn't built yet** — the rollout stalls on `ImagePullBackOff`. `kubectl apply` must precede
 `rollout restart`; a git push alone does not update the cluster.
 
-Verify all prod replicas converge on a single digest after rollout:
+**No `docker` CLI (e.g. a JupyterLab session)?** `imagetools inspect` needs it, and the
+GHCR packages API needs a `read:packages` token most of our `gh` logins don't carry. An
+anonymous pull token against the registry v2 API works anywhere `curl` does — substitute
+`main` or `vX.Y.Z` for `<tag>`:
+```
+repo=boettiger-lab/mcp-data-server
+tok=$(curl -s "https://ghcr.io/token?scope=repository:$repo:pull" \
+      | python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
+curl -sI -H "Authorization: Bearer $tok" \
+  -H "Accept: application/vnd.oci.image.index.v1+json" \
+  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
+  "https://ghcr.io/v2/$repo/manifests/<tag>" | grep -i '^docker-content-digest:'
+```
+Sanity-check the method before trusting it: run it on the tag prod already pins and
+confirm the digest matches `k8s/deployment.yaml`.
+
+**Verify all replicas converge on a single digest after rollout** (both dev and prod —
+dev is the multi-replica canary, so its convergence matters as much as prod's). Swap the
+label for the deployment you rolled (`duckdb-mcp` for prod, `dev-duckdb-mcp` for dev):
 ```
 kubectl -n biodiversity get pods -l app=duckdb-mcp \
   -o custom-columns='NAME:.metadata.name,IMAGE:.status.containerStatuses[0].imageID'
 ```
+Every live (non-`Terminating`) pod must report the **same** `imageID` digest, and it must
+match the digest pinned in the manifest. Stuck `Terminating` zombie pods on unreachable
+NRP nodes are out of the Service's endpoints, so ignore them here — **never** force-delete
+them (NRP ops policy).
 
 ---
 

k8s/dev-deployment.yaml

@@ -23,8 +23,17 @@ spec:
                     values: ["us-west"]
       containers:
       - name: server
-        image: ghcr.io/boettiger-lab/mcp-data-server:main
-        imagePullPolicy: Always
+        # Pinned by digest (not the moving :main tag) so every dev pod converges
+        # on one reproducible build. :main + imagePullPolicy: Always is
+        # non-convergent: Always re-resolves the mutable :main → digest
+        # independently per pod at each (re)start, so pods brought up seconds
+        # apart — or a single later liveness/eviction/cron restart — silently
+        # drift onto different builds with no signal, exactly the cross-pod skew
+        # dev's ≥2 replicas exist to catch. Bump this digest on redeploy (see
+        # AGENTS.md → Rollout workflow). The tag prefix is for humans; the digest
+        # is enforced. See issue #341.
+        image: ghcr.io/boettiger-lab/mcp-data-server:main@sha256:f9be5daeb9af3d96dc77a10d78c557e911c687d5091b1e513eb8b6068b048144
+        imagePullPolicy: IfNotPresent
         env:
         - name: STAC_CATALOG_URL
           value: "https://s3-west.nrp-nautilus.io/public-data/stac/catalog.json"