fix(#341): pin dev by digest so pods converge; symmetric convergence check - #342
Merged
Conversation
dev pinned the moving :main tag with imagePullPolicy: Always, which is non-convergent: Always re-resolves :main -> digest independently per pod at each (re)start, so pods started seconds apart or restarted later (liveness/eviction/cron) silently drift onto different builds. This is the exact cross-pod skew dev's >=2 replicas exist to surface, yet it was invisible: AGENTS.md documented convergence verification for prod only. - k8s/dev-deployment.yaml: pin main@sha256:<digest>, IfNotPresent (like prod) - AGENTS.md: dev redeploy is now a promote-by-digest flow; convergence check is stated once for both dev and prod, with the Terminating-zombie caveat (never force-delete).
…ss digest lookup The pin in this PR was read 18 days ago. :main has moved five builds since, and dev has been hand-pinned twice in the meantime to keep up — most recently to sha256:f9be5dae (git_sha 19b3c70, current main tip), which is what the live deployment and both serving pods run right now. Merging the stale sha256:1b0f133f would have made git disagree with the cluster and silently rolled dev back 18 days on the next `kubectl apply` — the same class of drift this PR exists to prevent, just slower. Re-read the digest from the registry and pin what dev actually runs, so the merge is a no-op against the live deployment. Also document the registry-v2 fallback for reading a digest. The `imagetools inspect` line assumes a docker CLI, which a JupyterLab session doesn't have, and the GHCR packages API needs a read:packages scope our gh logins generally lack — so the documented step was not executable from the environment the rollout often gets driven from. The curl form works anywhere, with a note to validate it against prod's existing pin before trusting it.
This was referenced Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #341.
Problem
dev-duckdb-mcpwas serving traffic from multiple mismatched builds at once. The live ReplicaSet ran two different digests on its two pods, so a request hit the metadata.json tile feature (≥ v0.8.6) or not, at random — which is what brokeadd_hex_tile_layeron the padus app.Root cause is the deploy design, not a code bug: dev pinned the moving
:maintag withimagePullPolicy: Always.Alwaysre-resolves the mutable:main→ digest independently per pod at each (re)start, and:mainmoves on every push tomainplus the weekly cron. So pods started seconds apart, or any single later liveness/eviction/crash/cron restart, silently drift onto different builds — the exact cross-pod skew dev's ≥2 replicas exist to surface. AGENTS.md documented convergence verification for prod only, so the skew was invisible in dev's own workflow.Fix (issue's Option A — pin dev by digest, same discipline as prod)
k8s/dev-deployment.yaml: pinmain@sha256:1b0f133f…(current:main) withimagePullPolicy: IfNotPresent. All dev pods now run one reproducible build; restarts/evictions/cron can't silently drift a pod. Redeploy is one extraimagetools inspectline — identical to the prod promote flow.AGENTS.md: dev redeploy is now a promote-by-digest flow (read digest → pin → apply → restart → verify). The convergence check is stated once for both dev and prod, with the Terminating-zombie caveat (out of Service endpoints; never force-delete per NRP policy).Trade-off: dev no longer auto-tracks
main/the cron. Accepted — dev tests a specific merge, so reproducibility beats auto-freshness, and the cron silently bumping dev to random builds was itself the footgun.Cluster remediation (already applied)
Applied the digest-pinned manifest to dev and rolled out — this both converges the live pods and implements the pin. Verified all live pods report the single pinned
imageID. The two stuckTerminatingzombie pods (07-05 / 07-09) on unreachable nodes are left alone per NRP ops policy.