Retry staged-image verify and tag promotion on transient registry errors

Problem 4 (intermittent Artifactory `Client.Timeout`) previously only
wrapped `docker login`. The deploy job's staged-image existence check
and the `docker buildx imagetools create` promotion still made
un-retried registry calls, so a transient timeout there failed the
deploy.

Add a shared `with_retries` helper (scripts/lib/retry.sh, 3 attempts /
15s backoff) and use it for:
- the verify-images check (accepts only HTTP 200; retries transient
  000/5xx, still fails fast-enough on a genuinely-missing 404 after
  attempts are exhausted),
- the CI promote step, and
- the local `oc-build-push.sh --promote` path (CI/local parity).

Docs updated to describe the broadened retry coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: alex-struk

.github/workflows/deploy-instance.yml

@@ -227,20 +227,35 @@ jobs:
           SHA_TAG: ${{ needs.metadata.outputs.sha-tag }}
         run: |
           set -euo pipefail
+          source scripts/lib/retry.sh
           AUTH="${ARTIFACTORY_SA_USERNAME}:${ARTIFACTORY_SA_PASSWORD}"
           DOCKER_API="https://${ARTIFACTORY_URL}/artifactory/api/docker/kfd3-fd34fb-local/v2"
           SERVICES=(backend-services frontend temporal ches-adapter)
-          for svc in "${SERVICES[@]}"; do
-            echo "Checking ${svc}:${SHA_TAG}..."
+
+          # Returns 0 only on HTTP 200. A transient failure (curl error -> 000,
+          # or a 5xx) is retried by with_retries so an intermittent registry
+          # timeout doesn't fail the deploy; a persistent non-200 (e.g. 404 for a
+          # genuinely-missing image) still fails after the attempts are exhausted.
+          check_manifest() {
+            local svc="$1" code
             code=$(curl --connect-timeout 30 --max-time 120 -s -o /dev/null -w "%{http_code}" \
               -u "${AUTH}" -I \
               -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
               -H "Accept: application/vnd.oci.image.manifest.v1+json" \
               -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
               -H "Accept: application/vnd.oci.image.index.v1+json" \
               "${DOCKER_API}/${svc}/manifests/${SHA_TAG}" || echo "000")
-            if [[ "${code}" != "200" ]]; then
-              echo "::error::Missing staged image ${svc}:${SHA_TAG} (HTTP ${code})"
+            if [[ "${code}" == "200" ]]; then
+              return 0
+            fi
+            echo "[WARN] ${svc}:${SHA_TAG} not confirmed (HTTP ${code})" >&2
+            return 1
+          }
+
+          for svc in "${SERVICES[@]}"; do
+            echo "Checking ${svc}:${SHA_TAG}..."
+            if ! with_retries 3 15 check_manifest "${svc}"; then
+              echo "::error::Missing staged image ${svc}:${SHA_TAG}"
               exit 1
             fi
           done
@@ -546,13 +561,14 @@ jobs:
         run: |
           set -euo pipefail
           source scripts/lib/artifactory-login.sh
+          source scripts/lib/retry.sh
           artifactory_docker_login "${ARTIFACTORY_URL}" "${ARTIFACTORY_SA_USERNAME}" "${ARTIFACTORY_SA_PASSWORD}"
 
           REGISTRY="${ARTIFACTORY_URL}/kfd3-fd34fb-local"
           SERVICES=(backend-services frontend temporal ches-adapter)
           for svc in "${SERVICES[@]}"; do
             echo "Promoting ${svc}:${SHA_TAG} -> ${svc}:${FLOATING_TAG}"
-            docker buildx imagetools create \
+            with_retries 3 15 docker buildx imagetools create \
               "${REGISTRY}/${svc}:${FLOATING_TAG}" \
               "${REGISTRY}/${svc}:${SHA_TAG}"
           done

docs-md/operations/AUTO_DEPLOY.md

@@ -52,9 +52,14 @@ The workflow uses a per-ref concurrency group with `cancel-in-progress: true`. I
 | Prod (`main`) | `bcgov-di-<sha12>` | `bcgov-di` | `oc set image .../<svc>=<registry>/<svc>:bcgov-di-<old-sha12>` | Keep 3 most recent SHA tags per image |
 | Manual (`workflow_dispatch`) | `<branch-tag>-<sha12>` | `<branch-tag>` | Rebuild and redeploy | Keep 10 most recent SHA tags |
 
-## Artifactory login retries
+## Artifactory retries
 
-The build and promote steps retry `docker login` up to three times with a 15-second backoff (`scripts/lib/artifactory-login.sh`) to handle intermittent `Client.Timeout exceeded` errors against the registry.
+To handle intermittent `Client.Timeout exceeded` errors against the registry, registry operations retry up to three times with a 15-second backoff:
+
+- `docker login` in the build and promote steps (`scripts/lib/artifactory-login.sh`).
+- The deploy job's staged-image existence check and the `docker buildx imagetools create` promotion, via a shared `with_retries` helper (`scripts/lib/retry.sh`). The existence check only accepts HTTP 200, so a transient timeout (`000`) or `5xx` is retried while a genuinely-missing image still fails after the attempts are exhausted.
+
+All Artifactory REST/registry `curl` calls additionally use `--connect-timeout 30 --max-time 120`.
 
 ## Rollout failure handling
 

scripts/ARTIFACTORY_README.md

@@ -129,4 +129,4 @@ After successful prod deploys, CI keeps the 3 most recent `bcgov-di-????????????
 ./scripts/artifactory-cleanup.sh --env prod --delete --keep 3 --match 'bcgov-di-????????????'
 ```
 
-All Artifactory API calls use `--connect-timeout 30 --max-time 120`. Docker login retries up to 3 times via `scripts/lib/artifactory-login.sh`.
+All Artifactory API calls use `--connect-timeout 30 --max-time 120`. Docker login retries up to 3 times via `scripts/lib/artifactory-login.sh`, and the CI staged-image existence check and `imagetools create` tag promotion retry via `scripts/lib/retry.sh` (`with_retries`).

scripts/lib/retry.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+#
+# retry.sh — run a command with retries and fixed backoff.
+#
+# Handles intermittent registry/network failures (e.g. Artifactory
+# "Client.Timeout exceeded while awaiting headers") without failing the job on
+# the first blip.
+#
+
+# with_retries <max_attempts> <wait_seconds> <command> [args...]
+# Runs the command; on non-zero exit, waits and retries up to max_attempts.
+# Returns the command's exit status from the final attempt.
+with_retries() {
+  local max_attempts="$1"
+  local wait_seconds="$2"
+  shift 2
+  local attempt=1
+  local status=0
+
+  while true; do
+    if "$@"; then
+      return 0
+    else
+      status=$?
+    fi
+    if [[ "${attempt}" -ge "${max_attempts}" ]]; then
+      echo "[ERROR] Command failed after ${max_attempts} attempts: $*" >&2
+      return "${status}"
+    fi
+    echo "[WARN] Attempt ${attempt}/${max_attempts} failed (exit ${status}); retrying in ${wait_seconds}s..." >&2
+    sleep "${wait_seconds}"
+    attempt=$((attempt + 1))
+  done
+}

scripts/oc-build-push.sh

@@ -18,6 +18,7 @@ PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
 source "${SCRIPT_DIR}/lib/config-loader.sh"
 source "${SCRIPT_DIR}/lib/image-tag.sh"
 source "${SCRIPT_DIR}/lib/artifactory-login.sh"
+source "${SCRIPT_DIR}/lib/retry.sh"
 
 usage() {
   cat <<EOF
@@ -228,7 +229,7 @@ if [[ "${PROMOTE}" == true ]]; then
   else
     echo "[INFO] Promoting ${PUSH_TAG} -> ${FLOATING_TAG} for all built services..."
     for svc in "${SERVICES[@]}"; do
-      docker buildx imagetools create \
+      with_retries 3 15 docker buildx imagetools create \
         "${IMAGE_BASE}/${svc}:${FLOATING_TAG}" \
         "${IMAGE_BASE}/${svc}:${PUSH_TAG}"
     done