fix(crud-service): self-recover from transient postgres pool init errors (#911)

[Cataldir]

Closes #911

Live diagnosis

Verified on holidaypeakhub405-dev-aks (running cluster):

• Flux GitRepository IS reconciled to current main@6b75f40d — the original commit-rendered-manifests failure described in [P1] deploy-azd: commit-rendered-manifests step fails, blocking Flux reconciliation and AKS image updates #911 is no longer the active blocker.

• holiday-peak-gitops-holiday-peak-crud Kustomization is permanently stuck in Reconciling/Progressing because the crud-service Deployment never goes Ready.

• holiday-peak-gitops-holiday-peak-agents Kustomization is False with dependency 'flux-system/holiday-peak-gitops-holiday-peak-crud' is not ready, so the entire downstream pipeline is gated.

• The CRUD pod runs and /health returns 200, but /ready returns 503 with:

  {"status":"degraded","checks":{"postgres":{"status":"unhealthy","detail":"TimeoutError: ; latest: TimeoutError: "}}}

• Inside the same pod, a fresh DefaultAzureCredential returns a 1980-byte Entra token in <1s and asyncpg.connect(...) followed by SELECT 1 succeeds. Postgres is reachable; only the in-process pool state is broken.

Root cause

BaseRepository.check_pool_health() short-circuited on a cached _pool_init_error:

if cls._pool_init_error and cls._pool is None:
    return "unhealthy", cls._pool_init_error

A single transient IMDS hiccup at startup (asyncio.TimeoutError) populated _pool_init_error and permanently bricked the readiness check, even though every subsequent retry would have succeeded.

Fix

apps/crud-service/src/crud_service/repositories/base.py — check_pool_health now always re-attempts initialize_pool() when the pool is absent. The cached error is reported only when the retry itself fails, and is cleared on success.

Tests

• 4 new unit tests in apps/crud-service/tests/unit/test_base_repository_pool_health.py:
  • happy path with initialized pool
  • stale _pool_init_error is cleared on successful retry (regression for [P1] deploy-azd: commit-rendered-manifests step fails, blocking Flux reconciliation and AKS image updates #911)
  • retry failure reports the latest exception
  • explicit no-short-circuit assertion (initialize_pool IS called even when an error is cached)
• All 6 existing test_health.py recovery tests continue to pass (10/10).
• Full crud-service suite: 265/265.
• Pre-push gate: 705 passed across lib + apps; isort/black/pylint/mypy/governance-link/event-schema all green.

Operational notes

• After merge + redeploy, the cluster operator can restart the deployment to pick up the fix:

  kubectl rollout restart deployment/crud-service-crud-service -n holiday-peak-crud
  

• Once the CRUD pod is Ready, the Flux Kustomization will move to Ready=True and the holiday-peak-agents Kustomization will unblock automatically.

• The original commit-rendered-manifests failure described in [P1] deploy-azd: commit-rendered-manifests step fails, blocking Flux reconciliation and AKS image updates #911 is no longer reproducible (Flux is on current main); if it returns, that's a separate workflow regression to track in a new issue.

ADR / governance

• ADR-033 (Flux GitOps deployment model) — fix preserves the readiness contract used by the Kustomization health check (Wait: true, Timeout: 10m0s).
• No public surface change to BaseRepository.