fix(infra): pin prod app-pg data volume to 5Gi - #214
Closed
alex-struk wants to merge 1 commit into
Closed
Conversation
The prod app Postgres data volume was expanded in-place from 1.5Gi to 5Gi after pg_wal accumulation filled the original volume and crashed the database (prod outage). The base manifest still requests 1.5Gi, so the Crunchy operator would emit the recurring "field can not be less than previous value" reconcile error. Pin app-pg pgdata to 5Gi via the prod-resources component (same pattern as temporal-pg). base stays 1.5Gi for ephemeral/test instances. Note: this only codifies the storage size. The underlying cause of the WAL growth is the pgBackRest backup repo (repo1) being full, which blocks WAL archiving — that must be addressed separately or the volume will refill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Folded into #215 so all the outage-related infra fixes (block incremental + app-pg 5Gi pin) are in a single PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Prod went down (2026-06-29) when the
bcgov-di-app-pgPostgres data volume (1.5Gi) filled to 100% and the database crashed — taking backend-services and temporal-worker down with it (both depend on this DB). The volume was expanded live 1.5Gi → 5Gi to restore service.Change
Pin app-pg pgdata to 5Gi (prod-only, via the
prod-resourcescomponent) so the manifest matches the live PVC. Without this, the Crunchy operator would keep trying to reconcile it down to the base 1.5Gi and emit the recurringspec.resources.requests.storage: field can not be less than previous valueerror (PVCs can't shrink). Same pattern already used for temporal-pg. base/test stay at 1.5Gi.Validation
oc kustomizeof the generated overlay:5Gi, temporal-pg5Gi1.5Gi, temporal-pg1.5GiThe disk filled because of WAL accumulation, and WAL accumulated because the pgBackRest backup repo (
repo1, 10Gi) is 100% full →archive-pushfails with "No space left on device" → Postgres won't recycle un-archived WAL. The repo is full of an old full backup (06-27) plus a long chain of ~260M hourly incrementals that aren't being expired.WAL archiving is still failing in prod right now, so the 5Gi volume will refill unless the backup repo is addressed:
netapp-file-backupquota (currently maxed at 32Gi: app 10Gi + temporal 22Gi).This PR only stops the manifest-vs-live drift; the repo/retention fix is tracked separately.
🤖 Generated with Claude Code