fix(infra): pgBackRest block incremental + prod Postgres data-volume sizing - #215
Merged
Conversation
…ters A prod outage occurred when WAL filled the app-pg data volume: the pgBackRest repo filled with huge hourly incrementals (~256MB each, re-copying whole changed file segments of the high-churn ocr_results table), which stalled WAL archiving, so Postgres stopped recycling WAL until pgdata hit 100% and the DB crashed. Enable block incremental (repo1-block) + bundling (repo1-bundle, required) on both the app and temporal Postgres clusters. Block incremental backs up only the changed blocks within a file instead of the whole file segment, which collapses those 256MB incrementals to a few MB and keeps the repo from filling. These were applied live during incident recovery; this codifies them so a redeploy does not revert the change. (pgBackRest 2.56 in cluster; needs 2.46+.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prod app Postgres data volume was expanded in-place from 1.5Gi to 5Gi after pg_wal accumulation filled the original volume and crashed the database (prod outage). The base manifest still requests 1.5Gi, so the Crunchy operator would emit the recurring "field can not be less than previous value" reconcile error. Pin app-pg pgdata to 5Gi via the prod-resources component (same pattern as temporal-pg). base stays 1.5Gi for ephemeral/test instances. Note: this only codifies the storage size. The underlying cause of the WAL growth is the pgBackRest backup repo (repo1) being full, which blocks WAL archiving — that must be addressed separately or the volume will refill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alex-struk
changed the title
alex-struk
marked this pull request as ready for review
June 29, 2026 18:04
alex-struk
requested review from
NoorChasib,
antsand,
dbarkowsky and
kmandryk
as code owners
June 29, 2026 18:04
dbarkowsky
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Prod outage (2026-06-29): the
app-pgdata volume filled to 100% and Postgres crashed, taking down backend-services and temporal-worker. Root cause chain:ocr_resultstable (309MB, mostly TOAST) is appended to constantly by document processing.archive-pushfailed with "No space left on device" → WAL archiving stalled → Postgres stopped recycling WAL →pg_walgrew to 1.1G → filled the 1.5Gi pgdata → crash.Change
Enable block incremental (
repo1-block=y) + bundling (repo1-bundle=y, required by block) on both the app and temporal PostgresCluster manifests. Block incremental backs up only the changed blocks within a file instead of the whole changed segment, collapsing those 256MB incrementals to a few MB and keeping the repo from filling.pgBackRest 2.56 is running in-cluster (block incremental needs 2.46+).
Incident recovery already performed live (this PR codifies the durable part)
During the outage I:
repo1-block/repo1-bundlelive on app-pg.Result: WAL archiving resumed (
archived_countclimbing, failures stopped),pg_waldrained 1.5G → 258M, pgdata back to 15%, all app pods healthy.This PR ensures a redeploy doesn't revert the block/bundle setting (it was a live
occhange). It applies to all instances — block incremental is a safe, forward-only format improvement.Validation
oc kustomizeof base shows both clusters withblock=y bundle=y.🤖 Generated with Claude Code