fix(restore): fall back to pg_resetwal when WAL recovery fails#49
Merged
Conversation
When a source snapshot is captured mid-online-backup the trailing WAL
segments needed to complete recovery are not present in the data
dir, and postgres fails to start with "WAL ends before end of online
backup" (or a related recovery error). Without intervention the init
container crashloops indefinitely.
For an analytics replica the priority is availability over byte-
perfect consistency. Add a pg_resetwal -f fallback in the postgres
--single locale-fix step in the init script, with two stages:
- Stage 1 (detected): if the failure log matches a known WAL-
recovery signature ("WAL ends before end of online backup",
"invalid record length at", "database system was interrupted
while in recovery", "could not locate required checkpoint
record"), short-circuit directly to pg_resetwal + retry. Retrying
the same command won't help when recovery itself is the blocker.
- Stage 2 (undetected): if the failure doesn't match a known
signature, try the same command once more — could be a transient
catalog / I/O blip. Only if that retry also fails, run pg_resetwal
as a last resort and retry once.
pg_resetwal does not restore missing data; it leaves the data dir at
whatever state the snapshot captured. That's acceptable for read-
only analytics — a replica that comes up with the snapshot's data
is more useful than one that never comes up at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a source snapshot is captured mid-online-backup the trailing WAL segments needed to complete recovery aren't present in the data dir, and postgres fails to start with
WAL ends before end of online backup(or a related recovery error). Without intervention the init container crashloops indefinitely and the replica never comes up.For an analytics replica the priority is availability over byte-perfect consistency. Add a
pg_resetwal -ffallback in the postgres-single locale-fix step of the init script, with two stages:WAL ends before end of online backup,invalid record length at,database system was interrupted while in recovery,could not locate required checkpoint record), short-circuit directly topg_resetwal+ retry. Retrying the same command won't help when recovery itself is the blocker.pg_resetwalas a last resort and retry once.pg_resetwaldoes not restore missing data; it leaves the data dir at whatever state the snapshot captured. That's acceptable for read-only analytics — a replica that comes up with the snapshot's data is more useful than one that never comes up at all.Unit test on the init script verifies both branches are present (signature detection, dual
pg_resetwalcalls, same-settings retry before the last-resort reset).