| name | certifying-backups |
|---|---|
| description | Use when backups exist but nobody has proven they restore, when a backup job should be verified on a schedule instead of trusted, or when you would otherwise discover a corrupt backup on the day you need it. |
Restore one backup per iteration into a scratch location and prove it is usable: the archive opens, the data loads, and a checksum or row count matches expectations. The loop replaces the hope that backups work with a rolling certificate that they do.
Core principle: one backup certified per iteration, every iteration starts by checking the exit conditions and ends by writing the ledger.
- A nightly dump, snapshot, or archive job runs and is assumed to work
- A compliance or disaster-recovery review needs evidence of restorability
- After changing backup tooling, retention, or encryption settings
When NOT to use: production restore drills that need a maintenance window and a human on call, run those as planned exercises instead; backups you lack credentials to read, report the gap rather than looping on failures.
All state lives in .loop/certify-backups.md, called the ledger. Every entry records: backup id and date, source system, restore target path, verification method and its result (checksum, row count, or sample query), duration, and running counters: consecutive failures per source, backups certified this cycle.
Counter rules: a passing certification resets that source's consecutive-failure counter to zero. A cycle is one pass over every backup in scope; the certified-this-cycle counter resets when the success exit fires or when a new scheduled run starts.
Iteration 1 bootstrap: if the ledger does not exist, create .loop/ and the file, list the backups in scope (aws s3 ls s3://backups/db/ | tail -30 or the equivalent for your store), and write a baseline entry: backup inventory, newest certified id per source (none), each source's verification method and expected ranges as given in the loop instruction or the repo's backup docs, and all counters zero. If a source's verification method is given nowhere, stop and report it rather than inventing one. That is the whole first iteration, always; certification starts at iteration 2.
- Read the ledger (bootstrap it if missing).
- List backups newer than the newest certified id per source, plus the oldest backup still inside the retention window that has never been certified.
- Check EVERY exit condition in the table below against the fresh listing and the ledger counters. Any row matches: do its action and stop. This happens before any work.
- Nothing uncertified: write a status entry, end the iteration.
- Pick the FIRST uncertified backup in the listing, oldest first, ties broken by source name alphabetically (deterministic order beats clever prioritization). Restore it into a scratch path or scratch database that is wiped first. Never restore toward any production path.
- Verify by the method recorded for that source: compare the stored checksum, or load and count rows against the expected range, or run the source's sample query. Success requires the verification output, not just a completed restore command.
- Clean up the scratch location, then write the ledger entry: pass or fail, evidence, counters updated.
| Condition | Action |
|---|---|
| Every backup in scope certified this cycle | Summarize the certificate (ids, dates, methods), stop |
| Same source failed verification 2 consecutive times | Stop, report: backups for that source are not restorable, with both failure outputs |
| A restore wrote outside the scratch path | Stop immediately, report exactly what was written where; do not clean up |
| Budget reached (default 10 certifications per run) | Stop, report what remains uncertified |
Any stop-and-report lists every source with uncertified or failed backups, not just the one being worked.
Daily or weekly via a scheduler; certification follows the backup cadence, not a tight loop. Use /schedule or cron. A timed /loop only makes sense when burning down a backlog of never-certified backups in one sitting, and self-paced is better for that.
- Certifying only the newest backup: the old one you will actually reach for during an incident stays unproven. The oldest-first pick order exists for this.
- Treating a completed restore command as success: a restore that exits zero can still produce an empty database. Verification output is the certificate.
- Restoring toward any path a production system reads: scratch means scratch, and the exit table treats a stray write as an emergency stop.
- Letting counters live in your head: every counter the exit table needs must be in the ledger.
/schedule weekly Certify this week's database backups using certifying-backups.
Budget: 10 certifications.
Ledger entry the loop should produce:
[2026-06-08 02:14] backup db-2026-06-07T01:00 (postgres/main), restored to
/scratch/certify/db-main in 4m12s. Verification: row count 1,482,330 within
expected 1.4M-1.6M, sample query ok. PASS. Counters: main failures 0,
certified this cycle 6/10.