-
-
Notifications
You must be signed in to change notification settings - Fork 0
feat(retention): chain-aware retention and pruning #8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Retention & pruning | ||
|
|
||
| `siphon prune` applies a retention policy to the dump catalog, deleting old | ||
| backups while guaranteeing it never orphans an incremental from its base. It | ||
| works against any storage backend (local or S3), since it prunes through the | ||
| same `Store.Delete` the catalog already uses. | ||
|
|
||
| ## Table of contents | ||
|
|
||
| - [The chain is the unit](#the-chain-is-the-unit) | ||
| - [Policy rules](#policy-rules) | ||
| - [Configuration](#configuration) | ||
| - [The CLI](#the-cli) | ||
| - [Safety properties](#safety-properties) | ||
|
|
||
| ## The chain is the unit | ||
|
|
||
| An incremental dump depends on its base (and any intermediate incrementals) to | ||
| restore. Retention therefore operates on **chains**, not individual dumps: a base | ||
| plus its incrementals is kept or deleted as a whole. A full backup is a | ||
| single-member chain. Counting and bucketing are done over chains, so "keep the | ||
| last 7" means seven restorable backups, not seven files. | ||
|
|
||
| A chain's age is the timestamp of its **newest** member, so a chain that is | ||
| still being appended to (an old base with a fresh incremental) is treated as | ||
| young and is never pruned mid-life. | ||
|
|
||
| ## Policy rules | ||
|
|
||
| Three rules, each independently enableable. A chain is kept if it satisfies | ||
| **any** active rule (union) — adding a rule can only protect more chains, never | ||
| fewer, so a policy change can't silently delete data a rule meant to keep. | ||
|
|
||
| | Rule | Meaning | | ||
| | --- | --- | | ||
| | `keep_last: N` | keep the N newest chains | | ||
| | `max_age: <duration>` | keep chains younger than the duration (Go duration string, e.g. `720h`) | | ||
| | `gfs: {daily, weekly, monthly}` | grandfather-father-son: keep the newest chain in each of the most-recent N days / ISO weeks / months | | ||
|
|
||
| An **all-zero / omitted policy keeps everything** — prune becomes a no-op. The | ||
| destructive direction always requires explicit configuration. | ||
|
|
||
| ## Configuration | ||
|
|
||
| Retention lives in a `retention:` block. Set a default for all profiles, and | ||
| optionally override per profile (the profile block **replaces** the default | ||
| block wholesale — it is not field-merged). | ||
|
|
||
| ```yaml | ||
| defaults: | ||
| retention: | ||
| keep_last: 7 | ||
| max_age: 720h # 30 days | ||
| gfs: { daily: 7, weekly: 4, monthly: 6 } | ||
|
|
||
| profiles: | ||
| prod: | ||
| driver: postgres | ||
| # ... | ||
| retention: # prod keeps more, independent of the default | ||
| keep_last: 30 | ||
| gfs: { daily: 14, weekly: 8, monthly: 12 } | ||
| ``` | ||
|
|
||
| Precedence, highest first: **CLI flags → profile block → defaults block → | ||
| built-in (keep everything)**. An invalid block (negative count, unparseable | ||
| duration) fails fast at config load. | ||
|
|
||
| ## The CLI | ||
|
|
||
| ```bash | ||
| # Dry-run (default): show which chains the policy would prune, for one profile. | ||
| siphon prune --profile prod | ||
|
|
||
| # Override the configured policy for this run, then actually delete. | ||
| siphon prune --profile prod --keep-last 14 --apply | ||
|
|
||
| # Pure flag-driven policy (no config), GFS only. | ||
| siphon prune --gfs-daily 7 --gfs-weekly 4 --gfs-monthly 6 --apply | ||
| ``` | ||
|
|
||
| `--apply` performs deletions; without it, prune only prints the plan. Flags | ||
| (`--keep-last`, `--max-age`, `--gfs-daily/-weekly/-monthly`) override the | ||
| config-derived policy, but only when explicitly set — an unset flag never zeroes | ||
| out a configured rule. | ||
|
|
||
| ## Safety properties | ||
|
|
||
| - **Dry-run by default** — deletion requires an explicit `--apply`. | ||
| - **Keep-everything on empty** — a missing or all-zero policy prunes nothing. | ||
| - **Union semantics** — every rule can only protect; none can cause a surprise deletion. | ||
| - **Chain-atomic** — a base is only deleted when its whole chain is pruned, so an incremental is never left unrestorable. | ||
| - **Leaf-inward deletion** — within a pruned chain, incrementals are deleted before the base, so an interrupted prune (Ctrl-C, network blip) leaves at worst a complete shorter chain, never a base missing under a surviving incremental. | ||
| - **Collected failures** — a single dump that fails to delete is reported and the run exits non-zero, but the remaining chains are still pruned. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| package app | ||
|
|
||
| import ( | ||
| "context" | ||
| "time" | ||
|
|
||
| "github.com/nixrajput/siphon/internal/dumps" | ||
| ) | ||
|
|
||
| // PruneOpts configures a prune run. Policy is resolved by the caller (the CLI | ||
| // maps config + flags into it), so this layer stays config-agnostic. Profile | ||
| // scopes the catalog to one profile's dumps ("" = all profiles). Apply performs | ||
| // deletions; otherwise the run is a dry-run that only computes the plan. | ||
| type PruneOpts struct { | ||
| Profile string | ||
| Policy dumps.RetentionPolicy | ||
| Apply bool | ||
| } | ||
|
|
||
| // ChainOutcome is one chain's place in the plan, flattened for reporting. | ||
| type ChainOutcome struct { | ||
| Root string | ||
| DumpIDs []string | ||
| SizeBytes int64 | ||
| Pruned bool // true = scheduled for (or performed) deletion | ||
| Deleted []string // dump IDs actually deleted (Apply only) | ||
| Errors []string // per-dump deletion failures (Apply only) | ||
| } | ||
|
|
||
| // PruneResult is the structured outcome the CLI renders. | ||
| type PruneResult struct { | ||
| Profile string | ||
| Apply bool | ||
| Outcomes []ChainOutcome | ||
| Reclaimed int64 // bytes freed (sum of successfully deleted dumps; Apply only) | ||
| Failed int // count of dumps that failed to delete | ||
| } | ||
|
|
||
| // Prune groups the catalog into chains, plans retention over them, and (when | ||
| // Apply) deletes the pruned chains. It is synchronous (like Verify): prune is a | ||
| // list + a few deletes, not a long stream, so it returns a structured result | ||
| // directly rather than over the job channel. | ||
| // | ||
| // Deletion is chain-aware and leaf-inward: within a pruned chain, incrementals | ||
| // are removed before the base, so an interrupted prune leaves at worst a | ||
| // complete shorter chain — never a base missing under a surviving incremental. | ||
| func Prune(ctx context.Context, d Deps, opt PruneOpts) (*PruneResult, error) { | ||
| all, err := d.Dumps.List(ctx) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| // Scope to the requested profile before grouping, so chains and the plan | ||
| // reflect only this profile's backups. | ||
| if opt.Profile != "" { | ||
| filtered := make([]dumps.Meta, 0, len(all)) | ||
| for _, m := range all { | ||
| if m.Profile == opt.Profile { | ||
| filtered = append(filtered, m) | ||
| } | ||
| } | ||
| all = filtered | ||
| } | ||
|
|
||
| plan := dumps.Plan(dumps.GroupChains(all), opt.Policy, time.Now()) | ||
|
|
||
| result := &PruneResult{Profile: opt.Profile, Apply: opt.Apply} | ||
| for _, ch := range plan.Keep { | ||
| result.Outcomes = append(result.Outcomes, chainOutcome(ch, false)) | ||
| } | ||
| for _, ch := range plan.Prune { | ||
| oc := chainOutcome(ch, true) | ||
| if opt.Apply { | ||
| applyChainDeletion(ctx, d, ch, &oc, result) | ||
| } | ||
| result.Outcomes = append(result.Outcomes, oc) | ||
| } | ||
| return result, nil | ||
| } | ||
|
|
||
| // applyChainDeletion deletes a pruned chain's members leaf-inward (incrementals | ||
| // before the base). A per-dump failure is collected and does not abort the rest | ||
| // of the chain or the run — a single stuck object should not block reclaiming | ||
| // the others. | ||
| func applyChainDeletion(ctx context.Context, d Deps, ch dumps.Chain, oc *ChainOutcome, result *PruneResult) { | ||
| for i := len(ch.Members) - 1; i >= 0; i-- { | ||
| m := ch.Members[i] | ||
| if err := d.Dumps.Delete(ctx, m.ID); err != nil { | ||
| oc.Errors = append(oc.Errors, m.ID+": "+err.Error()) | ||
| result.Failed++ | ||
| continue | ||
|
nixrajput marked this conversation as resolved.
Outdated
|
||
| } | ||
| oc.Deleted = append(oc.Deleted, m.ID) | ||
| result.Reclaimed += m.SizeBytes | ||
| } | ||
| } | ||
|
|
||
| func chainOutcome(ch dumps.Chain, pruned bool) ChainOutcome { | ||
| oc := ChainOutcome{Root: ch.Root, Pruned: pruned} | ||
| for _, m := range ch.Members { | ||
| oc.DumpIDs = append(oc.DumpIDs, m.ID) | ||
| oc.SizeBytes += m.SizeBytes | ||
| } | ||
| return oc | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.