|
| 1 | +# Storage backends |
| 2 | + |
| 3 | +By default siphon keeps the dump catalog — every dump body plus its sidecar |
| 4 | +metadata — in a local directory. Phase G adds a pluggable storage layer so the |
| 5 | +catalog can instead live in an S3 (or S3-compatible) bucket, with `backup`, |
| 6 | +`restore`, `verify`, and `dumps` all reading and writing through it transparently. |
| 7 | + |
| 8 | +## Table of contents |
| 9 | + |
| 10 | +- [How it works](#how-it-works) |
| 11 | +- [Configuration](#configuration) |
| 12 | +- [Integrity across backends](#integrity-across-backends) |
| 13 | +- [Scope and limitations](#scope-and-limitations) |
| 14 | + |
| 15 | +## How it works |
| 16 | + |
| 17 | +The dump catalog (`internal/dumps`) holds a `storage.Store` and addresses |
| 18 | +objects by opaque keys — `<id>.dump` for the envelope-prefixed dump body and |
| 19 | +`<id>.meta.json` for the sidecar — never by filesystem path. The `Store` |
| 20 | +interface is small and backend-neutral: |
| 21 | + |
| 22 | +```go |
| 23 | +type Store interface { |
| 24 | + Put(ctx, key, io.Reader) error // durable, atomic-on-complete |
| 25 | + Get(ctx, key) (io.ReadCloser, error) // one-shot forward stream |
| 26 | + Delete(ctx, key) error // idempotent |
| 27 | + List(ctx) ([]string, error) |
| 28 | + Stat(ctx, key) (size int64, exists bool, err error) |
| 29 | +} |
| 30 | +``` |
| 31 | + |
| 32 | +Two backends ship today (`internal/storage`): |
| 33 | + |
| 34 | +- **local** — a single directory. `Put` writes to a temp file and renames, so a |
| 35 | + failed or cancelled write never leaves a partial dump under its final key. |
| 36 | + Keys map verbatim to file names, so a pre-Phase-G local catalog keeps working |
| 37 | + with no migration. |
| 38 | +- **s3** — an S3 or S3-compatible bucket (AWS, MinIO, Cloudflare R2). `Put` |
| 39 | + streams through the SDK's transfer manager, so the object only becomes |
| 40 | + addressable once the upload completes — the same atomic-on-complete guarantee |
| 41 | + the local backend gets from rename. |
| 42 | + |
| 43 | +A backup stages the dump body to a local temp file (the dump tool needs a real |
| 44 | +fd), then streams `envelope ++ body` into the store in a single `Put`, teeing |
| 45 | +through SHA-256 as the bytes flow. Restore and verify open the dump with `Get` |
| 46 | +and stream it straight into the envelope reader — no full local download. |
| 47 | + |
| 48 | +## Configuration |
| 49 | + |
| 50 | +Storage is selected by a top-level `storage:` block in the config file. Omitting |
| 51 | +it (or `type: local`) uses the local filesystem at `defaults.dump_dir`. |
| 52 | + |
| 53 | +```yaml |
| 54 | +version: 1 |
| 55 | +defaults: |
| 56 | + dump_dir: ~/.local/share/siphon/dumps # used by the local backend |
| 57 | + |
| 58 | +storage: |
| 59 | + type: s3 # "local" (default) | "s3" |
| 60 | + bucket: my-siphon-dumps # required for s3 |
| 61 | + prefix: prod # optional key prefix within the bucket |
| 62 | + region: us-east-1 |
| 63 | + endpoint: "" # optional: custom endpoint for S3-compatible services |
| 64 | +``` |
| 65 | +
|
| 66 | +For an S3-compatible service such as MinIO or R2, set `endpoint` to its URL |
| 67 | +(path-style addressing is used automatically). |
| 68 | + |
| 69 | +**Credentials are never stored in the config file.** The S3 backend resolves |
| 70 | +them from the standard AWS chain — `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` |
| 71 | +environment variables, the shared `~/.aws/config`, or an instance/role profile — |
| 72 | +so the config stays safe to commit. |
| 73 | + |
| 74 | +## Integrity across backends |
| 75 | + |
| 76 | +The SHA-256 checksum is computed over the `envelope ++ body` stream at write |
| 77 | +time and recomputed over the `Get` stream at verify time — both in siphon, not |
| 78 | +in the backend. A dump written to S3 and a dump written locally therefore verify |
| 79 | +identically, and `siphon verify` catches corruption regardless of where the dump |
| 80 | +lives. siphon does not trust the backend's own ETag/MD5 (multipart uploads do |
| 81 | +not expose a plain object MD5). |
| 82 | + |
| 83 | +The live S3 path is integration-tested in CI against MinIO via testcontainers |
| 84 | +(`internal/storage/s3_integration_test.go`) using the same `RunStoreSuite` |
| 85 | +contract that the local backend runs, so the streaming upload, ranged read, |
| 86 | +listing, and not-found mapping all execute against a real object store — not |
| 87 | +just compile. |
| 88 | + |
| 89 | +## Scope and limitations |
| 90 | + |
| 91 | +- Backends covered: **local** and **s3** (incl. S3-compatible). **GCS and Azure |
| 92 | + Blob are not implemented yet** — they are a fast-follow, and will get the full |
| 93 | + correctness bar for free by running the same `RunStoreSuite` contract. |
| 94 | +- `dumps list` over S3 issues one `ListObjectsV2` plus one read per metadata |
| 95 | + object (N+1). This is fine at expected catalog sizes; no pagination/parallel |
| 96 | + optimization is done yet. |
| 97 | +- Each `Get` is a fresh one-shot forward stream — callers must not assume the |
| 98 | + returned reader is seekable. |
| 99 | +- Retention/lifecycle (chain-aware pruning over remote storage) remains a |
| 100 | + separate Phase G concern; the `Store.Delete` it needs is delivered here. |
0 commit comments