Phase G: multi-backend secrets (keychain + AWS Secrets Manager) (#10)

Author: nixrajput

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Phase G (ops) — **multi-backend secrets** (final G cycle): two new secret-ref backends on the existing resolver seam, so passwords can come from a credential store instead of config.
+  - **OS keychain** (`keychain://<account>` or `keychain://<service>/<account>`) via `go-keyring` — macOS Keychain, Windows Credential Manager, Linux Secret Service. No config, no network; short form looks up service `siphon`.
+  - **AWS Secrets Manager** (`awssm://<secret-id>`, or `awssm://<secret-id>#<json-key>` to pull one field of a JSON secret) via the AWS SDK, reusing the same credential chain as S3 storage. Off by default; enable with `secrets.awssm: true` (+ optional `awssm_region`).
+  - Registered in resolution order `env → keychain → awssm → passthrough`; an unknown scheme still falls through to a literal value. Backends unit-tested via the keyring mock and a faked Secrets Manager client (no real cloud). See [docs/OPS.md](docs/OPS.md#secret-backends). **Phase G (ops pillar) is now complete.**
+
 - Phase G (ops) — **operational suite** (audit, 2FA gating, telemetry, schedule, tunnel):
   - **Audit log** (`internal/audit`): an append-only JSONL trail of destructive ops (backup/restore/sync/prune) recording who/what/when/outcome, via a single `guardedOp` interception seam wired into the verbs. Off by default; enable with the `audit:` config block. The same seam is reused by 2FA gating and telemetry (no per-feature re-wrapping).
   - **2FA / group gating** (`internal/twofactor`): a profile's group can require a typed-name confirmation (`confirm_destructive`) and/or a TOTP code (`require_2fa`) before a destructive op runs. Stdlib RFC 6238 TOTP (HMAC-SHA1, 30s, 6 digits, ±1-step skew), no new dependency; the group's `totp_secret` is a secret-ref. `require_2fa` with no resolvable secret fails closed.

README.md

@@ -42,7 +42,7 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_
 - **Native, not reimplemented.** siphon shells out to `pg_dump`/`pg_restore`, `mysqldump`/`mysql`, and `mariadb-dump`/`mariadb` for the actual data movement — you inherit 20+ years of correctness from the official tools, wrapped in a consistent UX.
 - **Integrity by default.** Every dump is checksummed (SHA-256) and recorded in a sidecar metadata file. `siphon verify` re-hashes the dump and flags corruption or tampering — and fails with a distinct exit code so CI can catch it.
 - **Built for scripts and humans.** A Cobra command tree with predictable flags and POSIX exit codes for automation; an interactive Bubble Tea dashboard when you invoke `siphon` bare.
-- **Named profiles + secret refs.** Store connection details once; reference secrets as `env:VAR` today, with OS keychain / Vault / 1Password / AWS Secrets Manager backends on the roadmap. Plaintext passwords never have to live in your config.
+- **Named profiles + secret refs.** Store connection details once; with a secret ref like `env:VAR`, `keychain://<account>` (OS credential store), or `awssm://<id>#<key>` (AWS Secrets Manager), the password is resolved at runtime instead of being stored in config. (A value with no scheme is a literal, so don't commit a plaintext password.)
 - **Streaming sync.** `siphon sync src dst` pipes a backup straight into a restore with no intermediate file on disk.
 
 ## Project status
@@ -55,7 +55,7 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_
 | **D** — Driver hardening          | Shared cross-driver test harness (`RunDriverSuite`), capability-gating helper (`RequireCapability`), Postgres connection-probe retry, and a `docs/DRIVERS.md` contributor guide                                                                                                                                                                                                                                                         | ✅ Complete |
 | **E** — MySQL + MariaDB           | Both drivers via a shared `_mysqlcommon` package (shared `Conn`, `mysqldump`/`mariadb-dump` backup, client-pipe restore), exercised by the Phase D `RunDriverSuite` harness                                                                                                                                                                                                                                                             | ✅ Complete |
 | **F** — Advanced transfer         | All four advanced-transfer modes work end-to-end: bounded-buffer streaming sync; **incremental** backup/restore (`backup --incremental --base <id>` captures a bounded change set via Postgres logical decoding / MySQL-MariaDB binlog, `restore` replays the base→incremental chain, Postgres orphan-slot sweep); **cross-engine** sync (`sync --cross-engine` — typed `SchemaInspector` introspection → canonical type-mapping, e.g. Postgres → MySQL); and **CDC** (`siphon cdc` / `sync --continuous` — unbounded change streaming with snapshot→stream handoff, resumable, same- and cross-engine). Live DB paths are integration-tested in CI — see [docs/INCREMENTAL.md](docs/INCREMENTAL.md), [docs/CROSS_ENGINE.md](docs/CROSS_ENGINE.md), [docs/CDC.md](docs/CDC.md) | ✅ Complete |
-| **G** — Ops features              | **Cloud storage** (✅): the dump catalog can live in an S3 / S3-compatible bucket via a pluggable `storage.Store` backend (local + S3; GCS/Azure are a fast-follow), `storage:` config block, SHA-256 integrity end-to-end — see [docs/STORAGE.md](docs/STORAGE.md). **Retention** (✅): chain-aware pruning (`siphon dumps prune`) with keep-last-N / max-age / GFS rules, per-profile `retention:` config; see [docs/RETENTION.md](docs/RETENTION.md). **Ops suite** (✅): an append-only **audit log** of destructive ops (`audit:` config), **2FA/group gating** (a profile group can require a typed confirmation and/or TOTP before destructive ops), opt-in aggregate **telemetry** (`telemetry:` config), **`siphon schedule`** (manages recurring backups in your crontab), and **`siphon tunnel`** (SSH local-forward to a DB via a bastion). Multi-backend secrets are the remaining G item; see [docs/OPS.md](docs/OPS.md).                                                                                                                                                                                                                                                              | 🟡 In progress |
+| **G** — Ops features              | **Cloud storage** (✅): the dump catalog can live in an S3 / S3-compatible bucket via a pluggable `storage.Store` backend (local + S3; GCS/Azure are a fast-follow), `storage:` config block, SHA-256 integrity end-to-end — see [docs/STORAGE.md](docs/STORAGE.md). **Retention** (✅): chain-aware pruning (`siphon dumps prune`) with keep-last-N / max-age / GFS rules, per-profile `retention:` config; see [docs/RETENTION.md](docs/RETENTION.md). **Ops suite** (✅): an append-only **audit log** of destructive ops (`audit:` config), **2FA/group gating** (a profile group can require a typed confirmation and/or TOTP before destructive ops), opt-in aggregate **telemetry** (`telemetry:` config), **`siphon schedule`** (manages recurring backups in your crontab), and **`siphon tunnel`** (SSH local-forward to a DB via a bastion). **Multi-backend secrets** (✅): `keychain://` (OS credential store) and `awssm://` (AWS Secrets Manager) join `env:` as secret-ref schemes. See [docs/OPS.md](docs/OPS.md).                                                                                                                                                                                                                                                              | ✅ Complete |
 | **H** — Distribution              | GoReleaser, Homebrew tap, Scoop bucket, install script, docs site                                                                                                                                                                                                                                                                                                                                                                       | ⏳ Planned  |
 
 ## Requirements
@@ -166,7 +166,7 @@ profiles:
     sslmode: require
 ```
 
-Secret references (`env:VAR`) are resolved at runtime, so the config file is safe to commit. Plain values are also accepted.
+Secret references are resolved at runtime, which keeps the password out of the config file when you use one. Supported schemes: `env:VAR` (environment variable), `keychain://<account>` or `keychain://<service>/<account>` (OS keychain — macOS Keychain / Windows Credential Manager / Linux Secret Service), and `awssm://<secret-id>` or `awssm://<secret-id>#<json-key>` (AWS Secrets Manager; enable with `secrets.awssm: true`). A value matching no scheme is treated as a literal — so a plaintext password lives in the file and should not be committed. See [docs/OPS.md](docs/OPS.md#secret-backends).
 
 By default the dump catalog lives on local disk at `defaults.dump_dir`. To store dumps in an S3 / S3-compatible bucket instead, add a `storage:` block:
 

docs/OPS.md

@@ -120,3 +120,51 @@ siphon tunnel prod
 Run it in one terminal and point a client (or another siphon command) at the
 printed local address in another. siphon delegates to `ssh -L` rather than
 reimplementing SSH or holding a connection in a daemon.
+
+## Secret backends
+
+Any profile field that holds a secret (today: the password) is a **secret-ref**,
+resolved at runtime by a pluggable backend keyed on the ref's scheme. The config
+file therefore never has to contain a plaintext secret. A ref matching no known
+scheme is treated as a literal value, so an ordinary password still works.
+
+| Scheme | Ref shape | Resolves from |
+| --- | --- | --- |
+| `env` | `env:VAR` | the `VAR` environment variable |
+| `keychain` | `keychain://<account>` or `keychain://<service>/<account>` | the OS credential store (macOS Keychain, Windows Credential Manager, Linux Secret Service) |
+| `awssm` | `awssm://<secret-id>` or `awssm://<secret-id>#<json-key>` | AWS Secrets Manager |
+| *(none)* | `hunter2` | a literal value (passthrough) |
+
+```yaml
+profiles:
+  prod:
+    driver: postgres
+    password: keychain://prod-db        # OS keychain, service "siphon", account "prod-db"
+  staging:
+    driver: postgres
+    password: awssm://staging/db#password   # the "password" field of a JSON secret
+```
+
+**OS keychain** (`keychain://`) needs no config and no network — it reads the
+local credential store via a cross-platform keyring. The short form
+`keychain://<account>` looks up service `siphon`; the two-segment form addresses
+any stored credential. Store one with your OS tools (e.g.
+`security add-generic-password -s siphon -a prod-db -w` on macOS).
+
+**AWS Secrets Manager** (`awssm://`) is **off by default** — enable it so a
+machine without AWS credentials doesn't pay the config-load cost:
+
+```yaml
+secrets:
+  awssm: true
+  awssm_region: us-east-1   # optional; defaults to the AWS credential chain's region
+```
+
+It reuses the standard AWS credential chain (the same one S3 storage uses), so
+no separate credentials live in config. The `#<json-key>` selector extracts one
+field from a JSON secret — common for Secrets Manager entries like
+`{"username":...,"password":...}` — so a ref resolves to the field a DSN needs
+rather than the whole blob.
+
+A missing key, unknown field, or non-string field is a clear user error; a
+backend/transport failure is a system error.

go.mod

@@ -8,6 +8,7 @@ require (
 	github.com/aws/aws-sdk-go-v2/credentials v1.19.24
 	github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.2.11
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0
+	github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.42.3
 	github.com/aws/smithy-go v1.27.1
 	github.com/charmbracelet/bubbles v1.0.0
 	github.com/charmbracelet/bubbletea v1.3.10
@@ -25,6 +26,7 @@ require (
 	github.com/testcontainers/testcontainers-go/modules/minio v0.42.0
 	github.com/testcontainers/testcontainers-go/modules/mysql v0.42.0
 	github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
+	github.com/zalando/go-keyring v0.2.8
 	golang.org/x/term v0.40.0
 	gopkg.in/yaml.v3 v3.0.1
 )
@@ -68,6 +70,7 @@ require (
 	github.com/containerd/log v0.1.0 // indirect
 	github.com/containerd/platforms v0.2.1 // indirect
 	github.com/cpuguy83/dockercfg v0.3.2 // indirect
+	github.com/danieljoos/wincred v1.2.3 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/distribution/reference v0.6.0 // indirect
 	github.com/docker/go-connections v0.6.0 // indirect
@@ -79,6 +82,7 @@ require (
 	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-ole/go-ole v1.2.6 // indirect
+	github.com/godbus/dbus/v5 v5.2.2 // indirect
 	github.com/google/uuid v1.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgio v1.0.0 // indirect

go.sum

@@ -40,6 +40,8 @@ github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29 h1:hiME6pBzC7OTl
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.29/go.mod h1:G7RP+uhagpKtKhd1BM9N6JQqjCcGEU47K5lBVZQyRQw=
 github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0 h1:ta8csKy5vN91F3i5gGR85lFV0srBqySEji7Jroes6rE=
 github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0/go.mod h1:77ZAgynvx1txMvDG8gGWoWkO1augYDxkp9JElWFgjQU=
+github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.42.3 h1:L9gPLf3sFH1/ao3oB2QZcaX1xGYi8hj+WJlsf3/dN+M=
+github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.42.3/go.mod h1:9DKRlwDCw2OUDlyCIFcQCroL5M0mQTUU9qW8JEDcXmI=
 github.com/aws/aws-sdk-go-v2/service/signin v1.2.0 h1:3nXpRcFwRCW8n7HgO2QGy0Dc20eQNfBuUemGQhpF8m8=
 github.com/aws/aws-sdk-go-v2/service/signin v1.2.0/go.mod h1:LxYujSTLPRlp2vTtcUO/+1ilrew8ytt6SvQyOgejzFQ=
 github.com/aws/aws-sdk-go-v2/service/sso v1.31.3 h1:ey1XLTYXb9PcLt4535632o5kCGXNXEhNb620Dqwuylo=
@@ -111,6 +113,8 @@ github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHf
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
 github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
+github.com/danieljoos/wincred v1.2.3 h1:v7dZC2x32Ut3nEfRH+vhoZGvN72+dQ/snVXo/vMFLdQ=
+github.com/danieljoos/wincred v1.2.3/go.mod h1:6qqX0WNrS4RzPZ1tnroDzq9kY3fu1KwE7MRLQK4X0bs=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -137,6 +141,8 @@ github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
 github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
 github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw=
 github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
+github.com/godbus/dbus/v5 v5.2.2 h1:TUR3TgtSVDmjiXOgAAyaZbYmIeP3DPkld3jgKGV8mXQ=
+github.com/godbus/dbus/v5 v5.2.2/go.mod h1:3AAv2+hPq5rdnr5txxxRwiGjPXamgoIHgz9FPBfOp3c=
 github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
@@ -273,6 +279,8 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
 github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
+github.com/zalando/go-keyring v0.2.8 h1:6sD/Ucpl7jNq10rM2pgqTs0sZ9V3qMrqfIIy5YPccHs=
+github.com/zalando/go-keyring v0.2.8/go.mod h1:tsMo+VpRq5NGyKfxoBVjCuMrG47yj8cmakZDO5QGii0=
 go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
 go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 h1:sbiXRNDSWJOTobXh5HyQKjq6wUC5tNybqjIqDpAY4CU=

internal/cli/root.go

@@ -74,7 +74,10 @@ func buildDeps() (app.Deps, error) {
 	if err != nil {
 		return app.Deps{}, err
 	}
-	res := secrets.NewResolver(secrets.Env{}, secrets.Passthrough{})
+	res, err := buildResolver(cfg)
+	if err != nil {
+		return app.Deps{}, err
+	}
 	ps := profile.New(cfg, res, config.Save)
 
 	store, err := buildStore(cfg)
@@ -166,6 +169,23 @@ func osUser() string {
 	return "unknown"
 }
 
+// buildResolver assembles the secret-resolution backends. env: and keychain://
+// are always available; awssm:// is added only when enabled in config (its
+// construction loads AWS config). Passthrough MUST stay last — it is the
+// fallback that returns a literal cleartext value when no scheme matches.
+func buildResolver(cfg *config.Config) (*secrets.Resolver, error) {
+	backends := []secrets.Backend{secrets.Env{}, secrets.Keychain{}}
+	if cfg.Secrets.AWSSM {
+		sm, err := secrets.NewAWSSM(context.Background(), cfg.Secrets.AWSSMRegion)
+		if err != nil {
+			return nil, fmt.Errorf("init aws secrets manager backend: %w", err)
+		}
+		backends = append(backends, sm)
+	}
+	backends = append(backends, secrets.Passthrough{})
+	return secrets.NewResolver(backends...), nil
+}
+
 // buildStore selects the dump-catalog storage backend from config. Type "s3"
 // builds an S3-backed store; anything else (the default) uses the local
 // filesystem rooted at Defaults.DumpDir (or the XDG share dir when unset).

internal/config/config.go

@@ -17,6 +17,7 @@ type Config struct {
 	Storage   StorageConfig            `yaml:"storage"`
 	Audit     AuditConfig              `yaml:"audit"`
 	Telemetry TelemetryConfig          `yaml:"telemetry"`
+	Secrets   SecretsConfig            `yaml:"secrets"`
 	Profiles  map[string]ProfileConfig `yaml:"profiles"`
 	Groups    map[string]GroupConfig   `yaml:"groups"`
 }
@@ -28,6 +29,15 @@ type AuditConfig struct {
 	Path    string `yaml:"path,omitempty"`
 }
 
+// SecretsConfig enables optional secret backends. The keychain:// backend is
+// always available (no config, no network). AWS Secrets Manager (awssm://) is
+// gated by AWSSM because constructing it loads AWS config; off by default so a
+// machine without AWS creds doesn't pay that cost or fail at startup.
+type SecretsConfig struct {
+	AWSSM       bool   `yaml:"awssm"`                  // enable the awssm:// backend
+	AWSSMRegion string `yaml:"awssm_region,omitempty"` // optional; defaults to the AWS chain's region
+}
+
 // TelemetryConfig controls opt-in aggregate operational metrics (per-op counts
 // and error tallies — never identifying data). Disabled by default; Path
 // defaults to <state>/siphon/telemetry.json when empty.