Skip to content

CHANGELOG.md

Latest commit

 

History

History
48 lines (40 loc) · 11.2 KB

CHANGELOG.md

File metadata and controls

48 lines (40 loc) · 11.2 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

  • Phase G (ops) — operational suite (audit, 2FA gating, telemetry, schedule, tunnel):

    • Audit log (internal/audit): an append-only JSONL trail of destructive ops (backup/restore/sync/prune) recording who/what/when/outcome, via a single guardedOp interception seam wired into the verbs. Off by default; enable with the audit: config block. The same seam is reused by 2FA gating and telemetry (no per-feature re-wrapping).
    • 2FA / group gating (internal/twofactor): a profile's group can require a typed-name confirmation (confirm_destructive) and/or a TOTP code (require_2fa) before a destructive op runs. Stdlib RFC 6238 TOTP (HMAC-SHA1, 30s, 6 digits, ±1-step skew), no new dependency; the group's totp_secret is a secret-ref. require_2fa with no resolvable secret fails closed.
    • Telemetry (internal/telemetry): opt-in aggregate per-op counts and error tallies, flushed as JSON. Records the op name and outcome only — never profile, actor, target, or data. Off by default (telemetry: config block); composed onto the audit seam via audit.Multi.
    • siphon schedule: manage cron-scheduled recurring backups. siphon maintains a delimited, siphon-owned block in the user's crontab that invokes siphon backup <profile> — the host cron runs the jobs (no daemon). schedule add <profile> --cron, list, remove.
    • siphon tunnel <profile>: open a foreground ssh -L local-forward to a profile's database through a configured tunnel.bastion, using the system ssh client; held open until Ctrl-C.

  • Phase G (ops) — retention & lifecycle (second G cycle):

    • Chain-aware retention engine in internal/dumps (GroupChains + Plan, pure and DB/clock-free): the catalog is grouped into restorable chains (a base plus its incrementals), and retention keeps or prunes each chain as a unit, so an incremental is never orphaned from its base. A chain's age is its newest member, so an actively-appended chain is never pruned mid-life.
    • Three composable policy rules with union semantics (a chain is kept if it satisfies any active rule, so adding a rule only ever protects more): keep_last: N, max_age: <duration>, and grandfather-father-son gfs: {daily, weekly, monthly}. An all-zero/omitted policy keeps everything — prune is a no-op unless a rule is explicitly configured.
    • siphon prune rewired to the engine: --profile scoping, --keep-last / --max-age / --gfs-daily|weekly|monthly flags, dry-run by default with --apply to delete. Deletion is leaf-inward (incrementals before base), so an interrupted prune leaves at worst a complete shorter chain. Per-dump failures are collected and reported; the run exits non-zero on any failure but still prunes the rest. Works over any storage backend via the existing Store.Delete.
    • New retention: config block (defaults + per-profile override that replaces the default wholesale; keep_last, max_age, gfs) with fail-fast validation. Resolution precedence: CLI flags > profile > defaults > keep-everything. See docs/RETENTION.md.

  • Phase G (ops) — cloud storage backends (first G cycle):

    • New internal/storage leaf defining a backend-neutral Store interface (Put/Get/Delete/List/Stat, all context-aware) with two implementations: local (the previous filesystem layout, now behind the interface — temp-write+rename for atomic publish, keys map verbatim to file names so existing catalogs need no migration) and s3 (S3 and S3-compatible services like MinIO/R2 via AWS SDK v2; streaming transfer-manager upload that is atomic on completion, path-style for custom endpoints).
    • internal/dumps.Catalog refactored to hold a Store and address dumps by key (<id>.dump, <id>.meta.json) instead of filesystem paths; Path/MetaPath/Root removed. Catalog methods (PutDump/OpenDump/ReadMeta/WriteMeta/List/Delete/ResolveChain/PruneDryRun) are now context-aware. backup, restore, verify, and dumps stream through the store; SHA-256 integrity is computed app-side over envelope ++ body, so a dump verifies identically whether it lives locally or in S3.
    • New storage: config block (type: local|s3, bucket, prefix, region, endpoint) with fail-fast validation; S3 credentials resolve from the standard AWS chain (never stored in config). See docs/STORAGE.md.
    • A shared RunStoreSuite contract test (mirroring Phase D's RunDriverSuite) runs against every backend: the local backend runs it as a unit test, and the S3 backend runs it against MinIO via testcontainers under the integration tag, so the live object-store path executes in CI. GCS/Azure backends are a fast-follow.

  • Phase A skeleton: Go module, Cobra CLI with placeholder subcommands, Bubble Tea TUI placeholder, Driver interface + registry, errs package, config stub, Makefile, golangci-lint with depguard, CI workflow.

  • Phase B Postgres walking skeleton: backup, restore, sync, verify, inspect, dumps, and profile working end-to-end against PostgreSQL (shelling out to pg_dump/pg_restore, pgx for inspect), SHA-256 dump checksums with sidecar metadata, named profiles with env: secret refs, and POSIX exit-code taxonomy.

  • Phase C interactive TUI dashboard: multi-panel Bubble Tea UI (profiles · dumps · jobs) with live job-progress subscription, backup/restore modal forms, an error overlay, and golden snapshot tests.

  • Phase D driver-layer hardening:

    • Shared cross-driver test harness RunDriverSuite under internal/driver/_testing/, giving every driver four contract tests for free (connect/inspect, backup→restore round-trip, cancel propagation, bad-credentials sentinel). The Postgres integration test now runs on it.
    • Capability gating helper: app.RequireCapability resolves a profile to its driver and rejects unsupported affordances with a CodeUser error. The helper and the Capabilities flags are in place; verbs are wired to it only where the gated feature is implemented. Streaming (NativeStream) and parallel backup (Parallel) are deferred to Phase F, so sync --stream / backup --jobs are not yet gated — the wiring lands with those features.
    • Connection-probe retry on Postgres Connect (3 attempts, exponential backoff) per spec §4.3; the generic Retry helper moved to internal/jobs to keep the driver layer free of an app-layer import.
    • docs/DRIVERS.md contributor guide documenting the driver contract, registration, the test harness, capability flags, and error mapping.

  • Phase E MySQL + MariaDB drivers:

    • Shared internal/driver/_mysqlcommon package: DSN builder, mysqldump/mariadb-dump arg builder, fork detection, and a shared Conn implementing the full driver.Conn contract (Inspect via information_schema, Backup by shelling to the dump tool, Restore by piping SQL into the client, sha256 Verify, bounded connect-probe retry). Backup/Restore pass the password via MYSQL_PWD and stream through stdout/stdin.
    • mysql and mariadb drivers as thin wrappers that inject the fork-specific binary names and declare capabilities honestly (Parallel: false — the dump tools are single-threaded). Registered via side-effect import in internal/app/drivers.go.
    • Integration suites for both engines run on the Phase D RunDriverSuite harness via testcontainers (mysql:8.0, mariadb:11).
    • CI installs the Postgres/MySQL/MariaDB client tools and runs the full integration suite on the Ubuntu runner (where Docker is available); unit tests continue to run on Linux/macOS/Windows.

  • Phase F advanced transfer — all four modes work end-to-end (incremental, cross-engine, CDC, bounded streaming); live DB paths are integration-tested in CI. See docs/INCREMENTAL.md, docs/CROSS_ENGINE.md, docs/CDC.md.

    • Working today: every dump is prefixed with a 4 KB SIPH JSON envelope (type, base/parent IDs, WAL/binlog positions, checksum); restore resolves the base→incremental chain and applies it in order, with --up-to <id> to stop early (cycle + broken-chain detection). Sync now streams through a bounded jobs.Stream (default 64×1 MB) instead of io.Pipe, exposing a FillPercent backpressure metric and propagating backup failures to the restore side via CloseErr (no truncated dump committed as clean).
    • Incremental backup wired end-to-end (backup --incremental --base <id>): captures a bounded change set since the base dump's recorded end position, serialized as engine-neutral JSONL CanonicalChange records. Postgres bounds the pgoutput logical-decoding stream by pg_current_wal_lsn() captured at backup time; MySQL/MariaDB bound the binlog-tool decode by the current binlog file+offset. The incremental dump's envelope carries this capture's end position so the next incremental resumes exactly there. restore replays incremental links via ApplyChange (base links still restore natively). Postgres adds an orphan replication-slot sweep (SweepOrphanSlots) run before each capture — drops inactive siphon_* physical slots while preserving the persistent logical resume slot. Incremental capability is now true for all three drivers. Live-server behavior is integration-tested in CI (wal_level=logical); compile-checked locally.
    • CDC continuous sync wired end-to-end (siphon cdc <from> <to> / sync --continuous): unbounded ChangeStreamer.StreamChanges → CanonicalTransfer.ApplyChange on the target, with an initial schema+data snapshot→stream handoff on first run (consistent start position captured before the snapshot) and resume-from-saved-position on restart (stable per-(source,target) job ID). Works same-engine and cross-engine (changes are engine-neutral CanonicalChanges) and honors --table in both the initial snapshot and the streamed changes. Persists the streamer's delivered position on exit (no ahead-of-stream periodic checkpoint, which could resume past un-applied changes after a crash); at-least-once delivery is safe because ApplyChange is idempotent (INSERT upsert, UPDATE/DELETE by PK). ctx cancel is the normal clean stop. CDC capability is now true for all three drivers. Same-engine, cross-engine, and resume paths are integration-tested in CI.
    • Cross-engine sync wired end-to-end (sync --cross-engine, e.g. Postgres → MySQL): typed schema introspection via driver.SchemaInspector (information_schema/pg_catalog, incl. primary keys) builds a CanonicalSchema; the source emits canonical JSONL rows and the target re-creates tables (with primary keys) and inserts them, translating types via MapToNative — all with per-engine identifier quoting and parameterized values. Honors --table. Unmapped source column types now fail fast with an explicit error instead of silently coercing to text (which would corrupt binary/BLOB families). CrossEngineSource/CrossEngineTarget capabilities are now true for all three drivers. Scope is data + table structure + primary keys (secondary indexes, FKs, triggers, views, functions not translated). Integration-tested in CI (Postgres → MySQL); compile-checked locally.