Skip to content

README.md

Latest commit

 

History

History
242 lines (182 loc) · 17.9 KB

README.md

File metadata and controls

242 lines (182 loc) · 17.9 KB
Raw

siphon

Sync any database, anywhere.

A single binary that turns the painful, error-prone sprawl of pg_dump → pg_restore shell scripts into a guided, observable workflow — with named profiles, integrity-checked dumps, and server-to-server streaming, across multiple database engines.

CI Go Reference Go Version License: MIT Status

Warning

Pre-1.0 — active development. Postgres, MySQL, and MariaDB backup/restore/sync/verify/inspect work end-to-end today (Phases B + E), and bare siphon opens an interactive multi-panel dashboard (Phase C). The driver layer is hardened with a shared cross-driver test harness, capability gating, and connection retry (Phase D). Advanced transfer is complete (Phase F): incremental backup/restore, cross-engine sync, and CDC continuous replication all work end-to-end. Ops features (Phase G) are underway — the dump catalog can now live in an S3 / S3-compatible bucket (storage: config block); secret backends, retention, and telemetry are still on the roadmap. APIs, flags, and the on-disk dump format may change before 1.0. Track progress via the milestone tags (phase-a … phase-f).

Table of contents

Why siphon

  • One CLI, many databases. Postgres, MySQL, and MariaDB all work today (MySQL and MariaDB share a common _mysqlcommon backend). The driver interface is engine-agnostic, so SQLite, MongoDB, SQL Server, and ClickHouse can follow.
  • Native, not reimplemented. siphon shells out to pg_dump/pg_restore, mysqldump/mysql, and mariadb-dump/mariadb for the actual data movement — you inherit 20+ years of correctness from the official tools, wrapped in a consistent UX.
  • Integrity by default. Every dump is checksummed (SHA-256) and recorded in a sidecar metadata file. siphon verify re-hashes the dump and flags corruption or tampering — and fails with a distinct exit code so CI can catch it.
  • Built for scripts and humans. A Cobra command tree with predictable flags and POSIX exit codes for automation; an interactive Bubble Tea dashboard when you invoke siphon bare.
  • Named profiles + secret refs. Store connection details once; reference secrets as env:VAR today, with OS keychain / Vault / 1Password / AWS Secrets Manager backends on the roadmap. Plaintext passwords never have to live in your config.
  • Streaming sync. siphon sync src dst pipes a backup straight into a restore with no intermediate file on disk.

Project status

Phase What it delivers Status
A — Skeleton Go module, Cobra CLI, TUI placeholder, Driver interface + registry, errs/config/secrets/profile packages, golangci-lint + depguard, cross-platform CI ✅ Complete
B — Postgres walking skeleton backup, restore, sync, verify, inspect, dumps, config, profile working end-to-end against PostgreSQL ✅ Complete
C — TUI dashboard Multi-panel Bubble Tea dashboard (profiles · dumps · jobs) with live job progress, backup/restore modal forms, and snapshot tests ✅ Complete
D — Driver hardening Shared cross-driver test harness (RunDriverSuite), capability-gating helper (RequireCapability), Postgres connection-probe retry, and a docs/DRIVERS.md contributor guide ✅ Complete
E — MySQL + MariaDB Both drivers via a shared _mysqlcommon package (shared Conn, mysqldump/mariadb-dump backup, client-pipe restore), exercised by the Phase D RunDriverSuite harness ✅ Complete
F — Advanced transfer All four advanced-transfer modes work end-to-end: bounded-buffer streaming sync; incremental backup/restore (backup --incremental --base <id> captures a bounded change set via Postgres logical decoding / MySQL-MariaDB binlog, restore replays the base→incremental chain, Postgres orphan-slot sweep); cross-engine sync (sync --cross-engine — typed SchemaInspector introspection → canonical type-mapping, e.g. Postgres → MySQL); and CDC (siphon cdc / sync --continuous — unbounded change streaming with snapshot→stream handoff, resumable, same- and cross-engine). Live DB paths are integration-tested in CI — see docs/INCREMENTAL.md, docs/CROSS_ENGINE.md, docs/CDC.md ✅ Complete
G — Ops features Cloud storage ships first: the dump catalog can live in an S3 / S3-compatible bucket via a pluggable storage.Store backend (local + S3 today; GCS/Azure are a fast-follow), selected by a storage: config block, with SHA-256 integrity preserved end-to-end — see docs/STORAGE.md. Secret backends, profile groups + 2FA, team mode, audit log, retention, and telemetry follow in later G cycles. 🟡 In progress
H — Distribution GoReleaser, Homebrew tap, Scoop bucket, install script, docs site ⏳ Planned

Requirements

  • Go 1.26 or newer — to build from source (the only install method until Phase H).

  • Database client tools — siphon shells out to the native dump/restore tools; it does not embed a client. You only need the tools for the engines you actually use:

    • PostgreSQL profiles need pg_dump, pg_restore, psql.
    • MySQL profiles need mysqldump, mysql.
    • MariaDB profiles need mariadb-dump, mariadb (the renamed binaries shipped by MariaDB 10.5+; older installs that only ship mysqldump/mysql are not yet supported).
    Platform PostgreSQL MySQL / MariaDB
    macOS brew install postgresql@16 brew install mysql-client / brew install mariadb
    Debian/Ubuntu sudo apt install postgresql-client sudo apt install mysql-client mariadb-client
    Fedora/RHEL sudo dnf install postgresql sudo dnf install mysql mariadb
    Windows Install the EDB PostgreSQL package and add its bin/ to PATH Install the MySQL / MariaDB client and add to PATH

  • Docker (optional) — only needed to run the integration test suite (make test-integration).

Install

Pre-1.0: no Homebrew tap or prebuilt binaries yet. Build from source.

git clone https://github.com/nixrajput/siphon.git
cd siphon
make build
./bin/siphon --version

This produces ./bin/siphon. Move it onto your PATH (e.g. sudo install -m 0755 bin/siphon /usr/local/bin/siphon) to call it as siphon.

Quick start

# 1. Register a connection profile (secrets via env: refs, never plaintext in config)
export PROD_DB_PASS='…'
siphon profile add prod \
    --driver postgres \
    --host db.example.com \
    --user app_user \
    --password 'env:PROD_DB_PASS' \
    --database app_prod \
    --sslmode require

siphon profile list                 # show saved profiles

# 2. Inspect the schema (tables, row estimates, on-disk sizes)
siphon inspect prod

# 3. Back it up — written to ~/.local/share/siphon/dumps/ with a checksummed sidecar
siphon backup prod
siphon dumps list                   # newest first

# 4. Verify integrity (re-hashes the dump against the recorded checksum)
siphon verify <dump-id>

# 5. Restore into another profile
siphon restore --profile staging --dump <dump-id> --clean

# 6. Stream prod → staging directly, no intermediate file
siphon sync prod staging

Exit codes follow a POSIX-friendly taxonomy (0 ok, 1 user error, 2 system error, 3 integrity failure, 130 cancelled) so siphon backup prod && upload and CI pipelines behave correctly.

Commands

Command Description
siphon backup [profile] Dump a database to a checksummed file in the catalog
siphon restore [dump-id] Load a dump into a database (--clean to drop-and-recreate)
siphon sync [from] [to] Backup + restore in one streamed pass
siphon verify <dump-id> Re-hash a dump and check it against its recorded checksum
siphon inspect <profile> Show tables, row estimates, and sizes for a profile
siphon dumps list|inspect|prune List, inspect, or prune saved dumps
siphon profile add|list|show|rm Manage named connection profiles
siphon config path|edit Show or edit the config file
siphon schedule Cron-managed recurring backups (Phase G)
siphon tunnel SSH tunnel helper (Phase G)
siphon (bare) Launch the interactive multi-panel dashboard

Run siphon <command> --help for full flags.

Configuration

siphon reads a YAML config from an XDG-compliant path. Find it with siphon config path:

  • Linux: $XDG_CONFIG_HOME/siphon/config.yaml → ~/.config/siphon/config.yaml
  • macOS: ~/.config/siphon/config.yaml
  • Windows: %APPDATA%\siphon\config.yaml

Override the location with SIPHON_CONFIG_HOME. A profile entry looks like:

version: 1
defaults:
  dump_dir: ~/.local/share/siphon/dumps # where backups + sidecars are stored
  jobs: 4
profiles:
  prod:
    driver: postgres
    host: db.example.com
    port: 5432
    user: app_user
    password: env:PROD_DB_PASS # resolved from $PROD_DB_PASS at runtime
    database: app_prod
    sslmode: require

Secret references (env:VAR) are resolved at runtime, so the config file is safe to commit. Plain values are also accepted.

By default the dump catalog lives on local disk at defaults.dump_dir. To store dumps in an S3 / S3-compatible bucket instead, add a storage: block:

storage:
  type: s3                # "local" (default) | "s3"
  bucket: my-siphon-dumps # required for s3
  prefix: prod            # optional key prefix
  region: us-east-1
  endpoint: ""            # optional: custom endpoint for MinIO / R2

Credentials are resolved from the standard AWS chain (env vars, ~/.aws, instance role) — never stored in the config file. See docs/STORAGE.md for details.

Architecture

siphon is a strictly layered Go application; imports flow downward only, enforced at lint time by golangci-lint's depguard (an upward import fails CI):

  cmd/siphon                      entry point (one-line main)
        │
  internal/cli   internal/tui     presentation (Cobra · Bubble Tea) — siblings
        └──────┬───────┘
        internal/app              application verbs (backup, restore, sync, …)
               │
   internal/driver/<engine>       database adapters (postgres; mysql/mariadb in v1.0)
               │
  config · secrets · profile · dumps · jobs · errs   domain + support packages
  • Drivers are compile-time Go packages that shell out to native tools for data movement and use a client library (pgx) for fast schema reads.
  • Dumps live on disk as <id>.dump with a <id>.meta.json sidecar (profile, driver, size, SHA-256 checksum, timestamp).
  • Jobs run long operations on a goroutine and stream progress Events that the CLI heartbeat (and, later, the TUI) renders.

Development

make help              # list all targets
make build             # produce ./bin/siphon
make run               # go run ./cmd/siphon (launches the TUI on bare invoke)
make test              # unit tests
make test-integration  # integration tests against a real Postgres (needs Docker)
make lint              # golangci-lint, incl. depguard layer enforcement
make tidy              # go mod tidy
make clean             # remove bin/ and dist/

Tests run race-clean (go test -race ./...). The integration suite is gated behind the integration build tag and spins up postgres:16-alpine via testcontainers.

See CONTRIBUTING.md for the full contributor guide and SECURITY.md for reporting vulnerabilities.

Roadmap

v1.0 targets a single mega-release covering four pillars:

  1. Foundation — the CLI, TUI, profiles, config, and dump catalog (Phases A–C).
  2. Drivers — Postgres, MySQL, and MariaDB (Phases D–E).
  3. Advanced transfer — incremental backups, native server-to-server streaming, cross-engine sync, and CDC (Phase F).
  4. Ops — cloud storage (S3/GCS/Azure), multi-backend secrets, profile groups, team mode, audit log, retention policies, and opt-in telemetry (Phase G).

Distribution (Homebrew, Scoop, install script, signed release binaries) lands in Phase H alongside the 1.0 tag.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md, keep changes within the layered architecture (depguard will tell you if you stray), and make sure make test and make lint pass before opening a PR.

Adding a new database engine? See docs/DRIVERS.md for the driver contributor guide.

Concept docs: docs/INCREMENTAL.md (incremental backup + restore), docs/CROSS_ENGINE.md (cross-engine sync + the type-map matrix), and docs/CDC.md (continuous CDC sync, same- and cross-engine). All three work end-to-end; live DB behavior is integration-tested in CI.

License

MIT © Nikhil Rajput