Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,27 @@ jobs:
cache: true
- name: go vet
run: go vet ./...
# `./...` skips underscore-prefixed dirs, so name _*-prefixed packages
# explicitly (e.g. internal/driver/_mysqlcommon) to run their unit tests.
- name: go test
run: go test ./...
run: go test ./... ./internal/driver/_*/
# Integration tests use testcontainers, which needs a Docker daemon.
# Only ubuntu-latest runners have Docker available by default, so the
# DB client install and integration suite are scoped to Ubuntu.
# macOS/Windows run unit tests only (the integration-tagged files that
# shell out to the client binaries are tagged out of the unit build).
- name: Install DB client tools (integration, Ubuntu only)
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt-get update
# postgresql-client -> pg_dump/pg_restore
# mysql-client -> mysqldump/mysql
# mariadb-client -> mariadb-dump/mariadb
# NOTE: mysql-client and mariadb-client can conflict on some Ubuntu
# releases (both provide the mysql* binaries via alternatives). If apt
# reports a conflict in CI, split them or drop mysql-client (mariadb-client
# provides mysql-compatible binaries) as a one-line follow-up.
sudo apt-get install -y postgresql-client mysql-client mariadb-client
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
- name: Integration tests (Ubuntu only — needs Docker)
if: matrix.os == 'ubuntu-latest'
run: go test -tags=integration ./... ./internal/driver/_*/
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Capability gating helper: `app.RequireCapability` resolves a profile to its driver and rejects unsupported affordances with a `CodeUser` error. The helper and the `Capabilities` flags are in place; verbs are wired to it only where the gated feature is implemented. Streaming (`NativeStream`) and parallel backup (`Parallel`) are deferred to Phase F, so `sync --stream` / `backup --jobs` are not yet gated — the wiring lands with those features.
- Connection-probe retry on Postgres `Connect` (3 attempts, exponential backoff) per spec §4.3; the generic `Retry` helper moved to `internal/jobs` to keep the driver layer free of an app-layer import.
- `docs/DRIVERS.md` contributor guide documenting the driver contract, registration, the test harness, capability flags, and error mapping.
- Phase E MySQL + MariaDB drivers:
- Shared `internal/driver/_mysqlcommon` package: DSN builder, `mysqldump`/`mariadb-dump` arg builder, fork detection, and a shared `Conn` implementing the full `driver.Conn` contract (Inspect via `information_schema`, Backup by shelling to the dump tool, Restore by piping SQL into the client, sha256 Verify, bounded connect-probe retry). Backup/Restore pass the password via `MYSQL_PWD` and stream through stdout/stdin.
- `mysql` and `mariadb` drivers as thin wrappers that inject the fork-specific binary names and declare capabilities honestly (`Parallel: false` — the dump tools are single-threaded). Registered via side-effect import in `internal/app/drivers.go`.
- Integration suites for both engines run on the Phase D `RunDriverSuite` harness via testcontainers (`mysql:8.0`, `mariadb:11`).
- CI installs the Postgres/MySQL/MariaDB client tools and runs the full integration suite on the Ubuntu runner (where Docker is available); unit tests continue to run on Linux/macOS/Windows.
11 changes: 8 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,19 @@ help: ## Show this help
lint: ## Run golangci-lint
golangci-lint run

# NOTE: `go test ./...` skips directories whose names start with `_` (Go tool
# convention), so underscore-prefixed packages with real tests (e.g.
# internal/driver/_mysqlcommon) are silently excluded. Name them explicitly via
# the `_*` glob so their unit tests actually run. The glob auto-picks up any
# future _*-prefixed package under internal/driver/.
test: ## Run unit tests
go test ./...
go test ./... ./internal/driver/_*/

test-verbose: ## Run unit tests verbosely
go test -v ./...
go test -v ./... ./internal/driver/_*/

test-integration: ## Run integration tests (build tag: integration)
go test -tags=integration ./...
go test -tags=integration ./... ./internal/driver/_*/

build: ## Build the siphon binary into ./bin
@mkdir -p bin
Expand Down
25 changes: 14 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_
---

> [!WARNING]
> **Pre-1.0 — active development.** Postgres backup/restore/sync/verify/inspect work end-to-end today (Phase B), and bare `siphon` opens an interactive multi-panel dashboard (Phase C). The driver layer is hardened with a shared cross-driver test harness, capability gating, and connection retry (Phase D), so MySQL/MariaDB can land mechanically next. MySQL/MariaDB, incremental backups, and ops features are on the [roadmap](#roadmap). APIs, flags, and the on-disk dump format may change before 1.0. Track progress via the milestone tags (`phase-a`, `phase-b`, `phase-c`, `phase-d`, …).
> **Pre-1.0 — active development.** Postgres, MySQL, and MariaDB backup/restore/sync/verify/inspect work end-to-end today (Phases B + E), and bare `siphon` opens an interactive multi-panel dashboard (Phase C). The driver layer is hardened with a shared cross-driver test harness, capability gating, and connection retry (Phase D). Incremental backups, cross-engine sync, and ops features are on the [roadmap](#roadmap). APIs, flags, and the on-disk dump format may change before 1.0. Track progress via the milestone tags (`phase-a`, `phase-b`, `phase-c`, `phase-d`, `phase-e`, …).

## Table of contents

Expand All @@ -38,7 +38,7 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_

## Why siphon

- **One CLI, many databases.** Postgres works today; MySQL and MariaDB land in v1.0 (sharing a common backend). The driver interface is engine-agnostic, so SQLite, MongoDB, SQL Server, and ClickHouse can follow.
- **One CLI, many databases.** Postgres, MySQL, and MariaDB all work today (MySQL and MariaDB share a common `_mysqlcommon` backend). The driver interface is engine-agnostic, so SQLite, MongoDB, SQL Server, and ClickHouse can follow.
- **Native, not reimplemented.** siphon shells out to `pg_dump`/`pg_restore` (and `mysqldump`/`mariadb-dump` in v1.0) for the actual data movement — you inherit 20+ years of correctness from the official tools, wrapped in a consistent UX.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
- **Integrity by default.** Every dump is checksummed (SHA-256) and recorded in a sidecar metadata file. `siphon verify` re-hashes the dump and flags corruption or tampering — and fails with a distinct exit code so CI can catch it.
- **Built for scripts and humans.** A Cobra command tree with predictable flags and POSIX exit codes for automation; an interactive Bubble Tea dashboard when you invoke `siphon` bare.
Expand All @@ -53,22 +53,25 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_
| **B** — Postgres walking skeleton | `backup`, `restore`, `sync`, `verify`, `inspect`, `dumps`, `config`, `profile` working end-to-end against PostgreSQL | ✅ Complete |
| **C** — TUI dashboard | Multi-panel Bubble Tea dashboard (profiles · dumps · jobs) with live job progress, backup/restore modal forms, and snapshot tests | ✅ Complete |
| **D** — Driver hardening | Shared cross-driver test harness (`RunDriverSuite`), capability-gating helper (`RequireCapability`), Postgres connection-probe retry, and a `docs/DRIVERS.md` contributor guide | ✅ Complete |
| **E** — MySQL + MariaDB | Both drivers via a shared `_mysqlcommon` package | ⏳ Planned |
| **E** — MySQL + MariaDB | Both drivers via a shared `_mysqlcommon` package (shared `Conn`, `mysqldump`/`mariadb-dump` backup, client-pipe restore), exercised by the Phase D `RunDriverSuite` harness | ✅ Complete |
| **F** — Advanced transfer | Incremental backups, bounded-buffer streaming, cross-engine sync, CDC | ⏳ Planned |
| **G** — Ops features | Cloud storage, secret backends, profile groups + 2FA, team mode, audit log, retention, telemetry | ⏳ Planned |
| **H** — Distribution | GoReleaser, Homebrew tap, Scoop bucket, install script, docs site | ⏳ Planned |

## Requirements

- **[Go](https://go.dev/dl/) 1.26 or newer** — to build from source (the only install method until Phase H).
- **PostgreSQL client tools** — `pg_dump`, `pg_restore`, and `psql` must be on your `PATH`. siphon shells out to them; it does not embed a Postgres client.

| Platform | Install |
| ------------- | --------------------------------------------------------------------------------------------------------------- |
| macOS | `brew install postgresql@16` |
| Debian/Ubuntu | `sudo apt install postgresql-client` |
| Fedora/RHEL | `sudo dnf install postgresql` |
| Windows | Install the [EDB PostgreSQL](https://www.postgresql.org/download/windows/) package and add its `bin/` to `PATH` |
- **Database client tools** — siphon shells out to the native dump/restore tools; it does not embed a client. You only need the tools for the engines you actually use:
- **PostgreSQL** profiles need `pg_dump`, `pg_restore`, `psql`.
- **MySQL** profiles need `mysqldump`, `mysql`.
- **MariaDB** profiles need `mariadb-dump`, `mariadb` (the renamed binaries shipped by MariaDB 10.5+; older installs that only ship `mysqldump`/`mysql` are not yet supported).

| Platform | PostgreSQL | MySQL / MariaDB |
| ------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| macOS | `brew install postgresql@16` | `brew install mysql-client` / `brew install mariadb` |
| Debian/Ubuntu | `sudo apt install postgresql-client` | `sudo apt install mysql-client mariadb-client` |
| Fedora/RHEL | `sudo dnf install postgresql` | `sudo dnf install mysql mariadb` |
| Windows | Install the [EDB PostgreSQL](https://www.postgresql.org/download/windows/) package and add its `bin/` to `PATH` | Install the MySQL / MariaDB client and add to `PATH` |

- **Docker** _(optional)_ — only needed to run the integration test suite (`make test-integration`).

Expand Down
6 changes: 6 additions & 0 deletions docs/DRIVERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,12 @@ The `Fixtures` struct (in `internal/driver/_testing/fixtures.go`) you populate:

The worked example is `internal/driver/postgres/integration_test.go` — copy its
shape. The suite runs behind the `integration` build tag (`make test-integration`).
`internal/driver/mysql/` and `internal/driver/mariadb/` are a second worked example:
two engines that share almost everything live in `internal/driver/_mysqlcommon/`
(a shared `Conn` plus arg/DSN builders), leaving each driver a ~30-line wrapper
that only injects the fork-specific binary names and capabilities. If you're
adding an engine that's a fork of an existing one, follow that shared-helper
pattern rather than copying a whole driver.

## Error mapping

Expand Down
4 changes: 4 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,20 @@ require (
github.com/charmbracelet/huh v1.0.0
github.com/charmbracelet/lipgloss v1.1.0
github.com/charmbracelet/x/exp/teatest v0.0.0-20260527151214-009e6338d40d
github.com/go-sql-driver/mysql v1.10.0
github.com/jackc/pgx/v5 v5.9.2
github.com/muesli/termenv v0.16.0
github.com/oklog/ulid/v2 v2.1.1
github.com/spf13/cobra v1.10.2
github.com/testcontainers/testcontainers-go/modules/mariadb v0.42.0
github.com/testcontainers/testcontainers-go/modules/mysql v0.42.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
gopkg.in/yaml.v3 v3.0.1
)

require (
dario.cat/mergo v1.0.2 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
Expand Down
8 changes: 8 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
Expand Down Expand Up @@ -95,6 +97,8 @@ github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw=
github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
Expand Down Expand Up @@ -195,6 +199,10 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/testcontainers/testcontainers-go v0.42.0 h1:He3IhTzTZOygSXLJPMX7n44XtK+qhjat1nI9cneBbUY=
github.com/testcontainers/testcontainers-go v0.42.0/go.mod h1:vZjdY1YmUA1qEForxOIOazfsrdyORJAbhi0bp8plN30=
github.com/testcontainers/testcontainers-go/modules/mariadb v0.42.0 h1:ZfWUJSIDnbNgoLAXMV1fc7lqcxBIX3zdnhwjaVUo7N0=
github.com/testcontainers/testcontainers-go/modules/mariadb v0.42.0/go.mod h1:0kV+yHee7zAgp0yccydxjNnHvlC1EOavTLCeg/lnRBY=
github.com/testcontainers/testcontainers-go/modules/mysql v0.42.0 h1:Yhv1k7vDpyzZePntg5R5Oj4ZMCyWpAfpJeRu1ROsgiU=
github.com/testcontainers/testcontainers-go/modules/mysql v0.42.0/go.mod h1:Z7SCTuiZlghAdRjkv3Ir0iXJKC2T2avbtxLR0DRe+ng=
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0 h1:GCbb1ndrF7OTDiIvxXyItaDab4qkzTFJ48LKFdM7EIo=
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0/go.mod h1:IRPBaI8jXdrNfD0e4Zm7Fbcgaz5shKxOQv4axiL09xs=
github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA=
Expand Down
2 changes: 2 additions & 0 deletions internal/app/drivers.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ package app

import (
"github.com/nixrajput/siphon/internal/driver"
_ "github.com/nixrajput/siphon/internal/driver/mariadb" // register the mariadb driver
_ "github.com/nixrajput/siphon/internal/driver/mysql" // register the mysql driver
_ "github.com/nixrajput/siphon/internal/driver/postgres" // register the postgres driver
)

Expand Down
49 changes: 49 additions & 0 deletions internal/driver/_mysqlcommon/args.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
package mysqlcommon

import (
"strconv"

"github.com/nixrajput/siphon/internal/driver"
)

// BuildDumpArgs assembles the mysqldump argument vector for a backup. The
// binary name itself is supplied by the caller (mysqldump vs mariadb-dump).
func BuildDumpArgs(p driver.Profile, opt driver.BackupOpts) []string {
args := []string{
"-h", p.Host,
"-P", strconv.Itoa(p.Port),
"-u", p.User,
"--single-transaction",
"--routines",
"--triggers",
"--events",
"--no-tablespaces",
"--skip-comments",
p.Database,
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
if opt.SchemaOnly {
args = append(args, "--no-data")
}
if opt.DataOnly {
args = append(args, "--no-create-info")
}
args = append(args, opt.IncludeTables...)
for _, t := range opt.ExcludeTables {
args = append(args, "--ignore-table="+p.Database+"."+t)
}
return args
}

// BuildRestoreArgs assembles the mysql client argument vector for a restore.
// The dump file is authoritative for shape; the restore client just pipes it
// in. Clean is a no-op here because mysqldump output already emits
// DROP TABLE IF EXISTS / CREATE TABLE, making the restore idempotent.
func BuildRestoreArgs(p driver.Profile, _ driver.RestoreOpts) []string {
return []string{
"-h", p.Host,
"-P", strconv.Itoa(p.Port),
"-u", p.User,
"--default-character-set=utf8mb4",
p.Database,
}
}
Loading
Loading