Skip to content

Commit 95abeb4

Browse files
nixrajputnixrajput
authored
Feat/implement phase f (#5)
1 parent 4135acc commit 95abeb4

32 files changed

Lines changed: 2990 additions & 59 deletions

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2222
- `mysql` and `mariadb` drivers as thin wrappers that inject the fork-specific binary names and declare capabilities honestly (`Parallel: false` — the dump tools are single-threaded). Registered via side-effect import in `internal/app/drivers.go`.
2323
- Integration suites for both engines run on the Phase D `RunDriverSuite` harness via testcontainers (`mysql:8.0`, `mariadb:11`).
2424
- CI installs the Postgres/MySQL/MariaDB client tools and runs the full integration suite on the Ubuntu runner (where Docker is available); unit tests continue to run on Linux/macOS/Windows.
25+
- Phase F advanced-transfer machinery (some CLI paths gated pending follow-up wiring — see the `docs/INCREMENTAL.md`, `docs/CROSS_ENGINE.md`, `docs/CDC.md` Status sections):
26+
- **Working today:** every dump is prefixed with a 4 KB `SIPH` JSON envelope (type, base/parent IDs, WAL/binlog positions, checksum); `restore` resolves the base→incremental chain and applies it in order, with `--up-to <id>` to stop early (cycle + broken-chain detection). Sync now streams through a bounded `jobs.Stream` (default 64×1 MB) instead of `io.Pipe`, exposing a `FillPercent` backpressure metric and propagating backup failures to the restore side via `CloseErr` (no truncated dump committed as clean).
27+
- **Machinery in place, CLI gated (follow-up):** incremental backup capture (Postgres WAL replication slot + LSN; MySQL/MariaDB binlog position) — `backup --incremental` returns a clear "not yet wired" error; cross-engine type-mapping (canonical schema + JSONL emit/consume with per-engine identifier quoting and placeholders) — `sync --cross-engine` is capability-gated off until typed schema introspection exists; CDC continuous mode (state-file persistence + resume + capability gating) ships as a polling scaffold, not real logical-replication streaming.
28+
- Known follow-up gates before incremental backup ships to users: Postgres orphan-replication-slot cleanup, validation of the `pg_receivewal`/`mysqlbinlog` streaming invocations against live servers (compile-checked only here).

‎README.md‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -47,16 +47,16 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_
4747

4848
## Project status
4949

50-
| Phase | What it delivers | Status |
51-
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
52-
| **A** — Skeleton | Go module, Cobra CLI, TUI placeholder, `Driver` interface + registry, `errs`/`config`/`secrets`/`profile` packages, golangci-lint + depguard, cross-platform CI | ✅ Complete |
53-
| **B** — Postgres walking skeleton | `backup`, `restore`, `sync`, `verify`, `inspect`, `dumps`, `config`, `profile` working end-to-end against PostgreSQL | ✅ Complete |
54-
| **C** — TUI dashboard | Multi-panel Bubble Tea dashboard (profiles · dumps · jobs) with live job progress, backup/restore modal forms, and snapshot tests | ✅ Complete |
55-
| **D** — Driver hardening | Shared cross-driver test harness (`RunDriverSuite`), capability-gating helper (`RequireCapability`), Postgres connection-probe retry, and a `docs/DRIVERS.md` contributor guide | ✅ Complete |
56-
| **E** — MySQL + MariaDB | Both drivers via a shared `_mysqlcommon` package (shared `Conn`, `mysqldump`/`mariadb-dump` backup, client-pipe restore), exercised by the Phase D `RunDriverSuite` harness | ✅ Complete |
57-
| **F** — Advanced transfer | Incremental backups, bounded-buffer streaming, cross-engine sync, CDC | ⏳ Planned |
58-
| **G** — Ops features | Cloud storage, secret backends, profile groups + 2FA, team mode, audit log, retention, telemetry | ⏳ Planned |
59-
| **H** — Distribution | GoReleaser, Homebrew tap, Scoop bucket, install script, docs site | ⏳ Planned |
50+
| Phase | What it delivers | Status |
51+
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
52+
| **A** — Skeleton | Go module, Cobra CLI, TUI placeholder, `Driver` interface + registry, `errs`/`config`/`secrets`/`profile` packages, golangci-lint + depguard, cross-platform CI | ✅ Complete |
53+
| **B** — Postgres walking skeleton | `backup`, `restore`, `sync`, `verify`, `inspect`, `dumps`, `config`, `profile` working end-to-end against PostgreSQL | ✅ Complete |
54+
| **C** — TUI dashboard | Multi-panel Bubble Tea dashboard (profiles · dumps · jobs) with live job progress, backup/restore modal forms, and snapshot tests | ✅ Complete |
55+
| **D** — Driver hardening | Shared cross-driver test harness (`RunDriverSuite`), capability-gating helper (`RequireCapability`), Postgres connection-probe retry, and a `docs/DRIVERS.md` contributor guide | ✅ Complete |
56+
| **E** — MySQL + MariaDB | Both drivers via a shared `_mysqlcommon` package (shared `Conn`, `mysqldump`/`mariadb-dump` backup, client-pipe restore), exercised by the Phase D `RunDriverSuite` harness | ✅ Complete |
57+
| **F** — Advanced transfer | Bounded-buffer streaming sync + chain-walking incremental restore work today; the dump envelope, incremental WAL/binlog capture, cross-engine type-mapping, and CDC state machinery are in place but their CLI paths (`backup --incremental`, `sync --cross-engine`, CDC) are gated pending follow-up wiring — see [docs/INCREMENTAL.md](docs/INCREMENTAL.md), [docs/CROSS_ENGINE.md](docs/CROSS_ENGINE.md), [docs/CDC.md](docs/CDC.md) | 🟡 Partial |
58+
| **G** — Ops features | Cloud storage, secret backends, profile groups + 2FA, team mode, audit log, retention, telemetry | ⏳ Planned |
59+
| **H** — Distribution | GoReleaser, Homebrew tap, Scoop bucket, install script, docs site | ⏳ Planned |
6060

6161
## Requirements
6262

@@ -66,12 +66,12 @@ A single binary that turns the painful, error-prone sprawl of `pg_dump` → `pg_
6666
- **MySQL** profiles need `mysqldump`, `mysql`.
6767
- **MariaDB** profiles need `mariadb-dump`, `mariadb` (the renamed binaries shipped by MariaDB 10.5+; older installs that only ship `mysqldump`/`mysql` are not yet supported).
6868

69-
| Platform | PostgreSQL | MySQL / MariaDB |
70-
| ------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
71-
| macOS | `brew install postgresql@16` | `brew install mysql-client` / `brew install mariadb` |
72-
| Debian/Ubuntu | `sudo apt install postgresql-client` | `sudo apt install mysql-client mariadb-client` |
73-
| Fedora/RHEL | `sudo dnf install postgresql` | `sudo dnf install mysql mariadb` |
74-
| Windows | Install the [EDB PostgreSQL](https://www.postgresql.org/download/windows/) package and add its `bin/` to `PATH` | Install the MySQL / MariaDB client and add to `PATH` |
69+
| Platform | PostgreSQL | MySQL / MariaDB |
70+
| ------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
71+
| macOS | `brew install postgresql@16` | `brew install mysql-client` / `brew install mariadb` |
72+
| Debian/Ubuntu | `sudo apt install postgresql-client` | `sudo apt install mysql-client mariadb-client` |
73+
| Fedora/RHEL | `sudo dnf install postgresql` | `sudo dnf install mysql mariadb` |
74+
| Windows | Install the [EDB PostgreSQL](https://www.postgresql.org/download/windows/) package and add its `bin/` to `PATH` | Install the MySQL / MariaDB client and add to `PATH` |
7575

7676
- **Docker** _(optional)_ — only needed to run the integration test suite (`make test-integration`).
7777

@@ -222,6 +222,8 @@ Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md), keep
222222

223223
Adding a new database engine? See [docs/DRIVERS.md](docs/DRIVERS.md) for the driver contributor guide.
224224

225+
Concept docs (honest as-built status — several paths are documented follow-ups): [docs/INCREMENTAL.md](docs/INCREMENTAL.md) (incremental restore works; the backup path is a follow-up), [docs/CROSS_ENGINE.md](docs/CROSS_ENGINE.md) (type-map matrix; `--cross-engine` is gated off pending typed introspection), and [docs/CDC.md](docs/CDC.md) (scaffold only; not yet runnable).
226+
225227
## License
226228

227229
[MIT](LICENSE) © [Nikhil Rajput](https://github.com/nixrajput)

‎docs/CDC.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Change data capture (CDC)
2+
3+
CDC ("continuous sync" / follow mode) keeps a target database in step with a
4+
source by streaming changes as they happen, rather than re-running a full
5+
backup/restore. In siphon this is **scaffolding today** — the resume-state
6+
plumbing and capability gating are in place, but continuous streaming is a
7+
future deliverable and does not run.
8+
9+
## Table of contents
10+
11+
- [What exists](#what-exists)
12+
- [The CLI surface](#the-cli-surface)
13+
- [Status](#status)
14+
- [Resume state](#resume-state)
15+
- [Limitations](#limitations)
16+
17+
## What exists
18+
19+
`internal/app/cdc.go` holds:
20+
21+
- **`CDCState`** — the persisted resume record (`job_id`, source/target
22+
profiles, `last_applied_lsn` for Postgres, `last_binlog_file` +
23+
`last_binlog_pos` for MySQL/MariaDB, `updated_at`).
24+
- **`saveCDCState` / `loadCDCState`** — JSON state files under a per-user
25+
directory resolved from `SIPHON_STATE_HOME`, then `XDG_STATE_HOME`, then
26+
`$HOME/.local/state/siphon/cdc`.
27+
- **`RunCDC`** — a capability-gated entry point. It resolves the source and
28+
target profiles, then requires both drivers to advertise `CapCDC`. Its body is
29+
a **polling scaffold**: a 10-second ticker that persists state each tick and
30+
resumes from a prior run's state on restart. It is **not** real change
31+
streaming.
32+
33+
`CDCState` save/load round-trip, the state-dir resolution, and the
34+
no-capability rejection are unit-tested (`internal/app/cdc_test.go`).
35+
36+
## The CLI surface
37+
38+
There is **no `siphon cdc` subcommand wired today.** `RunCDC` exists as an
39+
application function but is not registered as a Cobra command
40+
(`internal/cli/root.go` wires `backup`, `restore`, `sync`, `verify`, `inspect`,
41+
`profile`, `dumps`, `config`, `schedule`, `tunnel` — no `cdc`).
42+
43+
`sync --continuous` exposes the flag but does not run CDC; it returns a clear
44+
error pointing at the (not-yet-wired) `siphon cdc` follow-up:
45+
46+
```bash
47+
siphon sync --from pg-prod --to pg-replica --continuous
48+
# Error: continuous CDC sync is not wired here; use `siphon cdc` (Phase F Task 10)
49+
```
50+
51+
## Status
52+
53+
| Capability | Status |
54+
| --- | --- |
55+
| `CDCState` persistence (save/load) | ✅ Works (unit-tested) |
56+
| State-dir resolution (`SIPHON_STATE_HOME` / `XDG_STATE_HOME`) | ✅ Works (unit-tested) |
57+
| Resume from prior state on restart | ✅ Works (in the scaffold loop) |
58+
| Capability gating on `CapCDC` | ✅ Works |
59+
| `siphon cdc` CLI subcommand | ❌ Not wired (no Cobra command) |
60+
| Continuous change streaming | ⚠️ Scaffold only (polling tick; not real CDC) |
61+
62+
CDC does **not run today**. No driver declares `CapCDC` true, so `RunCDC` is
63+
rejected with `ErrDriverUnsupported`. Even if a driver enabled it, the loop is a
64+
polling scaffold, not logical-replication streaming.
65+
66+
## Resume state
67+
68+
When CDC streaming is built, `RunCDC` resumes from the last persisted position.
69+
State files live at:
70+
71+
```text
72+
$SIPHON_STATE_HOME/cdc/<job-id>.state # if SIPHON_STATE_HOME is set
73+
$XDG_STATE_HOME/siphon/cdc/<job-id>.state # else if XDG_STATE_HOME is set
74+
$HOME/.local/state/siphon/cdc/<job-id>.state # default
75+
```
76+
77+
Each file is JSON: source/target profiles plus the last applied position (LSN
78+
for Postgres, binlog file + offset for MySQL/MariaDB).
79+
80+
## Limitations
81+
82+
- **Not enabled on any driver** — `CapCDC` is `false` everywhere, so CDC is
83+
rejected up front.
84+
- **No real streaming** — the scaffold polls on a ticker. Real logical-
85+
replication streaming (`pglogrepl` for Postgres, binlog tailing for
86+
MySQL/MariaDB) is deferred to a future revision.
87+
- **No CLI entry point** — `RunCDC` is internal application scaffolding, not a
88+
user-facing command yet.

0 commit comments

Comments
 (0)