-
-
Notifications
You must be signed in to change notification settings - Fork 0
Feat/implement phase f #5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 14 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
f18e6a1
feat(dumps): 4KB JSON envelope header for dump files
nixrajput 92d94fa
feat(app): write+read 4KB envelope around every dump
nixrajput 85141d0
feat(jobs): bounded-buffer Stream with FillPercent backpressure metric
nixrajput 396ba83
feat(dumps): chain resolution with cycle + broken-chain detection
nixrajput a57a6cb
feat(app): restore walks dump chain (base + incrementals) with --up-to
nixrajput 506ab9f
test(app): chain-ordering, clean-once, and --up-to restore tests
nixrajput 17aadba
feat(driver/postgres): incremental WAL slot lifecycle + LSN capture
nixrajput f878294
feat(driver/mysql,mariadb): binlog incremental machinery via shared Conn
nixrajput 7872816
feat(app): canonical schema + cross-engine JSONL emit/consume
nixrajput c3350ff
feat(app,cli): native bounded-stream sync; scaffold cross-engine + in…
nixrajput a76afe4
feat(app): CDC continuous-sync scaffold with persistent state file
nixrajput b1d0741
docs: incremental, cross-engine, and CDC concept docs
nixrajput ca2dd66
docs: mark Phase F advanced-transfer status (partial; gated CLI paths)
nixrajput 26f56c7
fix(sync): reword --continuous hint (no siphon cdc command exists yet)
nixrajput 55ee6fc
docs: update project status table for Phase F and clarify ops features
nixrajput 6f75785
fix(driver/postgres): lowercase replication slot name (pg allows only…
nixrajput 55d541f
ci: retrigger checks for slot-name fix
nixrajput d3cbf63
fix: address PR #5 review (cross-engine, restore safety, stream bound…
nixrajput File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| # Change data capture (CDC) | ||
|
|
||
| CDC ("continuous sync" / follow mode) keeps a target database in step with a | ||
| source by streaming changes as they happen, rather than re-running a full | ||
| backup/restore. In siphon this is **scaffolding today** — the resume-state | ||
| plumbing and capability gating are in place, but continuous streaming is a | ||
| future deliverable and does not run. | ||
|
|
||
| ## Table of contents | ||
|
|
||
| - [What exists](#what-exists) | ||
| - [The CLI surface](#the-cli-surface) | ||
| - [Status](#status) | ||
| - [Resume state](#resume-state) | ||
| - [Limitations](#limitations) | ||
|
|
||
| ## What exists | ||
|
|
||
| `internal/app/cdc.go` holds: | ||
|
|
||
| - **`CDCState`** — the persisted resume record (`job_id`, source/target | ||
| profiles, `last_applied_lsn` for Postgres, `last_binlog_file` + | ||
| `last_binlog_pos` for MySQL/MariaDB, `updated_at`). | ||
| - **`saveCDCState` / `loadCDCState`** — JSON state files under a per-user | ||
| directory resolved from `SIPHON_STATE_HOME`, then `XDG_STATE_HOME`, then | ||
| `$HOME/.local/state/siphon/cdc`. | ||
| - **`RunCDC`** — a capability-gated entry point. It resolves the source and | ||
| target profiles, then requires both drivers to advertise `CapCDC`. Its body is | ||
| a **polling scaffold**: a 10-second ticker that persists state each tick and | ||
| resumes from a prior run's state on restart. It is **not** real change | ||
| streaming. | ||
|
|
||
| `CDCState` save/load round-trip, the state-dir resolution, and the | ||
| no-capability rejection are unit-tested (`internal/app/cdc_test.go`). | ||
|
|
||
| ## The CLI surface | ||
|
|
||
| There is **no `siphon cdc` subcommand wired today.** `RunCDC` exists as an | ||
| application function but is not registered as a Cobra command | ||
| (`internal/cli/root.go` wires `backup`, `restore`, `sync`, `verify`, `inspect`, | ||
| `profile`, `dumps`, `config`, `schedule`, `tunnel` — no `cdc`). | ||
|
|
||
| `sync --continuous` exposes the flag but does not run CDC; it returns a clear | ||
| error pointing at the (not-yet-wired) `siphon cdc` follow-up: | ||
|
|
||
| ```bash | ||
| siphon sync --from pg-prod --to pg-replica --continuous | ||
| # Error: continuous CDC sync is not wired here; use `siphon cdc` (Phase F Task 10) | ||
| ``` | ||
|
|
||
| ## Status | ||
|
|
||
| | Capability | Status | | ||
| | --- | --- | | ||
| | `CDCState` persistence (save/load) | ✅ Works (unit-tested) | | ||
| | State-dir resolution (`SIPHON_STATE_HOME` / `XDG_STATE_HOME`) | ✅ Works (unit-tested) | | ||
| | Resume from prior state on restart | ✅ Works (in the scaffold loop) | | ||
| | Capability gating on `CapCDC` | ✅ Works | | ||
| | `siphon cdc` CLI subcommand | ❌ Not wired (no Cobra command) | | ||
| | Continuous change streaming | ⚠️ Scaffold only (polling tick; not real CDC) | | ||
|
|
||
| CDC does **not run today**. No driver declares `CapCDC` true, so `RunCDC` is | ||
| rejected with `ErrDriverUnsupported`. Even if a driver enabled it, the loop is a | ||
| polling scaffold, not logical-replication streaming. | ||
|
|
||
| ## Resume state | ||
|
|
||
| When CDC streaming is built, `RunCDC` resumes from the last persisted position. | ||
| State files live at: | ||
|
|
||
| ```text | ||
| $SIPHON_STATE_HOME/cdc/<job-id>.state # if SIPHON_STATE_HOME is set | ||
| $XDG_STATE_HOME/siphon/cdc/<job-id>.state # else if XDG_STATE_HOME is set | ||
| $HOME/.local/state/siphon/cdc/<job-id>.state # default | ||
| ``` | ||
|
|
||
| Each file is JSON: source/target profiles plus the last applied position (LSN | ||
| for Postgres, binlog file + offset for MySQL/MariaDB). | ||
|
|
||
| ## Limitations | ||
|
|
||
| - **Not enabled on any driver** — `CapCDC` is `false` everywhere, so CDC is | ||
| rejected up front. | ||
| - **No real streaming** — the scaffold polls on a ticker. Real logical- | ||
| replication streaming (`pglogrepl` for Postgres, binlog tailing for | ||
| MySQL/MariaDB) is deferred to a future revision. | ||
| - **No CLI entry point** — `RunCDC` is internal application scaffolding, not a | ||
| user-facing command yet. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # Cross-engine sync | ||
|
|
||
| Cross-engine sync moves data between *different* database engines (e.g. Postgres | ||
| → MySQL) by pivoting through a **canonical schema**: the source's native column | ||
| types are normalized into an engine-independent vocabulary, then mapped back to | ||
| the target engine's dialect. Rows travel as JSONL with per-engine identifier | ||
| quoting and bind-parameter placeholders. | ||
|
|
||
| ## Table of contents | ||
|
|
||
| - [How it works](#how-it-works) | ||
| - [The CLI surface](#the-cli-surface) | ||
| - [Status](#status) | ||
| - [Type-mapping matrix](#type-mapping-matrix) | ||
| - [Scope and limitations](#scope-and-limitations) | ||
|
|
||
| ## How it works | ||
|
|
||
| The canonical-schema machinery lives in `internal/app/canonical*.go`: | ||
|
|
||
| - `CanonicalType` / `CanonicalColumn` / `CanonicalSchema` are the | ||
| engine-independent pivot (`canonical.go`). | ||
| - `MapToNative(engine, col)` renders a canonical column as a native column type | ||
| for `postgres`, `mysql`, or `mariadb`. `VARCHAR`/`NUMERIC` carry their | ||
| precision (and scale, for `NUMERIC`) when known; otherwise the type is emitted | ||
| bare and the engine applies its default. | ||
| - Identifiers are quoted per engine (`"…"` for Postgres, `` `…` `` for | ||
| MySQL/MariaDB) and escaped, since table/column names cannot be passed as bind | ||
| parameters. Placeholders are `$n` (Postgres) or `?` (MySQL/MariaDB). | ||
| - The emit/consume halves (`canonical_emit.go`, `canonical_consume.go`) stream a | ||
| schema + rows as JSONL. | ||
|
|
||
| This machinery is built and unit-tested (`canonical_test.go`). | ||
|
|
||
| ## The CLI surface | ||
|
|
||
| ```bash | ||
| # Cross-engine sync from a Postgres source profile into a MySQL target profile. | ||
| siphon sync --from pg-prod --to mysql-staging --cross-engine | ||
| ``` | ||
|
|
||
| ## Status | ||
|
|
||
| | Capability | Status | | ||
| | --- | --- | | ||
| | Canonical type vocabulary + `MapToNative` matrix | ✅ Works (unit-tested) | | ||
| | JSONL emit/consume with per-engine quoting + placeholders | ✅ Works (unit-tested) | | ||
| | `sync --cross-engine` end-to-end | ⚠️ Gated off (follow-up) | | ||
|
|
||
| The type-mapping and JSONL-transfer machinery exists and is tested, but | ||
| `sync --cross-engine` is **capability-gated** on `CrossEngineSource` | ||
| (source driver) and `CrossEngineTarget` (target driver). **No driver declares | ||
| either capability today**, so the request is rejected with | ||
| `ErrDriverUnsupported` (`internal/app/sync.go`, `runCrossEngineSync`). | ||
|
|
||
| The reason is deliberate: cross-engine translation needs a *typed* | ||
| `CanonicalSchema` (column types), and `driver.Inspect` does not carry column | ||
| types yet. Rather than fabricate a schema, the path stays gated until typed | ||
| schema introspection lands and a driver flips its cross-engine capabilities to | ||
| true. At that point `runCrossEngineSync` gains the real | ||
| emit → translate → consume pipeline. | ||
|
|
||
| ```bash | ||
| siphon sync --from pg-prod --to mysql-staging --cross-engine | ||
| # Error: cross-engine sync requires typed schema introspection, not yet available | ||
| ``` | ||
|
|
||
| ## Type-mapping matrix | ||
|
|
||
| `MapToNative` maps each canonical type to a native type. These mappings are real | ||
| and usable even though the CLI path is gated: | ||
|
|
||
| | Canonical type | Postgres | MySQL | MariaDB | | ||
| | --- | --- | --- | --- | | ||
| | `int` | `integer` | `INT` | `INT` | | ||
| | `bigint` | `bigint` | `BIGINT` | `BIGINT` | | ||
| | `text` | `text` | `TEXT` | `TEXT` | | ||
| | `varchar` | `varchar` | `VARCHAR` | `VARCHAR` | | ||
| | `boolean` | `boolean` | `TINYINT(1)` | `TINYINT(1)` | | ||
| | `numeric` | `numeric` | `DECIMAL` | `DECIMAL` | | ||
| | `uuid` | `uuid` | `CHAR(36)` | `CHAR(36)` | | ||
| | `timestamptz` | `timestamptz` | `TIMESTAMP` | `TIMESTAMP` | | ||
| | `json` | `jsonb` | `JSON` | `JSON` | | ||
|
|
||
| `varchar` with a known length renders as e.g. `VARCHAR(255)`; `numeric` with | ||
| known precision/scale renders as e.g. `DECIMAL(10,2)`. | ||
|
|
||
| ## Scope and limitations | ||
|
|
||
| The v1 cross-engine scope (once the path is wired) is **data only**: | ||
|
|
||
| - Triggers, views, and stored functions are **skipped**. | ||
| - Indexes are created but **not optimized** for the target engine. | ||
| - Foreign keys are **deferred**. | ||
| - Engines covered by the matrix: `postgres`, `mysql`, `mariadb`. An unknown | ||
| engine or an unmappable canonical type is an error from `MapToNative`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # Incremental backups | ||
|
|
||
| Incremental backups capture only what changed since an earlier dump, so a chain | ||
| of small incrementals can stand in for repeated full dumps. siphon records the | ||
| relationship between dumps in the **dump envelope** (a 4 KB JSON header | ||
| prepended to every dump) and reconstructs the full picture at restore time by | ||
| walking the base → incremental chain. | ||
|
|
||
| ## Table of contents | ||
|
|
||
| - [How it works](#how-it-works) | ||
| - [The CLI surface](#the-cli-surface) | ||
| - [Status](#status) | ||
| - [Examples](#examples) | ||
| - [Limitations and runtime gates](#limitations-and-runtime-gates) | ||
|
|
||
| ## How it works | ||
|
|
||
| Every dump carries an envelope (`internal/dumps/envelope.go`) with a `type` | ||
| (`base` or `incremental`), a `base_id`/`parent_id`, and engine-specific resume | ||
| coordinates: `wal_start`/`wal_end` for Postgres, `binlog_file` + | ||
| `binlog_start`/`binlog_end` for MySQL/MariaDB. | ||
|
|
||
| `Catalog.ResolveChain` (`internal/dumps/chain.go`) walks `parent_id` backwards | ||
| from a target dump to its base, detecting cycles and broken chains rather than | ||
| looping or silently truncating. `Restore` then applies the resolved chain in | ||
| order, base first (`internal/app/restore.go`). A plain, non-incremental dump | ||
| resolves to a single-element chain, so the same restore path serves both. | ||
|
|
||
| The driver-level capture machinery also exists: | ||
|
|
||
| - **Postgres** (`internal/driver/postgres/incremental.go`) creates a temporary | ||
| physical replication slot and records the start/end LSN around a base backup, | ||
| so a later incremental can resume from the correct WAL position. | ||
| - **MySQL/MariaDB** (`internal/driver/_mysqlcommon/incremental.go`) captures the | ||
| binlog file + position via `SHOW BINARY LOG STATUS` (MySQL 8.4+) or | ||
| `SHOW MASTER STATUS` (older MySQL / MariaDB). | ||
|
|
||
| ## The CLI surface | ||
|
|
||
| ```bash | ||
| # Restore a dump, walking its base→incremental chain automatically. | ||
| siphon restore <dump-id> --profile <target> | ||
|
|
||
| # Stop applying the chain after a specific dump (point-in-chain restore). | ||
| siphon restore <dump-id> --profile <target> --up-to <intermediate-id> | ||
|
|
||
| # Request an incremental backup (NOT yet wired — see Status). | ||
| siphon backup <profile> --incremental --base <base-dump-id> | ||
| ``` | ||
|
|
||
| ## Status | ||
|
|
||
| | Capability | Status | | ||
| | --- | --- | | ||
| | Dump envelope (type/base/parent, WAL & binlog fields) | ✅ Works | | ||
| | Chain resolution (`ResolveChain`) | ✅ Works | | ||
| | Chain-walking restore (base → incrementals, in order) | ✅ Works | | ||
| | `restore --up-to <id>` (stop chain early) | ✅ Works | | ||
| | Driver-level capture machinery (Postgres WAL slot/LSN, MySQL/MariaDB binlog pos) | ✅ Exists | | ||
| | `backup --incremental --base <id>` end-to-end | ⚠️ Not yet wired (follow-up) | | ||
|
|
||
| The restore-side chain machinery and the envelope are in place and tested. The | ||
| `--incremental` backup path is a documented follow-up: the driver machinery and | ||
| the chain restore are both ready, but the wiring that ties them together — read | ||
| the base envelope for the parent WAL/binlog position, invoke the driver's | ||
| incremental capture method, and write an `incremental`-type catalog entry — is | ||
| not yet connected. Until then `siphon backup --incremental` returns a clear | ||
| error: *"incremental backup is not yet wired end-to-end (Phase F follow-up); the | ||
| driver-level machinery exists"* (`internal/cli/backup.go`). | ||
|
|
||
| ## Examples | ||
|
|
||
| Chain-walking restore (works today). If `inc-2` was built on `inc-1` on | ||
| `base-0`, restoring `inc-2` applies all three in order: | ||
|
|
||
| ```bash | ||
| siphon restore inc-2 --profile prod-replica | ||
| ``` | ||
|
|
||
| Point-in-chain restore with `--up-to` (works today) — apply only up to and | ||
| including `inc-1`, skipping `inc-2`: | ||
|
|
||
| ```bash | ||
| siphon restore inc-2 --profile prod-replica --up-to inc-1 | ||
| ``` | ||
|
|
||
| A typo'd `--up-to` is rejected (the dump isn't in the chain) rather than | ||
| silently restoring more than asked. | ||
|
|
||
| Requesting an incremental backup (returns the not-wired error today): | ||
|
|
||
| ```bash | ||
| siphon backup prod --incremental --base base-0 | ||
| # Error: incremental backup is not yet wired end-to-end (Phase F follow-up); | ||
| # the driver-level machinery exists | ||
| ``` | ||
|
|
||
| ## Limitations and runtime gates | ||
|
|
||
| These apply to the incremental **backup** path once it is wired: | ||
|
|
||
| - **Postgres** anchors WAL retention with a temporary physical replication slot. | ||
| Orphan-slot cleanup is **not yet built** — an orphaned slot pins WAL on the | ||
| server and must be dropped manually until automatic cleanup ships. | ||
| - **MySQL/MariaDB** require `binlog_format=ROW` for usable incrementals. | ||
| - **Cross-version incrementals are unsupported** (`CrossVersionIncremental: | ||
| false`): a chain must be captured and restored against the same engine major | ||
| version. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.