Add PostgreSQL catalog-level CDC acceleration recipe - #542
Draft
bjchambers wants to merge 11 commits into
Draft
Conversation
Demonstrates `acceleration: { refresh_mode: changes }` on a `from: pg`
catalog: auto-discover and CDC-accelerate every table (TPC-H SF1) into
Cayenne through a single shared replication slot, then watch source
INSERT/UPDATE/DELETE converge in the accelerated catalog.
Depends on the unreleased catalog-CDC feature (Spice v2.2+).
Depends on the companion fix in spiceai/spiceai#11897 making the CDC replication connection's password optional (mirroring the pool). A trust-auth source now needs only PG_USER, matching the sibling catalogs/postgres recipe.
Merged
5 tasks
A host-local PostgreSQL bound to localhost:5432 shadows the recipe's Docker container for host connections, causing 'PostgreSQL connection failed' at catalog setup. Document how to detect it (lsof) and resolve (free the port or change the published port).
Each insert/update/delete now echoes the exact 'docker exec ... psql -c' it issues against the source database before running it, so it's clear what the demo is doing to PostgreSQL (vs. querying Spice). Executes via psql -c so the printed command is exactly what runs.
Catalog CDC no longer requires a primary key on every table / errors otherwise (spiceai/spiceai#11951, shipping in v2.2 alongside the base support): each table is accelerated per its REPLICA IDENTITY (primary key, or a unique index via USING INDEX; FULL also works), and tables with no usable replica identity are skipped with a warning rather than failing the catalog. Update the intro, the comparison-table row, the config note, and the startup-summary log line (which gained a '; N tables skipped ...' clause).
…ligible-tables troubleshooting
… restart/multi-instance slot semantics (#11850)
… periodic refresh, not one-shot
…eds a key (FULL alone is skipped)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PostgreSQL Catalog-level CDC Acceleration recipe
Adds
catalogs/postgres-cdc/— a runnable recipe for catalog-level CDC acceleration: point afrom: pgcatalog at PostgreSQL, addacceleration: { refresh_mode: changes }, and Spice auto-discovers every table, snapshots each into Cayenne, and keeps them live from the WAL through a single shared replication slot — zero per-table config.It builds on the existing
catalogs/postgresrecipe (reuses the same TPC-H SF1 loader; every TPC-H table has a primary key, which catalog CDC requires) and adds a mutate-and-watch step showing sourceINSERT/UPDATE/DELETEconverging in the accelerated catalog.Contents
compose.yaml—postgres:16withwal_level=logical+ replication slots enabledspicepod.yaml— the catalog +acceleration: { refresh_mode: changes }blockmutate.sh—insert/update/deleteagainst the source for the convergence steppostgres-init/— TPC-H SF1 loader (verbatim from the sibling recipe)README.md— full walkthrough with captured outputValidated
Run end-to-end locally: 8 tables → 1 replication slot / 1 publication, exact bootstrap row counts (
lineitem6,001,215), and INSERT/UPDATE/DELETE each converging in ~1s (verified 150k=150k row parity). All output blocks in the README are from that run.Catalog-level CDC acceleration ships in Spice v2.2+ (feature PR not yet merged). Opening as a draft — hold merge until the feature lands and the version note is confirmed.