Skip to content

faq: add incremental ingestion Q and native PostgreSQL replication CDC option#1541

Merged
lukekim merged 8 commits into
trunkfrom
lukim/faq
Apr 21, 2026
Merged

faq: add incremental ingestion Q and native PostgreSQL replication CDC option#1541
lukekim merged 8 commits into
trunkfrom
lukim/faq

Conversation

@lukekim
Copy link
Copy Markdown
Contributor

@lukekim lukekim commented Apr 20, 2026

Documents incremental ingestion in the FAQ and adds coverage for native PostgreSQL logical replication as a first-class CDC option alongside Debezium.

Changes

FAQ (website/docs/faq/index.md)

  • New Q18. How do I keep an accelerated dataset incrementally up-to-date? — points to existing Data Refresh docs (time_column + refresh_mode: append + refresh_append_overlap + retention_period + primary_key/on_conflict: upsert).
  • Updated Q17. Does Spice support Change Data Capture (CDC)? to list:
    1. Native PostgreSQL logical replication (recommended for PG sources) — links to new page below.
    2. Debezium (existing) — for Debezium + Kafka deployments or non-PG databases.
  • Renumbered subsequent questions.

CDC overview (website/docs/features/cdc/index.md)

Replaced the "Currently, the only supported data connector is Debezium" line with a bullet list presenting native PG replication and Debezium as the two supported CDC paths.

New page: website/docs/features/cdc/postgres-replication.md

Full user-facing documentation for PostgreSQL logical replication, covering:

  • How it works (publication + replication slot + pgoutput + REPEATABLE READ snapshot, resume from confirmed_flush_lsn)
  • Prerequisites: wal_level=logical, managed service notes (RDS/Aurora/Cloud SQL/Azure/Supabase/Neon), replica identity, required grants
  • Minimal configuration example (refresh_mode: changes + primary_key + on_conflict: upsert)
  • Full pg_replication_* parameter reference, SSL mode table for the WAL stream
  • Accelerator engine matrix (INSERT/UPDATE/DELETE support per engine)
  • Multi-replica deployments (StatefulSet + SPICE_INSTANCE_ID)
  • Operations: monitoring lag, decommissioning replicas, rebuilding, resilience
  • Metrics (dataset_postgres_replication_*)
  • Troubleshooting table, limitations, and a Debezium + Kafka comparison

Sourced from the internal docs/features/postgres-replication.md in the spiceai repo.

…C option

- Add new FAQ entry pointing to data-refresh docs for incremental ingestion
- Extend CDC FAQ entry with native PostgreSQL logical replication (recommended for PG)
- Add new Features > CDC > PostgreSQL Logical Replication page documenting
  wal_level=logical + pgoutput direct replication (no Debezium/Kafka)
- Update CDC overview to list native PG replication alongside Debezium
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 20, 2026

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ Has at least one of the required labels: area/blog, area/docs, area/cookbook, dependencies
  • ✅ No banned labels detected
  • ✅ Has at least one assignee: lukekim

@lukekim lukekim self-assigned this Apr 21, 2026
@lukekim lukekim added this to the v2.0 milestone Apr 21, 2026
@github-actions
Copy link
Copy Markdown

🚀 deployed to https://5f418b54.spiceai-org-website.pages.dev

@github-actions
Copy link
Copy Markdown

🚀 deployed to https://beda62d3.spiceai-org-website.pages.dev

@lukekim lukekim enabled auto-merge (squash) April 21, 2026 00:53
Comment thread website/docs/faq/index.md Outdated
Comment thread website/docs/features/cdc/postgres-replication.md Outdated
Comment thread website/docs/features/cdc/postgres-replication.md Outdated
Comment thread website/docs/features/cdc/postgres-replication.md Outdated
Comment thread website/docs/features/cdc/postgres-replication.md
lukekim and others added 4 commits April 20, 2026 18:55
Co-authored-by: Phillip LeBlanc <phillip@spice.ai>
Co-authored-by: Phillip LeBlanc <phillip@spice.ai>
Co-authored-by: Phillip LeBlanc <phillip@spice.ai>
Co-authored-by: Phillip LeBlanc <phillip@spice.ai>
@github-actions
Copy link
Copy Markdown

🚀 deployed to https://fc7937c3.spiceai-org-website.pages.dev

@github-actions
Copy link
Copy Markdown

🚀 deployed to https://f037d1a1.spiceai-org-website.pages.dev

@lukekim lukekim merged commit af90c27 into trunk Apr 21, 2026
6 checks passed
@lukekim lukekim deleted the lukim/faq branch April 21, 2026 02:01
@github-actions
Copy link
Copy Markdown

🚀 deployed to https://7e930642.spiceai-org-website.pages.dev

@github-actions
Copy link
Copy Markdown

🚀 deployed to https://d89e6ead.spiceai-org-website.pages.dev

Copilot AI pushed a commit that referenced this pull request Apr 21, 2026
…C option (#1541)

* faq: add incremental ingestion Q and native PostgreSQL replication CDC option

- Add new FAQ entry pointing to data-refresh docs for incremental ingestion
- Extend CDC FAQ entry with native PostgreSQL logical replication (recommended for PG)
- Add new Features > CDC > PostgreSQL Logical Replication page documenting
  wal_level=logical + pgoutput direct replication (no Debezium/Kafka)
- Update CDC overview to list native PG replication alongside Debezium

* faq, cdc: mention direct Kafka and DynamoDB Streams ingestion

* docs: fix broken links by using .md extensions for cross-page refs

* data-refresh: add combined incremental ingestion example (overlap, retention, soft deletes)
Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants