Skip to content

Commit 59d80fa

Browse files
authored
faq: add incremental ingestion Q and native PostgreSQL replication CDC option (#1541)
* faq: add incremental ingestion Q and native PostgreSQL replication CDC option - Add new FAQ entry pointing to data-refresh docs for incremental ingestion - Extend CDC FAQ entry with native PostgreSQL logical replication (recommended for PG) - Add new Features > CDC > PostgreSQL Logical Replication page documenting wal_level=logical + pgoutput direct replication (no Debezium/Kafka) - Update CDC overview to list native PG replication alongside Debezium * faq, cdc: mention direct Kafka and DynamoDB Streams ingestion * docs: fix broken links by using .md extensions for cross-page refs * data-refresh: add combined incremental ingestion example (overlap, retention, soft deletes) Co-authored-by: lukekim <80174+lukekim@users.noreply.github.com>
1 parent a8a0223 commit 59d80fa

4 files changed

Lines changed: 348 additions & 7 deletions

File tree

website/docs/faq/index.md

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -98,25 +98,34 @@ Spice uses [Apache DataFusion](https://datafusion.apache.org/) as its primary qu
9898

9999
## 17. Does Spice support Change Data Capture (CDC)?
100100

101-
Yes. Spice supports CDC via Debezium, enabling real-time data ingestion and materialization from databases such as PostgreSQL and MySQL. [Learn More](../features/cdc).
101+
Yes. Spice supports streaming ingestion from several sources:
102102

103-
## 18. Can Spice integrate with existing BI tools?
103+
- **Native PostgreSQL logical replication** (recommended for Postgres sources). Spice connects directly to the source using Postgres' `wal_level=logical` + pgoutput and streams `INSERT`/`UPDATE`/`DELETE` events into the accelerator. [Learn more](../features/cdc/postgres-replication.md).
104+
- **[DynamoDB Streams](../components/data-connectors/dynamodb)** for Amazon DynamoDB sources — Spice consumes the table's change stream and applies `INSERT`/`UPDATE`/`DELETE` events to the accelerator with `refresh_mode: changes`.
105+
- **[Apache Kafka](../components/data-connectors/kafka)** for event-streaming topics — Spice consumes records directly with `refresh_mode: append` for real-time, append-only acceleration.
106+
- **[Debezium](../components/data-connectors/debezium)** (over Kafka), for sources where Debezium is already deployed, or for databases without a native Spice CDC path (MySQL, SQL Server, etc.). [Learn more](../features/cdc).
107+
108+
## 18. How do I keep an accelerated dataset incrementally up-to-date?
109+
110+
For sources with a monotonically-increasing version column (e.g. `updated_at`), Spice incrementally ingests new and modified records using [`time_column`](../reference/spicepod/datasets#time_column) + [`refresh_mode: append`](../features/data-acceleration/data-refresh#append), with [`refresh_append_overlap`](../reference/spicepod/datasets#accelerationrefresh_append_overlap) to tolerate clock skew and [`retention_period`](../reference/spicepod/datasets#accelerationretention_period) to evict old or soft-deleted records. Pair with `primary_key` + `on_conflict: upsert` to deduplicate re-read rows within the overlap window. See [Data Refresh](../features/data-acceleration/data-refresh) for configuration details and examples.
111+
112+
## 19. Can Spice integrate with existing BI tools?
104113

105114
Yes. Spice integrates with BI tools through standard SQL interfaces (ODBC, JDBC, Arrow Flight SQL), enabling accelerated, real-time analytics for dashboards and reporting. An official [Tableau Connector](../clients/tableau) is available and a [BI Acceleration](https://www.youtube.com/watch?v=blEtLgRKu0c) demo using Apache Superset.
106115

107-
## 19. How does Spice handle data privacy and compliance?
116+
## 20. How does Spice handle data privacy and compliance?
108117

109118
Spice provides secure, auditable data access through sandboxed runtimes, secure endpoint checks, and detailed telemetry and tracing. The Spice Cloud Platform (SCP) is SOC 2 Type II compliant, meeting enterprise security and compliance requirements.
110119

111-
## 20. Can Spice be used for real-time analytics?
120+
## 21. Can Spice be used for real-time analytics?
112121

113122
Yes. Spice accelerates data locally using Apache Arrow, Spice Cayenne (Vortex), DuckDB, SQLite, or PostgreSQL, enabling real-time analytics and sub-second query performance for data-intensive applications and dashboards.
114123

115-
## 21. How can developers contribute to Spice?
124+
## 22. How can developers contribute to Spice?
116125

117126
Developers can contribute by submitting code, documentation, or raising issues on [GitHub](https://github.com/spiceai/spiceai). See [CONTRIBUTING.md](https://github.com/spiceai/spiceai/blob/trunk/CONTRIBUTING) for guidelines.
118127

119-
## 22. Does Spice support schema evolution?
128+
## 23. Does Spice support schema evolution?
120129

121130
Spice infers the schema for datasets and views at startup and does not apply runtime schema changes by default. If the source schema changes while the runtime is running (for example, columns are added, removed, or their types change), data refreshes will fail with a schema mismatch error rather than silently applying the new schema. This behavior is intentional — it protects against unintentional or breaking schema changes propagating into accelerated tables.
122131

website/docs/features/cdc/index.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,12 @@ It is recommended to use CDC-accelerated datasets with persistent data accelerat
3333

3434
Enabling CDC by setting `refresh_mode: changes` in the acceleration settings requires support from the data connector to provide a stream of row-level changes.
3535

36-
Currently, the only supported data connector is [Debezium](../components/data-connectors/debezium).
36+
Spice currently supports streaming ingestion via:
37+
38+
- **[PostgreSQL Logical Replication](./postgres-replication.md)****recommended** for PostgreSQL sources. Spice connects directly to the source using Postgres' native logical replication protocol (`wal_level=logical` + pgoutput) and streams `INSERT`/`UPDATE`/`DELETE` events into the accelerator. No Kafka, no Debezium, no external services.
39+
- **[DynamoDB Streams](../../components/data-connectors/dynamodb#streams)** — for Amazon DynamoDB sources. Spice consumes the table's DynamoDB Streams directly and applies `INSERT`/`UPDATE`/`DELETE` events to the accelerator.
40+
- **[Apache Kafka](../../components/data-connectors/kafka)** — for event-streaming topics. Spice consumes records directly with `refresh_mode: append` for real-time, append-only acceleration (no separate CDC connector required).
41+
- **[Debezium](../../components/data-connectors/debezium)** (over Kafka) — for sources where Debezium + Kafka is already deployed, or for databases without a native Spice CDC path (MySQL, SQL Server, etc.).
3742

3843
## Example
3944

0 commit comments

Comments
 (0)