Skip to content

Commit a385b33

Browse files
committed
faq: add incremental ingestion Q and native PostgreSQL replication CDC option
- Add new FAQ entry pointing to data-refresh docs for incremental ingestion - Extend CDC FAQ entry with native PostgreSQL logical replication (recommended for PG) - Add new Features > CDC > PostgreSQL Logical Replication page documenting wal_level=logical + pgoutput direct replication (no Debezium/Kafka) - Update CDC overview to list native PG replication alongside Debezium
1 parent 6df6517 commit a385b33

3 files changed

Lines changed: 311 additions & 7 deletions

File tree

website/docs/faq/index.md

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -98,25 +98,32 @@ Spice uses [Apache DataFusion](https://datafusion.apache.org/) as its primary qu
9898

9999
## 17. Does Spice support Change Data Capture (CDC)?
100100

101-
Yes. Spice supports CDC via Debezium, enabling real-time data ingestion and materialization from databases such as PostgreSQL and MySQL. [Learn More](../features/cdc).
101+
Yes. Spice supports CDC in two ways:
102102

103-
## 18. Can Spice integrate with existing BI tools?
103+
- **Native PostgreSQL logical replication** (recommended for Postgres sources). Spice connects directly to the source using Postgres' `wal_level=logical` + pgoutput and streams `INSERT`/`UPDATE`/`DELETE` events into the accelerator — no Debezium, no Kafka, no external services. [Learn more](../features/cdc/postgres-replication).
104+
- **[Debezium](../components/data-connectors/debezium)**, for sources where Debezium + Kafka is already deployed, or for non-PostgreSQL databases (MySQL, SQL Server, etc.). [Learn more](../features/cdc).
105+
106+
## 18. How do I keep an accelerated dataset incrementally up-to-date?
107+
108+
For sources with a monotonically-increasing version column (e.g. `updated_at`), Spice incrementally ingests new and modified records using [`time_column`](../reference/spicepod/datasets#time_column) + [`refresh_mode: append`](../features/data-acceleration/data-refresh#append), with [`refresh_append_overlap`](../reference/spicepod/datasets#accelerationrefresh_append_overlap) to tolerate clock skew and [`retention_period`](../reference/spicepod/datasets#accelerationretention_period) to evict old or soft-deleted records. Pair with `primary_key` + `on_conflict: upsert` to deduplicate re-read rows within the overlap window. See [Data Refresh](../features/data-acceleration/data-refresh) for configuration details and examples.
109+
110+
## 19. Can Spice integrate with existing BI tools?
104111

105112
Yes. Spice integrates with BI tools through standard SQL interfaces (ODBC, JDBC, Arrow Flight SQL), enabling accelerated, real-time analytics for dashboards and reporting. An official [Tableau Connector](../clients/tableau) is available and a [BI Acceleration](https://www.youtube.com/watch?v=blEtLgRKu0c) demo using Apache Superset.
106113

107-
## 19. How does Spice handle data privacy and compliance?
114+
## 20. How does Spice handle data privacy and compliance?
108115

109116
Spice provides secure, auditable data access through sandboxed runtimes, secure endpoint checks, and detailed telemetry and tracing. The Spice Cloud Platform (SCP) is SOC 2 Type II compliant, meeting enterprise security and compliance requirements.
110117

111-
## 20. Can Spice be used for real-time analytics?
118+
## 21. Can Spice be used for real-time analytics?
112119

113120
Yes. Spice accelerates data locally using Apache Arrow, Spice Cayenne (Vortex), DuckDB, SQLite, or PostgreSQL, enabling real-time analytics and sub-second query performance for data-intensive applications and dashboards.
114121

115-
## 21. How can developers contribute to Spice?
122+
## 22. How can developers contribute to Spice?
116123

117124
Developers can contribute by submitting code, documentation, or raising issues on [GitHub](https://github.com/spiceai/spiceai). See [CONTRIBUTING.md](https://github.com/spiceai/spiceai/blob/trunk/CONTRIBUTING) for guidelines.
118125

119-
## 22. Does Spice support schema evolution?
126+
## 23. Does Spice support schema evolution?
120127

121128
Spice infers the schema for datasets and views at startup and does not apply runtime schema changes by default. If the source schema changes while the runtime is running (for example, columns are added, removed, or their types change), data refreshes will fail with a schema mismatch error rather than silently applying the new schema. This behavior is intentional — it protects against unintentional or breaking schema changes propagating into accelerated tables.
122129

website/docs/features/cdc/index.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,10 @@ It is recommended to use CDC-accelerated datasets with persistent data accelerat
3333

3434
Enabling CDC by setting `refresh_mode: changes` in the acceleration settings requires support from the data connector to provide a stream of row-level changes.
3535

36-
Currently, the only supported data connector is [Debezium](../components/data-connectors/debezium).
36+
Spice currently supports CDC via:
37+
38+
- **[PostgreSQL Logical Replication](./postgres-replication)****recommended** for PostgreSQL sources. Spice connects directly to the source using Postgres' native logical replication protocol (`wal_level=logical` + pgoutput) and streams `INSERT`/`UPDATE`/`DELETE` events into the accelerator. No Kafka, no Debezium, no external services.
39+
- **[Debezium](../components/data-connectors/debezium)** — for sources where Debezium + Kafka is already deployed, or for non-PostgreSQL databases (MySQL, SQL Server, etc.).
3740

3841
## Example
3942

0 commit comments

Comments
 (0)