Skip to content

Commit a9e52f4

Browse files
claudespiceclaudespice
authored
docs(features): add Database Replication and CDC, and the Analytics Replica use case (#136)
1 parent d279ae8 commit a9e52f4

5 files changed

Lines changed: 191 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ With the **Spice.ai Cloud Platform** you can:
4545
| Use Case | Description |
4646
| --------------------------------------------------- | ---------------------------------------------------------- |
4747
| [Agentic AI Apps](use-cases/agentic-ai-apps.md) | Build AI agent backends with unified data and model access |
48+
| [Analytics Replica](use-cases/analytics-replica.md) | Run analytics on operational data without ETL or migration |
4849
| [Database CDN](use-cases/database-cdn.md) | Cache and accelerate hot data for low-latency applications |
4950
| [Data Lakehouse](use-cases/data-lakehouse.md) | Federated queries across warehouses, lakes, and databases |
5051
| [Enterprise Search](use-cases/enterprise-search.md) | Semantic search across enterprise data sources |

‎SUMMARY.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
* [DuckDB Data Accelerator](features/data-acceleration/duckdb-data-accelerator.md)
2020
* [PostgreSQL Data Accelerator](features/data-acceleration/postgresql-data-accelerator.md)
2121
* [SQLite Data Accelerator](features/data-acceleration/sqlite-data-accelerator.md)
22+
* [Database Replication and CDC](features/database-replication-and-cdc.md)
2223
* [Search & Retrieval](features/search-and-retrieval.md)
2324
* [AI Gateway](features/ai-gateway.md)
2425
* [Semantic Models](features/semantic-models.md)
@@ -79,6 +80,7 @@
7980
## Use-Cases
8081

8182
* [Agentic AI Apps](use-cases/agentic-ai-apps.md)
83+
* [Analytics Replica](use-cases/analytics-replica.md)
8284
* [Database CDN](use-cases/database-cdn.md)
8385
* [Data Lakehouse](use-cases/data-lakehouse.md)
8486
* [Enterprise Search](use-cases/enterprise-search.md)

‎features/data-acceleration/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ Spice supports three modes to refresh/update locally accelerated data from a con
3333
| `append` | Append/add data to the dataset on each refresh | Append-only, immutable datasets, such as time-series or log data |
3434
| `changes` | Apply incremental changes | Customer order lifecycle table |
3535

36+
`refresh_mode: changes` streams committed inserts, updates, and deletes from the source's own changelog. See [Database Replication and CDC](../database-replication-and-cdc.md) for supported sources and configuration.
37+
3638
#### Example - Accelerate with arrow accelerator under full refresh mode <a href="#example" id="example"></a>
3739

3840
```yaml

‎features/database-replication-and-cdc.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
icon: arrows-rotate
3+
description: Replicate committed changes from operational databases into an accelerated, query-ready replica using change data capture
4+
---
5+
6+
# Database Replication and CDC
7+
8+
**Database replication** keeps an accelerated dataset continuously in step with its source by reading the source database's own changelog. Committed inserts, updates, and deletes are applied to the local replica within seconds, with no batch window and no external pipeline.
9+
10+
The mechanism is **change data capture (CDC)**: rather than re-reading the source table on a schedule, Spice consumes the stream of changes the database already produces for its own recovery and replication — the PostgreSQL write-ahead log, a MongoDB change stream, a DynamoDB stream — and applies each change to the accelerator as it commits.
11+
12+
Replication is enabled by setting `refresh_mode: changes` on an accelerated dataset.
13+
14+
{% hint style="info" %}
15+
`changes` is one of three [refresh modes](data-acceleration/README.md#refresh-modes). Use `full` to replace a dataset on each refresh, `append` for immutable or time-series data, and `changes` to mirror a mutable source that emits a change feed.
16+
{% endhint %}
17+
18+
### Why replicate
19+
20+
Running analytical queries against a production database competes with transaction processing for the same connections, buffer pool, and CPU. The usual alternatives each carry a cost:
21+
22+
* **ETL pipelines** add latency measured in minutes or hours, plus the infrastructure to build, schedule, and monitor them.
23+
* **Read replicas** relieve the primary but run the same row-oriented engine, so analytical scans remain slow.
24+
* **HTAP databases** require migrating off the existing system and couple transactional and analytical failure domains.
25+
26+
CDC-based replication into a columnar accelerator avoids all three. The operational database keeps serving transactions, analytical load lands on separate storage and compute, and the replica stays seconds behind rather than hours.
27+
28+
For the architecture built on this capability, see [Analytics Replica](../use-cases/analytics-replica.md).
29+
30+
### Supported sources
31+
32+
| Source | Mechanism | Configuration |
33+
| ------ | --------- | ------------- |
34+
| [PostgreSQL](../building-blocks/data-connectors/postgres.md) | Logical replication from the write-ahead log | `refresh_mode: changes` |
35+
| [MongoDB](../building-blocks/data-connectors/mongodb.md) | Change streams on the source collection | `refresh_mode: changes` |
36+
| [DynamoDB](../building-blocks/data-connectors/dynamodb.md) | DynamoDB Streams | `refresh_mode: changes` |
37+
| [Apache Kafka](../building-blocks/data-connectors/kafka.md) | Event stream consumption | `refresh_mode: append` |
38+
| [Debezium](../building-blocks/data-connectors/debezium.md) | Debezium change events over Kafka | `refresh_mode: changes` |
39+
40+
{% hint style="info" %}
41+
Sources without a native Spice change feed — including MySQL and SQL Server — replicate through [Debezium](../building-blocks/data-connectors/debezium.md) over Kafka.
42+
{% endhint %}
43+
44+
### Configuration
45+
46+
A replicated dataset needs a `primary_key` so that updates and deletes can be matched to existing rows, and an `on_conflict` rule so that repeated keys upsert rather than duplicate.
47+
48+
```yaml
49+
datasets:
50+
- from: postgres:public.orders
51+
name: orders
52+
params:
53+
pg_host: postgres.example-org.com
54+
pg_port: '5432'
55+
pg_user: spice
56+
pg_pass: ${secrets:pg_pass}
57+
pg_db: myapp
58+
pg_sslmode: verify-full
59+
acceleration:
60+
enabled: true
61+
engine: cayenne
62+
mode: file
63+
refresh_mode: changes
64+
primary_key: id
65+
on_conflict:
66+
id: upsert
67+
```
68+
69+
On startup Spice loads an initial snapshot of the table, then switches to streaming changes. No `refresh_check_interval` is required — changes are applied as they arrive rather than on a poll.
70+
71+
Any accelerator engine can back a replicated dataset. [Cayenne](../building-blocks/data-accelerators/cayenne.md) is built for this workload, sustaining a high-throughput change feed while serving analytical scans from the same table.
72+
73+
### PostgreSQL prerequisites
74+
75+
Logical replication must be enabled on the source server:
76+
77+
```
78+
wal_level = logical
79+
max_replication_slots = 10
80+
max_wal_senders = 10
81+
```
82+
83+
Each replicated table needs a primary key, or `REPLICA IDENTITY FULL`, so that updates and deletes carry enough information to identify the affected row:
84+
85+
```sql
86+
ALTER TABLE public.orders REPLICA IDENTITY FULL;
87+
```
88+
89+
The connecting role needs the `REPLICATION` attribute, plus `SELECT` on the replicated tables. Spice creates and manages its own replication slot and publication.
90+
91+
{% hint style="warning" %}
92+
An inactive replication slot causes the source server to retain write-ahead log segments indefinitely, which can exhaust disk on the primary. Drop the slot on the source if a replicated dataset is removed permanently.
93+
{% endhint %}
94+
95+
### Handling deletes
96+
97+
CDC propagates hard deletes, which sets replication apart from incremental ingestion. A `DELETE` on the source removes the row from the replica on the next change event, with no reconciling full refresh and no soft-delete convention in the source schema.
98+
99+
Sources that expose no change feed at all — HTTP APIs, for example — use [incremental ingestion](data-acceleration/README.md#incremental-ingestion) with `refresh_mode: append` instead, where deletes are handled by soft-delete tombstones or a periodic full refresh.
100+
101+
### Related
102+
103+
* [Analytics Replica](../use-cases/analytics-replica.md) — the deployment pattern built on replication
104+
* [Data Acceleration](data-acceleration/README.md) — refresh modes, incremental ingestion, and retention
105+
* [Database CDN](../use-cases/database-cdn.md) — colocating a hot working set with an application
106+
* [Change data capture](https://spiceai.org/docs/features/cdc) in the Spice.ai OSS documentation

‎use-cases/analytics-replica.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
---
2+
description: Attach a columnar analytics replica to an operational database without ETL or migration
3+
icon: database
4+
---
5+
6+
# Analytics Replica
7+
8+
The **analytics replica pattern** attaches a dedicated columnar node to an operational database and lets it absorb the analytical query load. The primary keeps serving transactions. Analytical questions — which customers churned today, which orders are stuck, how revenue moved this hour — run against a replica that stays seconds behind.
9+
10+
It is the shortest path from an operational database to analytical and AI workloads, because nothing has to be migrated and no pipeline has to be built.
11+
12+
### The problem
13+
14+
Analytical queries and transactional queries want opposite things from a database. Transactions want narrow row lookups and short-lived locks. Analytics wants wide scans over large ranges. Running both on one primary means the reporting query and the checkout path compete for the same buffer pool and CPU.
15+
16+
The established workarounds each trade one problem for another:
17+
18+
| Approach | What it costs |
19+
| -------- | ------------- |
20+
| **ETL pipeline** | Data arrives minutes or hours late, and the pipeline itself becomes infrastructure to build, schedule, monitor, and repair |
21+
| **Read replica** | Removes load from the primary, but runs the same row-oriented engine — analytical scans are just as slow |
22+
| **HTAP database** | Requires migrating off the current database, and recouples transactional and analytical failure domains |
23+
| **Data warehouse** | Strong for analytics, but reached through a pipeline, so it inherits the latency and the operational burden |
24+
25+
### How it works
26+
27+
An analytics replica connects to the operational database, loads an initial snapshot, then uses change data capture (CDC) to apply committed changes continuously from the source's native changelog — the PostgreSQL write-ahead log, a MongoDB change stream, a DynamoDB stream. There is no batch interval; a committed change is queryable within seconds.
28+
29+
The replica stores data in a columnar format on its own storage and compute, so scans are fast and the load never reaches the primary. Queries run through the same federated SQL interface as the rest of the platform, which means a replicated table can be joined against a data lake, a warehouse, or another operational system in a single query.
30+
31+
```yaml
32+
datasets:
33+
- from: postgres:public.orders
34+
name: orders
35+
params:
36+
pg_host: postgres.example-org.com
37+
pg_user: spice
38+
pg_pass: ${secrets:pg_pass}
39+
pg_db: myapp
40+
acceleration:
41+
enabled: true
42+
engine: cayenne
43+
mode: file
44+
refresh_mode: changes
45+
primary_key: id
46+
on_conflict:
47+
id: upsert
48+
```
49+
50+
See [Database Replication and CDC](../features/database-replication-and-cdc.md) for the supported sources, source prerequisites, and full configuration reference.
51+
52+
### Why it differs from a read replica
53+
54+
A read replica and an analytics replica solve different halves of the problem:
55+
56+
* A **read replica** moves load off the primary but keeps the row-oriented storage and execution engine, so a query scanning millions of rows is no faster than it was.
57+
* An **analytics replica** changes the storage layout and the execution engine. Data is columnar, scans read only the columns a query touches, and segment statistics skip data that cannot match.
58+
59+
Both keep the primary healthy. Only one makes the analytical query fast.
60+
61+
### Incremental adoption
62+
63+
The pattern is adopted table by table. Replicate one table, point one dashboard or one agent at it, and leave everything else untouched. There is no cutover, no schema migration, and no change to how the application writes.
64+
65+
That property matters most when the destination is an AI agent. Grounding an agent in production data usually stalls on the pipeline needed to expose that data safely. Replicating the handful of tables the agent needs is a smaller commitment than building a warehouse feed, and the agent queries data that is seconds old rather than a day stale.
66+
67+
### Governing access
68+
69+
Because queries reach the replica rather than the operational database, access is governed at the replica:
70+
71+
* Agents and applications authenticate to Spice instead of holding database credentials, so the primary's credentials never leave the infrastructure that owns them.
72+
* [Policy](../enterprise/features/policy.md) rules apply row-level filters and column masking at query time, based on the identity of the caller.
73+
* Queries are recorded, so what an agent read is auditable after the fact.
74+
75+
### Related
76+
77+
* [Database Replication and CDC](../features/database-replication-and-cdc.md) — configuration, supported sources, and prerequisites
78+
* [Federated SQL Query](../features/federated-sql-query.md) — joining a replica against other systems
79+
* [Database CDN](database-cdn.md) — colocating a hot working set with an application
80+
* [Agentic AI Apps](agentic-ai-apps.md) — grounding agents in operational data

0 commit comments

Comments
 (0)