Skip to content

Commit b552215

Browse files
claudespicelukekim
authored andcommitted
docs: Document CDC pipeline tuning runtime params
Add documentation for new CDC runtime parameters introduced in spiceai/spiceai#10745: cdc_prefetch_buffer, cdc_max_coalesced_envelopes, cdc_max_coalesced_bytes, and cdc_commit_timeout_ms.
1 parent d1bf5cf commit b552215

1 file changed

Lines changed: 27 additions & 1 deletion

File tree

website/docs/reference/spicepod/runtime.md

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,11 @@ Use `xxh3` (the default) for its superior speed in most scenarios. Use `ahash`,
127127

128128
## `runtime.params`
129129

130-
Optional. Global key-value parameters for the runtime. HTTP-based connectors (HTTP/HTTPS, GraphQL, GitHub) support the following rate control defaults:
130+
Optional. Global key-value parameters for the runtime.
131+
132+
### HTTP Rate Control
133+
134+
HTTP-based connectors (HTTP/HTTPS, GraphQL, GitHub) support the following rate control defaults:
131135

132136
| Parameter Name | Description |
133137
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -145,6 +149,28 @@ runtime:
145149
http_requests_per_minute_limit: 200
146150
```
147151

152+
### CDC Pipeline Tuning
153+
154+
Datasets using `refresh_mode: changes` (CDC) support the following pipeline tunables. These control how change envelopes from CDC sources (e.g. PostgreSQL logical replication, Kafka, DynamoDB Streams) are buffered, coalesced, and committed.
155+
156+
| Parameter Name | Default | Range | Description |
157+
| ------------------------------ | ------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
158+
| `cdc_prefetch_buffer` | `32` | `1`–`1024` | Channel depth between the CDC source-stream reader and the apply loop. Each slot holds one decoded change envelope. |
159+
| `cdc_max_coalesced_envelopes` | `64` | `1`–`4096` | Maximum number of change envelopes coalesced into a single accelerator write. Coalescing amortizes per-envelope planning cost. |
160+
| `cdc_max_coalesced_bytes` | `67108864` | `1`–`1073741824` | Byte budget for a coalesced burst (default 64 MiB, max 1 GiB). A single envelope may exceed this; otherwise the next envelope starts a new burst. |
161+
| `cdc_commit_timeout_ms` | `30000` | `1`–`3600000` | Maximum time in milliseconds to wait for a source-side commit before logging a stall warning (default 30s, max 1hr). |
162+
163+
Out-of-range or unparseable values fall back to defaults with a warning. Environment variables (`SPICE_CDC_PREFETCH_BUFFER`, `SPICE_CDC_MAX_COALESCED_ENVELOPES`, `SPICE_CDC_MAX_COALESCED_BYTES`, `SPICE_CDC_COMMIT_TIMEOUT_MS`) are used as fallback when the corresponding `runtime.params` key is not set.
164+
165+
```yaml
166+
runtime:
167+
params:
168+
cdc_prefetch_buffer: "32"
169+
cdc_max_coalesced_envelopes: "64"
170+
cdc_max_coalesced_bytes: "67108864"
171+
cdc_commit_timeout_ms: "30000"
172+
```
173+
148174
## `runtime.functions`
149175

150176
Controls whether [functions](../../features/functions) declared in the top-level `functions:` section (and `tools:` entries with `as_sql: true`) are registered with the SQL engine. Defaults to disabled.

0 commit comments

Comments
 (0)