You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add documentation for new CDC runtime parameters introduced in
spiceai/spiceai#10745: cdc_prefetch_buffer, cdc_max_coalesced_envelopes,
cdc_max_coalesced_bytes, and cdc_commit_timeout_ms.
Copy file name to clipboardExpand all lines: website/docs/reference/spicepod/runtime.md
+27-1Lines changed: 27 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -127,7 +127,11 @@ Use `xxh3` (the default) for its superior speed in most scenarios. Use `ahash`,
127
127
128
128
## `runtime.params`
129
129
130
-
Optional. Global key-value parameters for the runtime. HTTP-based connectors (HTTP/HTTPS, GraphQL, GitHub) support the following rate control defaults:
130
+
Optional. Global key-value parameters for the runtime.
131
+
132
+
### HTTP Rate Control
133
+
134
+
HTTP-based connectors (HTTP/HTTPS, GraphQL, GitHub) support the following rate control defaults:
Datasets using `refresh_mode: changes` (CDC) support the following pipeline tunables. These control how change envelopes from CDC sources (e.g. PostgreSQL logical replication, Kafka, DynamoDB Streams) are buffered, coalesced, and committed.
155
+
156
+
| Parameter Name | Default | Range | Description |
| `cdc_prefetch_buffer` | `32` | `1`–`1024` | Channel depth between the CDC source-stream reader and the apply loop. Each slot holds one decoded change envelope. |
159
+
| `cdc_max_coalesced_envelopes` | `64` | `1`–`4096` | Maximum number of change envelopes coalesced into a single accelerator write. Coalescing amortizes per-envelope planning cost. |
160
+
| `cdc_max_coalesced_bytes` | `67108864` | `1`–`1073741824` | Byte budget for a coalesced burst (default 64 MiB, max 1 GiB). A single envelope may exceed this; otherwise the next envelope starts a new burst. |
161
+
| `cdc_commit_timeout_ms` | `30000` | `1`–`3600000` | Maximum time in milliseconds to wait for a source-side commit before logging a stall warning (default 30s, max 1hr). |
162
+
163
+
Out-of-range or unparseable values fall back to defaults with a warning. Environment variables (`SPICE_CDC_PREFETCH_BUFFER`, `SPICE_CDC_MAX_COALESCED_ENVELOPES`, `SPICE_CDC_MAX_COALESCED_BYTES`, `SPICE_CDC_COMMIT_TIMEOUT_MS`) are used as fallback when the corresponding `runtime.params` key is not set.
164
+
165
+
```yaml
166
+
runtime:
167
+
params:
168
+
cdc_prefetch_buffer: "32"
169
+
cdc_max_coalesced_envelopes: "64"
170
+
cdc_max_coalesced_bytes: "67108864"
171
+
cdc_commit_timeout_ms: "30000"
172
+
```
173
+
148
174
## `runtime.functions`
149
175
150
176
Controls whether [functions](../../features/functions) declared in the top-level `functions:` section (and `tools:` entries with `as_sql: true`) are registered with the SQL engine. Defaults to disabled.
0 commit comments