Skip to content

Commit 1926bfc

Browse files
authored
Merge branch 'trunk' into dependabot/npm_and_yarn/website/npm_and_yarn-3d438dea79
2 parents bc70907 + 9661a93 commit 1926bfc

14 files changed

Lines changed: 45 additions & 15 deletions

File tree

website/docs/components/data-connectors/delta-lake/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,7 @@ The table below shows the Delta Lake data types supported, along with the type m
181181
| `Decimal` | `Decimal128` |
182182
| `Array` | `List` |
183183
| `Struct` | `Struct` |
184+
| `Variant` | `Struct` |
184185
| `Map` | `Map` |
185186

186187
## Limitations

website/docs/components/data-connectors/mysql/deployment.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,9 @@ TLS is controlled via `mysql_sslmode`:
3636
| ------------- | ---------------------------------------------------------------- |
3737
| `disabled` | No TLS. |
3838
| `preferred` | Try TLS, fall back to plaintext. Not recommended for production. |
39-
| `required` | Require TLS; do **not** verify the server certificate. |
39+
| `required` | Require TLS and verify the server certificate against system root CAs. |
4040

41-
For production, use `required` with `mysql_sslrootcert` pointing to the CA bundle. The default is `required`, which encrypts the connection but does not validate the server's identity.
41+
For production, use `required` (the default). To verify against a specific CA rather than the system trust store, also set `mysql_sslrootcert` to the CA bundle path.
4242

4343
## Resilience Controls
4444

@@ -96,7 +96,7 @@ MySQL operations participate in Spice [task history](../../../reference/task_his
9696
## Known Limitations
9797

9898
- Only TCP connections are supported. Unix socket connections are not exposed through Spice configuration.
99-
- TLS with certificate verification (`verify_ca`, `verify_identity`) is not supported; only `disabled`, `preferred`, and `required` modes are available.
99+
- Only `disabled`, `preferred`, and `required` SSL modes are exposed. The `required` mode verifies the server certificate and domain name (equivalent to `verify_identity`). There is no mode that encrypts without verifying.
100100
- Large text/blob columns are fetched in their entirety per row; consider selecting only the columns you need when federating.
101101
- `mysql_sslmode: preferred` silently downgrades to plaintext on TLS negotiation failure and is not recommended for production.
102102

website/docs/features/data-acceleration/data-refresh.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,7 @@ datasets:
211211

212212
## Fast Cold Starts with Snapshots
213213

214-
File-based acceleration engines (DuckDB or SQLite) can rely on [acceleration snapshots](./snapshots) to download a pre-built database file on startup instead of waiting for the first refresh to finish. Configure a shared snapshot location under the top-level `snapshots` block and opt individual datasets in with `acceleration.snapshots: enabled`, `bootstrap_only`, or `create_only`. Snapshots are stored using Hive-style partitions (`month=YYYY-MM/day=YYYY-MM-DD/dataset=<name>`) and are only supported when each dataset writes to its own acceleration file.
214+
File-based acceleration engines (DuckDB, SQLite, Cayenne, or Turso) can rely on [acceleration snapshots](./snapshots) to download a pre-built database file on startup instead of waiting for the first refresh to finish. Configure a shared snapshot location under the top-level `snapshots` block and opt individual datasets in with `acceleration.snapshots: enabled`, `bootstrap_only`, or `create_only`. Snapshots are stored using Hive-style partitions (`month=YYYY-MM/day=YYYY-MM-DD/dataset=<name>`) and are only supported when each dataset writes to its own acceleration file.
215215

216216
## Filtered Refresh
217217

website/docs/features/data-acceleration/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ Consider a high-volume e-trading frontend application backed by an AWS RDS datab
4343
| `sqlite` | OLTP-style point lookups, concurrent reads/writes | `memory`, `file`, `file_create`, or `file_update` |
4444
| `postgres` | When a full SQL database is needed as accelerator | External |
4545
| `cayenne` | Large datasets (1TB+), high-performance columnar | `file`, `file_create`, or `file_update` |
46+
| `turso` | Embedded libSQL, lightweight file-based caching | `memory`, `file`, `file_create`, or `file_update` |
4647

4748
## Example
4849

website/docs/features/data-acceleration/snapshots.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ datasets:
3232
3333
## Overview
3434
35-
Acceleration snapshots let Spice reuse a pre-built acceleration file on startup instead of waiting for a full refresh. When a dataset uses a file-mode acceleration engine (DuckDB or SQLite) and the local file is missing (for example on first boot or when using ephemeral NVMe storage), Spice downloads the most recent snapshot from object storage and moves the dataset straight to a ready state.
35+
Acceleration snapshots let Spice reuse a pre-built acceleration file on startup instead of waiting for a full refresh. When a dataset uses a file-mode acceleration engine (DuckDB, SQLite, Cayenne, or Turso) and the local file is missing (for example on first boot or when using ephemeral NVMe storage), Spice downloads the most recent snapshot from object storage and moves the dataset straight to a ready state.
3636
3737
## How it works
3838

website/docs/reference/spicepod/datasets.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -361,9 +361,9 @@ The acceleration engine to use, defaults to `arrow`. The following engines are s
361361
Optional. The mode of acceleration. The following values are supported:
362362

363363
- `memory` - Store acceleration data in-memory. Not supported for Spice Cayenne (`cayenne`).
364-
- `file` - Store acceleration data in a file. Reuses any existing file on startup. Supported for Spice Cayenne (`cayenne`), `duckdb` and `sqlite` acceleration engines.
365-
- `file_create` - Always create a new acceleration file on startup, removing any existing file. When [snapshots](../../features/data-acceleration/snapshots) are enabled, the existing file is snapshotted before deletion. Supported for Spice Cayenne (`cayenne`), `duckdb` and `sqlite` acceleration engines.
366-
- `file_update` - Open an existing acceleration file if it exists, then check schema compatibility on refresh. If the source schema change is additive (new columns only), the existing file is kept. If the schema change is incompatible (columns removed, renamed, or type changed), the file is snapshotted (if [snapshots](../../features/data-acceleration/snapshots) are enabled) and recreated from scratch. Supported for Spice Cayenne (`cayenne`), `duckdb` and `sqlite` acceleration engines.
364+
- `file` - Store acceleration data in a file. Reuses any existing file on startup. Supported for Spice Cayenne (`cayenne`), `duckdb`, `sqlite`, and `turso` acceleration engines.
365+
- `file_create` - Always create a new acceleration file on startup, removing any existing file. When [snapshots](../../features/data-acceleration/snapshots) are enabled, the existing file is snapshotted before deletion. Supported for Spice Cayenne (`cayenne`), `duckdb`, `sqlite`, and `turso` acceleration engines.
366+
- `file_update` - Open an existing acceleration file if it exists, then check schema compatibility on refresh. If the source schema change is additive (new columns only), the existing file is kept. If the schema change is incompatible (columns removed, renamed, or type changed), the file is snapshotted (if [snapshots](../../features/data-acceleration/snapshots) are enabled) and recreated from scratch. Supported for Spice Cayenne (`cayenne`), `duckdb`, `sqlite`, and `turso` acceleration engines.
367367

368368
## `acceleration.snapshots`
369369

website/docs/reference/spicepod/runtime.md

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,11 @@ Use `xxh3` (the default) for its superior speed in most scenarios. Use `ahash`,
127127

128128
## `runtime.params`
129129

130-
Optional. Global key-value parameters for the runtime. HTTP-based connectors (HTTP/HTTPS, GraphQL, GitHub) support the following rate control defaults:
130+
Optional. Global key-value parameters for the runtime.
131+
132+
### HTTP Rate Control
133+
134+
HTTP-based connectors (HTTP/HTTPS, GraphQL, GitHub) support the following rate control defaults:
131135

132136
| Parameter Name | Description |
133137
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -145,6 +149,28 @@ runtime:
145149
http_requests_per_minute_limit: 200
146150
```
147151

152+
### CDC Pipeline Tuning
153+
154+
Datasets using `refresh_mode: changes` (CDC) support the following pipeline tunables. These control how change envelopes from CDC sources (e.g. PostgreSQL logical replication, Kafka, DynamoDB Streams) are buffered, coalesced, and committed.
155+
156+
| Parameter Name | Default | Range | Description |
157+
| ------------------------------ | ------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
158+
| `cdc_prefetch_buffer` | `32` | `1`–`1024` | Channel depth between the CDC source-stream reader and the apply loop. Each slot holds one decoded change envelope. |
159+
| `cdc_max_coalesced_envelopes` | `64` | `1`–`4096` | Maximum number of change envelopes coalesced into a single accelerator write. Coalescing amortizes per-envelope planning cost. |
160+
| `cdc_max_coalesced_bytes` | `67108864` | `1`–`1073741824` | Byte budget for a coalesced burst (default 64 MiB, max 1 GiB). A single envelope may exceed this; otherwise the next envelope starts a new burst. |
161+
| `cdc_commit_timeout_ms` | `30000` | `1`–`3600000` | Maximum time in milliseconds to wait for a source-side commit before logging a stall warning (default 30s, max 1hr). |
162+
163+
Out-of-range or unparseable values fall back to defaults with a warning. Environment variables (`SPICE_CDC_PREFETCH_BUFFER`, `SPICE_CDC_MAX_COALESCED_ENVELOPES`, `SPICE_CDC_MAX_COALESCED_BYTES`, `SPICE_CDC_COMMIT_TIMEOUT_MS`) are used as fallback when the corresponding `runtime.params` key is not set.
164+
165+
```yaml
166+
runtime:
167+
params:
168+
cdc_prefetch_buffer: "32"
169+
cdc_max_coalesced_envelopes: "64"
170+
cdc_max_coalesced_bytes: "67108864"
171+
cdc_commit_timeout_ms: "30000"
172+
```
173+
148174
## `runtime.functions`
149175

150176
Controls whether [functions](../../features/functions) declared in the top-level `functions:` section (and `tools:` entries with `as_sql: true`) are registered with the SQL engine. Defaults to disabled.

website/versioned_docs/version-1.10.x/components/data-connectors/delta-lake.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,7 @@ The table below shows the Delta Lake data types supported, along with the type m
179179
| `Decimal` | `Decimal128` |
180180
| `Array` | `List` |
181181
| `Struct` | `Struct` |
182+
| `Variant` | `Struct` |
182183
| `Map` | `Map` |
183184

184185
## Limitations

website/versioned_docs/version-1.10.x/features/data-acceleration/data-refresh.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@ datasets:
164164

165165
## Fast Cold Starts with Snapshots
166166

167-
File-based acceleration engines (DuckDB or SQLite) can rely on [acceleration snapshots](./snapshots) to download a pre-built database file on startup instead of waiting for the first refresh to finish. Configure a shared snapshot location under the top-level `snapshots` block and opt individual datasets in with `acceleration.snapshots: enabled`, `bootstrap_only`, or `create_only`. Snapshots are stored using Hive-style partitions (`month=YYYY-MM/day=YYYY-MM-DD/dataset=<name>`) and are only supported when each dataset writes to its own acceleration file.
167+
File-based acceleration engines (DuckDB, SQLite, or Turso) can rely on [acceleration snapshots](./snapshots) to download a pre-built database file on startup instead of waiting for the first refresh to finish. Configure a shared snapshot location under the top-level `snapshots` block and opt individual datasets in with `acceleration.snapshots: enabled`, `bootstrap_only`, or `create_only`. Snapshots are stored using Hive-style partitions (`month=YYYY-MM/day=YYYY-MM-DD/dataset=<name>`) and are only supported when each dataset writes to its own acceleration file.
168168

169169
## Filtered Refresh
170170

website/versioned_docs/version-1.10.x/features/data-acceleration/snapshots.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ datasets:
2727
2828
## Overview
2929
30-
Acceleration snapshots let Spice reuse a pre-built acceleration file on startup instead of waiting for a full refresh. When a dataset uses a file-mode acceleration engine (DuckDB or SQLite) and the local file is missing (for example on first boot or when using ephemeral NVMe storage), Spice downloads the most recent snapshot from object storage and moves the dataset straight to a ready state.
30+
Acceleration snapshots let Spice reuse a pre-built acceleration file on startup instead of waiting for a full refresh. When a dataset uses a file-mode acceleration engine (DuckDB, SQLite, or Turso) and the local file is missing (for example on first boot or when using ephemeral NVMe storage), Spice downloads the most recent snapshot from object storage and moves the dataset straight to a ready state.
3131
3232
:::info Preview
3333
Acceleration snapshots are available in preview.

0 commit comments

Comments
 (0)