You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add documentation for the new snapshot refresh mode that creates
read-only accelerations driven exclusively from the snapshot store.
Also remove the Cayenne single-dataset-per-spicepod snapshot limitation,
which was lifted by per-dataset metastore slices.
Source: spiceai/spiceai#10651
|`full`| Replace/overwrite the entire dataset on each refresh | A table of users |
29
+
|`append`| Append/add data to the dataset on each refresh | Append-only, immutable datasets, such as time-series or log data |
30
+
|`changes`| Apply incremental changes | Customer order lifecycle table |
31
+
|`caching`| Read-through caching for SQL queries | API search results or dynamic content endpoints |
32
+
|`snapshot`| Reload exclusively from the snapshot store | Read-only replicas bootstrapped from centralized snapshots |
32
33
33
34
Learn more about each mode:
34
35
@@ -136,6 +137,48 @@ The `caching` refresh mode is designed for HTTP-based datasets where request met
136
137
137
138
See [Caching Mode](./refresh-modes/caching) for detailed documentation and examples.
138
139
140
+
### Snapshot
141
+
142
+
The `snapshot` refresh mode creates a read-only acceleration that reloads exclusively from the [snapshot store](./snapshots). The federated data source is never queried for refreshes — instead, the runtime polls the snapshot store on a configurable interval and atomically swaps in newer snapshots when available.
143
+
144
+
```yaml
145
+
snapshots:
146
+
enabled: true
147
+
location: s3://my-bucket/snapshots/
148
+
params:
149
+
s3_auth: iam_role
150
+
151
+
datasets:
152
+
- from: postgres:public.my_table
153
+
name: my_table
154
+
acceleration:
155
+
enabled: true
156
+
engine: duckdb
157
+
mode: file
158
+
refresh_mode: snapshot
159
+
refresh_check_interval: 30s # Poll interval; defaults to 1m
160
+
snapshots: enabled
161
+
params:
162
+
duckdb_file: /nvme/my_table.db
163
+
```
164
+
165
+
**Requirements:**
166
+
167
+
- `acceleration.snapshots`must be `enabled` or `bootstrap_only`
168
+
- The acceleration engine must be a snapshot-capable file-based engine: **DuckDB**, **SQLite**, **Cayenne**, or **Turso**
169
+
170
+
**Behavior:**
171
+
172
+
- On startup, the runtime bootstraps from the most recent snapshot (same as other snapshot-enabled modes)
173
+
- After bootstrap, the runtime polls the snapshot store at `refresh_check_interval` (default: 60 seconds) for newer snapshots
174
+
- When a newer snapshot is found, its schema is validated against the current acceleration schema before downloading
175
+
- The accelerator file is swapped atomically — queries continue to be served from the previous snapshot until the swap completes
176
+
- `INSERT INTO`statements are rejected with an error since the acceleration is driven exclusively from snapshots
177
+
178
+
:::tip
179
+
Use `refresh_mode: snapshot` for read-only replicas that don't need direct access to the federated source — for example, edge nodes that receive snapshots from a centralized writer.
@@ -436,6 +436,7 @@ Optional. How to refresh the dataset. The following values are supported:
436
436
- `append`- Append new data to the dataset. When `time_column` is specified, new records are fetched from the latest timestamp in the accelerated data at the `acceleration.refresh_check_interval`.
437
437
- `changes`- Apply change data capture (CDC) events to incrementally update the dataset.
438
438
- `caching`- Cache data based on request metadata (HTTP requests). Uses row-level replacement based on cache keys. See [Caching Mode](../../features/data-acceleration/refresh-modes/caching) for details.
439
+
- `snapshot` - Reload exclusively from the [snapshot store](../../features/data-acceleration/snapshots). The federated source is never queried; the runtime polls for newer snapshots at `refresh_check_interval` (default: 60s). Requires `acceleration.snapshots: enabled` or `bootstrap_only` and a snapshot-capable file-based engine (DuckDB, SQLite, Cayenne, or Turso). Writes (`INSERT INTO`) are rejected. See [Snapshot Refresh Mode](../../features/data-acceleration/data-refresh#snapshot).
0 commit comments