Skip to content

Commit f7c6d94

Browse files
phillipleblancpeasee
authored andcommitted
v1.10.2: Add snapshots_create_interval acceleration parameter (#1282)
* localpod caching * docs: Add snapshots_create_interval acceleration parameter Document the new acceleration.params.snapshots_create_interval parameter that schedules periodic snapshot creation independent of refresh cycles. This is useful for datasets using refresh_mode: caching where row-level changes occur frequently but traditional refresh cycles do not apply. Updates: - Reference documentation in datasets.md - Feature documentation in snapshots.md * Update website/docs/features/data-acceleration/snapshots.md Co-authored-by: William <98815791+peasee@users.noreply.github.com> --------- Co-authored-by: William <98815791+peasee@users.noreply.github.com>
1 parent 7c28a6e commit f7c6d94

3 files changed

Lines changed: 45 additions & 1 deletion

File tree

website/docs/components/data-connectors/localpod.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ The dataset created by the `localpod` connector will logically have the same dat
1111

1212
## Synchronized Refreshes
1313

14-
The `localpod` connector supports synchronized refreshes, which ensures that the child dataset is refreshed from the same data as the parent dataset. Synchronized refreshes require that both the parent and child datasets are accelerated with `refresh_mode: full` (which is the default).
14+
The `localpod` connector supports synchronized refreshes, which ensures that the child dataset is refreshed from the same data as the parent dataset. Synchronized refreshes require that both the parent and child datasets are accelerated with `refresh_mode: full` (which is the default) or `refresh_mode: caching`.
1515

1616
When synchronization is enabled, the following logs will be emitted:
1717

website/docs/features/data-acceleration/snapshots.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ Acceleration snapshots are available in preview.
3939
- If the file is missing and snapshots are enabled, Spice looks under the configured snapshot location and downloads the newest snapshot for that dataset.
4040
- If no snapshot is available, the acceleration boots empty and refreshes from the source.
4141
- After each refresh, Spice writes a new snapshot unless the dataset is configured in `bootstrap_only` mode.
42+
- When `snapshots_create_interval` is configured, Spice writes additional periodic snapshots at the specified interval, independent of refresh cycles.
4243

4344
Snapshots are organized with Hive-style partitioning so they are easy to retain and prune. For a dataset named `my_dataset`, Spice writes files such as:
4445

@@ -82,6 +83,24 @@ Each dataset opts into snapshotting through the `acceleration.snapshots` field.
8283
- `create_only` – write new snapshots after refreshes, but never download them on startup.
8384
- `disabled` – disable snapshot usage for this dataset. (Default.)
8485

86+
### Periodic snapshot creation
87+
88+
By default, snapshots are created after each data refresh. For datasets using `refresh_mode: caching`, where row-level changes occur frequently but traditional refresh cycles do not apply, configure `acceleration.params.snapshots_create_interval` to write snapshots at a fixed interval:
89+
90+
```yaml
91+
acceleration:
92+
enabled: true
93+
engine: duckdb
94+
mode: file
95+
refresh_mode: caching
96+
snapshots: enabled
97+
params:
98+
snapshots_create_interval: 60s # Write a snapshot every 60 seconds
99+
duckdb_file: /nvme/my_dataset.db
100+
```
101+
102+
This ensures snapshot persistence even when no explicit refresh triggers occur.
103+
85104
Example Spicepod configuration:
86105

87106
```yaml

website/docs/reference/spicepod/datasets.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,31 @@ datasets:
402402

403403
See [Caching Mode](../../features/data-acceleration/refresh-modes/caching.md#stale-if-error-behavior) for detailed behavior.
404404

405+
## `acceleration.params.snapshots_create_interval`
406+
407+
Optional. The interval at which snapshots are created for the accelerated dataset. Only applicable when `acceleration.snapshots` is set to `enabled` or `create_only`. Defaults to creating snapshots only after each refresh.
408+
409+
When configured, Spice writes a new snapshot at the specified interval in addition to writing snapshots after refreshes. This is useful for datasets that use `refresh_mode: caching` where there is no refresh interval.
410+
411+
**Example**:
412+
413+
```yaml
414+
datasets:
415+
- from: https://api.example.com
416+
name: my_data
417+
acceleration:
418+
enabled: true
419+
engine: duckdb
420+
mode: file
421+
refresh_mode: caching
422+
snapshots: enabled
423+
params:
424+
snapshots_create_interval: 60s # Create a snapshot every 60 seconds
425+
duckdb_file: /nvme/my_data.db
426+
```
427+
428+
See [Duration](../duration/index.md) and [Acceleration Snapshots](../../features/data-acceleration/snapshots.md) for more details.
429+
405430
## `acceleration.refresh_sql`
406431

407432
Optional. Filters the data fetched from the source to be stored in the accelerator engine. Only supported for `full` refresh_mode datasets.

0 commit comments

Comments
 (0)