Skip to content

Commit 6635a03

Browse files
y-f-udigadeesh
andauthored
add refresh append overlap doc; address some inaccurate doc issue (#284)
* add refresh append overlap doc; address some inaccurate doc issue * Update spiceaidocs/docs/reference/spicepod/datasets.md Co-authored-by: Aurash Behbahani <aurashb@gmail.com> * Update spiceaidocs/docs/reference/spicepod/datasets.md Co-authored-by: Aurash Behbahani <aurashb@gmail.com> * doc change --------- Co-authored-by: Aurash Behbahani <aurashb@gmail.com>
1 parent dd6c09f commit 6635a03

2 files changed

Lines changed: 17 additions & 3 deletions

File tree

spiceaidocs/docs/reference/spicepod/datasets.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,11 +145,11 @@ Optional. The mode of acceleration. The following values are supported:
145145
Optional. How to refresh the dataset. The following values are supported:
146146

147147
- `full` - Refresh the entire dataset.
148-
- `append` - Append new data to the dataset.
148+
- `append` - Append new data to the dataset. When `time_column` is specified, new records are fetched from the latest timestamp in the accelerated data at the `acceleration.refresh_check_interval`.
149149

150150
## `acceleration.refresh_check_interval`
151151

152-
Optional. How often data should be refreshed. Only supported for `full` refresh_mode datasets. For `append` datasets, the refresh check interval not used.
152+
Optional. How often data should be refreshed. For `append` datasets without a specific `time_column`, this config is not used.
153153

154154
See [Duration](../duration/index.md)
155155

@@ -173,6 +173,16 @@ For example, `refresh_data_window: 24h` will include only records with a timesta
173173

174174
See [Duration](../duration/index.md)
175175

176+
## `acceleration.refresh_append_overlap`
177+
178+
Optional. A duration to specify how far back to include records based on the most recent timestamp found in the accelerated data. Requires `time_column` to also be configured. Only supported for `append` refresh mode datasets.
179+
180+
This setting can help mitigate missing data issues caused by late arriving data.
181+
182+
Example: If the latest timestamp in the accelerated data table is `2020-01-01T02:00:00Z`, setting `refresh_append_overlap: 1h` will include records starting from `2020-01-01T01:00:00Z`.
183+
184+
See [Duration](../duration/index.md)
185+
176186
## `acceleration.params`
177187

178188
Optional. Parameters to pass to the acceleration engine. The parameters are specific to the acceleration engine used.

spiceaidocs/docs/reference/spicepod/index.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,11 @@ secrets:
4343
store: env
4444
```
4545
46-
## `runtime`
46+
## `runtime`
47+
48+
### `num_of_parallel_loading_at_start_up`
49+
50+
This configuration setting determines the maximum number of datasets that can be loaded in parallel during startup. This parallel loading capability accelerates Spice's startup process when multiple datasets are configured.
4751

4852
### `results_cache`
4953

0 commit comments

Comments
 (0)