Skip to content

Commit 2d2305c

Browse files
lukekimlukekim
committed
docs: add deployment guide stubs for RC+ components
Adds deployment.md stubs for all RC/Stable (GA) components per the new Deployment Guide requirement in spiceai/docs/criteria/*/stable.md. Components covered (23): - Data connectors: delta-lake, dremio, duckdb, dynamodb, file, github, graphql, mysql, postgres, s3, spiceai - Data accelerators: arrow, cayenne, duckdb, sqlite, postgres - Catalogs: unity-catalog - Models: filesystem, huggingface, openai - Embeddings: local, huggingface, openai Each stub follows the structure of the Databricks Deployment Guide with TODO markers for component owners to fill in production content: Authentication, Resilience Controls, Capacity & Sizing, Metrics, Task History, Known Limitations, and Troubleshooting. Flat-file components were migrated to the <slug>/index.md + <slug>/deployment.md subfolder layout used by Databricks. Relative .md links in moved files were updated to preserve their resolution targets.
1 parent ee611de commit 2d2305c

44 files changed

Lines changed: 1390 additions & 10 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎website/docs/components/catalogs/unity-catalog/deployment.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: 'Unity Catalog Catalog Connector Deployment Guide'
3+
sidebar_label: 'Deployment Guide'
4+
description: 'Production operating guide for the Unity Catalog catalog connector: resilience controls, authentication, metrics, and observability.'
5+
sidebar_position: 10
6+
pagination_prev: null
7+
pagination_next: null
8+
tags:
9+
- catalogs
10+
- deployment
11+
- observability
12+
---
13+
14+
Production operating guide for the **Unity Catalog Catalog Connector** covering resilience tuning, authentication, capacity sizing, metrics, and observability.
15+
16+
:::info
17+
This deployment guide is a work in progress. For a complete reference example, see the [Databricks Deployment Guide](../../data-connectors/databricks/deployment).
18+
:::
19+
20+
## Authentication & Secrets
21+
22+
Guidance for production authentication, credential rotation, and secret store integration.
23+
24+
<!-- TODO: Document supported auth methods, required IAM/roles/permissions, recommended secret store, and rotation procedures. -->
25+
26+
## Resilience Controls
27+
28+
Production resilience parameters such as concurrency limits, retry budgets, backoff, and permanent-error handling.
29+
30+
<!-- TODO: Document component-specific resilience parameters, defaults, and recommended overrides for production. -->
31+
32+
## Capacity & Sizing
33+
34+
Recommended sizing guidance (CPU, memory, disk, network) and scaling behavior under load.
35+
36+
<!-- TODO: Document per-dataset resource expectations, batch sizing, and expected throughput characteristics. -->
37+
38+
## Metrics
39+
40+
Operational metrics exposed by the catalog connector. See [Component Metrics](../../../features/observability/component_metrics) for general configuration.
41+
42+
<!-- TODO: List component metrics (counter/gauge/histogram), their meaning, and how to enable them in the spicepod. -->
43+
44+
## Task History & Tracing
45+
46+
Spans emitted by the catalog connector for the [task history](../../../reference/task_history) system.
47+
48+
<!-- TODO: List span names and input/output fields, and any trace attributes specific to this component. -->
49+
50+
## Known Limitations
51+
52+
Any production limitations, compatibility caveats, or unsupported features.
53+
54+
<!-- TODO: Document known limitations (data types, query patterns, concurrency ceilings, etc.). -->
55+
56+
## Troubleshooting
57+
58+
Common failure modes and resolutions.
59+
60+
<!-- TODO: Document common errors, diagnostic steps, and recovery procedures. -->

website/docs/components/catalogs/unity-catalog.md renamed to website/docs/components/catalogs/unity-catalog/index.md

File renamed without changes.

‎website/docs/components/data-accelerators/arrow/deployment.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: 'Arrow Data Accelerator Deployment Guide'
3+
sidebar_label: 'Deployment Guide'
4+
description: 'Production operating guide for the Arrow data accelerator: resilience controls, authentication, metrics, and observability.'
5+
sidebar_position: 10
6+
pagination_prev: null
7+
pagination_next: null
8+
tags:
9+
- data-accelerators
10+
- deployment
11+
- observability
12+
---
13+
14+
Production operating guide for the **Arrow Data Accelerator** covering resilience tuning, authentication, capacity sizing, metrics, and observability.
15+
16+
:::info
17+
This deployment guide is a work in progress. For a complete reference example, see the [Databricks Deployment Guide](../../data-connectors/databricks/deployment).
18+
:::
19+
20+
## Authentication & Secrets
21+
22+
Guidance for production authentication, credential rotation, and secret store integration.
23+
24+
<!-- TODO: Document supported auth methods, required IAM/roles/permissions, recommended secret store, and rotation procedures. -->
25+
26+
## Resilience Controls
27+
28+
Production resilience parameters such as concurrency limits, retry budgets, backoff, and permanent-error handling.
29+
30+
<!-- TODO: Document component-specific resilience parameters, defaults, and recommended overrides for production. -->
31+
32+
## Capacity & Sizing
33+
34+
Recommended sizing guidance (CPU, memory, disk, network) and scaling behavior under load.
35+
36+
<!-- TODO: Document per-dataset resource expectations, batch sizing, and expected throughput characteristics. -->
37+
38+
## Metrics
39+
40+
Operational metrics exposed by the data accelerator. See [Component Metrics](../../../features/observability/component_metrics) for general configuration.
41+
42+
<!-- TODO: List component metrics (counter/gauge/histogram), their meaning, and how to enable them in the spicepod. -->
43+
44+
## Task History & Tracing
45+
46+
Spans emitted by the data accelerator for the [task history](../../../reference/task_history) system.
47+
48+
<!-- TODO: List span names and input/output fields, and any trace attributes specific to this component. -->
49+
50+
## Known Limitations
51+
52+
Any production limitations, compatibility caveats, or unsupported features.
53+
54+
<!-- TODO: Document known limitations (data types, query patterns, concurrency ceilings, etc.). -->
55+
56+
## Troubleshooting
57+
58+
Common failure modes and resolutions.
59+
60+
<!-- TODO: Document common errors, diagnostic steps, and recovery procedures. -->

website/docs/components/data-accelerators/arrow.md renamed to website/docs/components/data-accelerators/arrow/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ See [Hash Index](../../features/data-acceleration/hash-index) for configuration
6565

6666
When accelerating a dataset using the In-Memory Arrow Data Accelerator, some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.
6767

68-
In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by [`duckdb`](./duckdb) and [`sqlite`](./sqlite) accelerators by specifying `mode: file`.
68+
In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by [`duckdb`](duckdb) and [`sqlite`](sqlite) accelerators by specifying `mode: file`.
6969

7070
:::
7171

‎website/docs/components/data-accelerators/cayenne/deployment.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: 'Cayenne Data Accelerator Deployment Guide'
3+
sidebar_label: 'Deployment Guide'
4+
description: 'Production operating guide for the Cayenne data accelerator: resilience controls, authentication, metrics, and observability.'
5+
sidebar_position: 10
6+
pagination_prev: null
7+
pagination_next: null
8+
tags:
9+
- data-accelerators
10+
- deployment
11+
- observability
12+
---
13+
14+
Production operating guide for the **Cayenne Data Accelerator** covering resilience tuning, authentication, capacity sizing, metrics, and observability.
15+
16+
:::info
17+
This deployment guide is a work in progress. For a complete reference example, see the [Databricks Deployment Guide](../../data-connectors/databricks/deployment).
18+
:::
19+
20+
## Authentication & Secrets
21+
22+
Guidance for production authentication, credential rotation, and secret store integration.
23+
24+
<!-- TODO: Document supported auth methods, required IAM/roles/permissions, recommended secret store, and rotation procedures. -->
25+
26+
## Resilience Controls
27+
28+
Production resilience parameters such as concurrency limits, retry budgets, backoff, and permanent-error handling.
29+
30+
<!-- TODO: Document component-specific resilience parameters, defaults, and recommended overrides for production. -->
31+
32+
## Capacity & Sizing
33+
34+
Recommended sizing guidance (CPU, memory, disk, network) and scaling behavior under load.
35+
36+
<!-- TODO: Document per-dataset resource expectations, batch sizing, and expected throughput characteristics. -->
37+
38+
## Metrics
39+
40+
Operational metrics exposed by the data accelerator. See [Component Metrics](../../../features/observability/component_metrics) for general configuration.
41+
42+
<!-- TODO: List component metrics (counter/gauge/histogram), their meaning, and how to enable them in the spicepod. -->
43+
44+
## Task History & Tracing
45+
46+
Spans emitted by the data accelerator for the [task history](../../../reference/task_history) system.
47+
48+
<!-- TODO: List span names and input/output fields, and any trace attributes specific to this component. -->
49+
50+
## Known Limitations
51+
52+
Any production limitations, compatibility caveats, or unsupported features.
53+
54+
<!-- TODO: Document known limitations (data types, query patterns, concurrency ceilings, etc.). -->
55+
56+
## Troubleshooting
57+
58+
Common failure modes and resolutions.
59+
60+
<!-- TODO: Document common errors, diagnostic steps, and recovery procedures. -->

website/docs/components/data-accelerators/cayenne.md renamed to website/docs/components/data-accelerators/cayenne/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ Spice Cayenne uses Vortex as its storage format, providing significant performan
2929

3030
Vortex is a Linux Foundation (LF AI & Data) project under Apache-2.0 license with neutral governance. For performance benchmarks, see [bench.vortex.dev](https://bench.vortex.dev/).
3131

32-
While [DuckDB](./duckdb) excels for datasets up to approximately 1TB, Spice Cayenne with Vortex is designed to scale beyond these limits.
32+
While [DuckDB](duckdb) excels for datasets up to approximately 1TB, Spice Cayenne with Vortex is designed to scale beyond these limits.
3333

3434
## Architecture
3535

‎website/docs/components/data-accelerators/duckdb/deployment.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: 'DuckDB Data Accelerator Deployment Guide'
3+
sidebar_label: 'Deployment Guide'
4+
description: 'Production operating guide for the DuckDB data accelerator: resilience controls, authentication, metrics, and observability.'
5+
sidebar_position: 10
6+
pagination_prev: null
7+
pagination_next: null
8+
tags:
9+
- data-accelerators
10+
- deployment
11+
- observability
12+
---
13+
14+
Production operating guide for the **DuckDB Data Accelerator** covering resilience tuning, authentication, capacity sizing, metrics, and observability.
15+
16+
:::info
17+
This deployment guide is a work in progress. For a complete reference example, see the [Databricks Deployment Guide](../../data-connectors/databricks/deployment).
18+
:::
19+
20+
## Authentication & Secrets
21+
22+
Guidance for production authentication, credential rotation, and secret store integration.
23+
24+
<!-- TODO: Document supported auth methods, required IAM/roles/permissions, recommended secret store, and rotation procedures. -->
25+
26+
## Resilience Controls
27+
28+
Production resilience parameters such as concurrency limits, retry budgets, backoff, and permanent-error handling.
29+
30+
<!-- TODO: Document component-specific resilience parameters, defaults, and recommended overrides for production. -->
31+
32+
## Capacity & Sizing
33+
34+
Recommended sizing guidance (CPU, memory, disk, network) and scaling behavior under load.
35+
36+
<!-- TODO: Document per-dataset resource expectations, batch sizing, and expected throughput characteristics. -->
37+
38+
## Metrics
39+
40+
Operational metrics exposed by the data accelerator. See [Component Metrics](../../../features/observability/component_metrics) for general configuration.
41+
42+
<!-- TODO: List component metrics (counter/gauge/histogram), their meaning, and how to enable them in the spicepod. -->
43+
44+
## Task History & Tracing
45+
46+
Spans emitted by the data accelerator for the [task history](../../../reference/task_history) system.
47+
48+
<!-- TODO: List span names and input/output fields, and any trace attributes specific to this component. -->
49+
50+
## Known Limitations
51+
52+
Any production limitations, compatibility caveats, or unsupported features.
53+
54+
<!-- TODO: Document known limitations (data types, query patterns, concurrency ceilings, etc.). -->
55+
56+
## Troubleshooting
57+
58+
Common failure modes and resolutions.
59+
60+
<!-- TODO: Document common errors, diagnostic steps, and recovery procedures. -->

website/docs/components/data-accelerators/duckdb.md renamed to website/docs/components/data-accelerators/duckdb/index.md

File renamed without changes.

‎website/docs/components/data-accelerators/postgres/deployment.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: 'PostgreSQL Data Accelerator Deployment Guide'
3+
sidebar_label: 'Deployment Guide'
4+
description: 'Production operating guide for the PostgreSQL data accelerator: resilience controls, authentication, metrics, and observability.'
5+
sidebar_position: 10
6+
pagination_prev: null
7+
pagination_next: null
8+
tags:
9+
- data-accelerators
10+
- deployment
11+
- observability
12+
---
13+
14+
Production operating guide for the **PostgreSQL Data Accelerator** covering resilience tuning, authentication, capacity sizing, metrics, and observability.
15+
16+
:::info
17+
This deployment guide is a work in progress. For a complete reference example, see the [Databricks Deployment Guide](../../data-connectors/databricks/deployment).
18+
:::
19+
20+
## Authentication & Secrets
21+
22+
Guidance for production authentication, credential rotation, and secret store integration.
23+
24+
<!-- TODO: Document supported auth methods, required IAM/roles/permissions, recommended secret store, and rotation procedures. -->
25+
26+
## Resilience Controls
27+
28+
Production resilience parameters such as concurrency limits, retry budgets, backoff, and permanent-error handling.
29+
30+
<!-- TODO: Document component-specific resilience parameters, defaults, and recommended overrides for production. -->
31+
32+
## Capacity & Sizing
33+
34+
Recommended sizing guidance (CPU, memory, disk, network) and scaling behavior under load.
35+
36+
<!-- TODO: Document per-dataset resource expectations, batch sizing, and expected throughput characteristics. -->
37+
38+
## Metrics
39+
40+
Operational metrics exposed by the data accelerator. See [Component Metrics](../../../features/observability/component_metrics) for general configuration.
41+
42+
<!-- TODO: List component metrics (counter/gauge/histogram), their meaning, and how to enable them in the spicepod. -->
43+
44+
## Task History & Tracing
45+
46+
Spans emitted by the data accelerator for the [task history](../../../reference/task_history) system.
47+
48+
<!-- TODO: List span names and input/output fields, and any trace attributes specific to this component. -->
49+
50+
## Known Limitations
51+
52+
Any production limitations, compatibility caveats, or unsupported features.
53+
54+
<!-- TODO: Document known limitations (data types, query patterns, concurrency ceilings, etc.). -->
55+
56+
## Troubleshooting
57+
58+
Common failure modes and resolutions.
59+
60+
<!-- TODO: Document common errors, diagnostic steps, and recovery procedures. -->

‎website/docs/components/data-accelerators/sqlite/deployment.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: 'SQLite Data Accelerator Deployment Guide'
3+
sidebar_label: 'Deployment Guide'
4+
description: 'Production operating guide for the SQLite data accelerator: resilience controls, authentication, metrics, and observability.'
5+
sidebar_position: 10
6+
pagination_prev: null
7+
pagination_next: null
8+
tags:
9+
- data-accelerators
10+
- deployment
11+
- observability
12+
---
13+
14+
Production operating guide for the **SQLite Data Accelerator** covering resilience tuning, authentication, capacity sizing, metrics, and observability.
15+
16+
:::info
17+
This deployment guide is a work in progress. For a complete reference example, see the [Databricks Deployment Guide](../../data-connectors/databricks/deployment).
18+
:::
19+
20+
## Authentication & Secrets
21+
22+
Guidance for production authentication, credential rotation, and secret store integration.
23+
24+
<!-- TODO: Document supported auth methods, required IAM/roles/permissions, recommended secret store, and rotation procedures. -->
25+
26+
## Resilience Controls
27+
28+
Production resilience parameters such as concurrency limits, retry budgets, backoff, and permanent-error handling.
29+
30+
<!-- TODO: Document component-specific resilience parameters, defaults, and recommended overrides for production. -->
31+
32+
## Capacity & Sizing
33+
34+
Recommended sizing guidance (CPU, memory, disk, network) and scaling behavior under load.
35+
36+
<!-- TODO: Document per-dataset resource expectations, batch sizing, and expected throughput characteristics. -->
37+
38+
## Metrics
39+
40+
Operational metrics exposed by the data accelerator. See [Component Metrics](../../../features/observability/component_metrics) for general configuration.
41+
42+
<!-- TODO: List component metrics (counter/gauge/histogram), their meaning, and how to enable them in the spicepod. -->
43+
44+
## Task History & Tracing
45+
46+
Spans emitted by the data accelerator for the [task history](../../../reference/task_history) system.
47+
48+
<!-- TODO: List span names and input/output fields, and any trace attributes specific to this component. -->
49+
50+
## Known Limitations
51+
52+
Any production limitations, compatibility caveats, or unsupported features.
53+
54+
<!-- TODO: Document known limitations (data types, query patterns, concurrency ceilings, etc.). -->
55+
56+
## Troubleshooting
57+
58+
Common failure modes and resolutions.
59+
60+
<!-- TODO: Document common errors, diagnostic steps, and recovery procedures. -->

0 commit comments

Comments
 (0)