Skip to content

Commit 00ed6fc

Browse files
phillipleblancphillipleblanc
authored
recipe(cloud-connect-dev): add a development-machine Cloud Connect tutorial (#589)
* recipe(cloud-connect-dev): add a development-machine Cloud Connect tutorial A minimal recipe named for its directory, so the project-name default that `spice connect` derives is `cloud-connect-dev` with no generated fallback. The tutorial connects the directory, deploys a change that applies live, deploys one that needs a restart and reads the pending sections from the terminal, `spice connect status`, and `/v1/cloud-connect/status`, then stops with Ctrl-C and reconnects with `spice run` on the same identity. It ends at `spice connect remove`, which deletes the project and clears local state. `validate.sh` covers the parts that need no Spice Cloud account: the CLI version, the directory-derived project name, the status snapshot in both output formats, the two status commands rendering one service object, the non-interactive refusal, the positional-key refusal, and that no credential or generated identity is committable. CI runs it on pull requests, so it works on forks; the cloud-backed steps stay in the credentialed release gate. * recipe(cloud-connect-dev): make the validation honest about CLI version and real secrets CI installs the released Spice CLI, which predates this flow, so every assertion failed for one reason and the secret scan was checking the validator's own patterns. - Exit 2 with a single explanation when the CLI is older than v2.2, instead of running assertions that cannot pass. The workflow reports that as a notice rather than a pass, so an unvalidated run never reads as green. - Scan for enrollment-key-shaped values by length and content, so a real key is caught and the documented `spice-enroll-…` placeholder is not, and match identity material on its PEM markers. Exclude this script, whose content is the patterns themselves. - Scan the recipe directory when git lists nothing, so the check cannot pass vacuously outside a git checkout. * recipe(cloud-connect-dev): the managed service covers Linux and macOS The foreground flow stays the recipe's story; the pointer to the managed service now names the platforms it runs on — systemd on Linux, launchd on macOS — and leaves Windows as the only host without one. * recipe(cloud-connect-dev): demonstrate secrets sync against a local Postgres The recipe gains a step where the portal delivers the password for a local Postgres dataset: the Spicepod names `${secrets:pg_password}`, the value is set once in the project's secrets and arrives with the deployment, and nothing about the database password is written to the repository or to the instance's config in plaintext. It teaches the whole model rather than the happy path — that no `secrets:` section is needed because the delivered store is built in at the lowest precedence, that a local env override still wins, that a first delivery resolves live while a rotation of an already-resolved value waits for a restart, and that the encrypted local cache is what lets that restart succeed with the gateway unreachable. * recipe(cloud-connect-dev): validate the recipe's files even on an older CLI The file checks — gitignore, the credential scan, and the secrets step's internal consistency — need no CLI, so they run before the version gate and always report. An older CLI now stops only the flow-specific assertions, and a broken file is a failure rather than something the skip hides. CI installs the released CLI, so this is the difference between the workflow asserting nothing about this recipe and asserting everything that does not need the unreleased flow. * recipe(cloud-connect-dev): state the check ordering as a rule, not a moment The comment explaining why the file checks come first should read the same way a year from now, so it names what the ordering guarantees rather than what it enables for the current unreleased version. * recipe(cloud-connect-dev): correct the query output and keep the delivery model The transcript now matches what `spice sql` prints, including the type row and the timing line. Step 3 also states the three properties of a delivered secret an operator has to know — no `secrets:` section, local values win, and only the first delivery applies live — plus what a missing secret looks like. * recipe(cloud-connect-dev): deploy from the project Spicepod, and name the secret as the project defines it A deployment replaces the Spicepod the instance runs, so the local one has to be copied into the project before a deployed change can reference its datasets. Secret names are matched exactly against the project's, so the recipe uses PG_PASSWORD and says which Spicepod each step edits. * recipe(cloud-connect-dev): read pending sections from the runtime output `spice connect status` prints no restart line, so steps 4 and 5 read the pending sections from the runtime's own output and confirm the restart by the instance serving the deployed value. * recipe(cloud-connect-dev): point the doc links at the Cloud Connect section The section sits under the Spice Cloud Platform page.
1 parent e1cdf8e commit 00ed6fc

8 files changed

Lines changed: 617 additions & 0 deletions

File tree

‎.github/workflows/validate-cloud-connect-dev.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
name: Validate cloud-connect-dev
2+
3+
# Deterministic checks only: the cloud-connect-dev validation needs no Spice
4+
# Cloud account and no credentials, so it runs on forks and pull requests.
5+
# The cloud-backed steps of the recipe (enrollment, project creation,
6+
# deployment) are covered by the credentialed release gate, not here.
7+
8+
on:
9+
push:
10+
branches:
11+
- trunk
12+
paths:
13+
- 'cloud-connect-dev/**'
14+
- '.github/workflows/validate-cloud-connect-dev.yml'
15+
pull_request:
16+
paths:
17+
- 'cloud-connect-dev/**'
18+
- '.github/workflows/validate-cloud-connect-dev.yml'
19+
workflow_dispatch:
20+
21+
jobs:
22+
validate:
23+
runs-on: ubuntu-latest
24+
permissions:
25+
contents: read
26+
27+
steps:
28+
- name: Checkout repository
29+
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
30+
31+
- name: Install the Spice CLI
32+
run: |
33+
curl -fsSL https://install.spiceai.org | /bin/bash
34+
echo "$HOME/.spice/bin" >> "$GITHUB_PATH"
35+
36+
# Exit 2 means the installed CLI predates the flow this recipe documents,
37+
# so nothing was asserted. Surface that as a notice rather than a pass or
38+
# a failure: the release that ships the flow turns it into a real run.
39+
- name: Validate the recipe
40+
run: |
41+
set +e
42+
./cloud-connect-dev/validate.sh
43+
code=$?
44+
set -e
45+
if [ "$code" -eq 2 ]; then
46+
echo "::notice title=cloud-connect-dev not validated::The installed Spice CLI predates v2.2; the recipe's assertions did not run."
47+
exit 0
48+
fi
49+
exit "$code"

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ Welcome to the Spice.ai OSS Cookbook—a comprehensive collection of recipes for
121121

122122
### Deployment and Installation
123123

124+
- [Cloud Connect on a Development Machine](./cloud-connect-dev/README.md) - Connect a directory to Spice Cloud with one command, deploy without restarting, and reconnect.
124125
- [Deploying to Kubernetes](./kubernetes/README.md)
125126
- [Running in Docker](./docker/README.md)
126127
- [Sidecar Deployment Architecture](./architectures/sidecar/README.md)

‎cloud-connect-dev/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Cloud Connect writes the enrolled instance identity and the cloud-managed
2+
# Spicepod here. Neither belongs in version control.
3+
.spice/
4+
# The demo password lives in the shell, never in a file.
5+
.env
6+
.env.local

‎cloud-connect-dev/README.md‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
# Cloud Connect on a Development Machine
2+
3+
Works with Spice CLI v2.2 or later.
4+
5+
Use this recipe to connect a local Spice instance to Spice Cloud. You will deploy live changes, deliver a secret, and reconnect the instance.
6+
7+
## Prerequisites
8+
9+
- A Spice Cloud organization where you are an owner or admin
10+
- Docker
11+
- Spice CLI v2.2 or later
12+
13+
Install the Spice CLI:
14+
15+
```shell
16+
curl https://install.spiceai.org | /bin/bash
17+
```
18+
19+
## 1. Connect the instance
20+
21+
Clone the cookbook and open this recipe:
22+
23+
```shell
24+
git clone https://github.com/spiceai/cookbook.git
25+
cd cookbook/cloud-connect-dev
26+
spice connect
27+
```
28+
29+
Log in to Spice Cloud when the command prompts you. Select an organization. Accept `cloud-connect-dev` as the project name.
30+
31+
If that name is in use, accept the suggested name.
32+
33+
Leave the runtime open. Open the monitor link in the output.
34+
35+
In a second terminal, check the connection:
36+
37+
```shell
38+
cd cookbook/cloud-connect-dev
39+
spice connect status
40+
```
41+
42+
## 2. Deploy a live change
43+
44+
A deployment replaces the Spicepod the instance runs. In the Spice Cloud portal, open the project Spicepod and paste in the contents of this recipe's local `spicepod.yaml`, so the deployed Spicepod still defines the `data` dataset.
45+
46+
Add this view to the project Spicepod, below the datasets:
47+
48+
```yaml
49+
views:
50+
- name: recent
51+
sql: SELECT * FROM data LIMIT 10
52+
```
53+
54+
Deploy the Spicepod. You do not have to restart the instance.
55+
56+
Query the view:
57+
58+
```shell
59+
spice sql
60+
```
61+
62+
```sql
63+
SELECT count(*) FROM recent;
64+
```
65+
66+
## 3. Deliver a secret
67+
68+
Create a password and start PostgreSQL:
69+
70+
```shell
71+
export SPICE_DEMO_PG_PASSWORD="$(openssl rand -hex 16)"
72+
docker compose up -d
73+
```
74+
75+
Confirm that PostgreSQL contains the sample data:
76+
77+
```shell
78+
docker compose exec postgres \
79+
psql -U spice_reader -d spice_demo \
80+
-c 'SELECT count(*) FROM public.orders;'
81+
```
82+
83+
In the portal, open the project's **Settings → Secrets** and add:
84+
85+
| Name | Value |
86+
| ------------- | ---------------------------------- |
87+
| `PG_PASSWORD` | Value of `$SPICE_DEMO_PG_PASSWORD` |
88+
89+
Secret names are matched exactly. A Spicepod that references a name the project does not define is rejected when you deploy it, and the portal names the closest match it holds.
90+
91+
Add this dataset to the project Spicepod in the portal — not to the local `spicepod.yaml`:
92+
93+
```yaml
94+
datasets:
95+
- from: postgres:public.orders
96+
name: orders
97+
params:
98+
pg_host: localhost
99+
pg_port: "55432"
100+
pg_db: spice_demo
101+
pg_user: spice_reader
102+
pg_pass: ${secrets:PG_PASSWORD}
103+
pg_sslmode: disable
104+
```
105+
106+
Deploy the Spicepod. Then query the PostgreSQL data:
107+
108+
```shell
109+
spice sql
110+
```
111+
112+
```sql
113+
SELECT customer, total_cents FROM orders ORDER BY id LIMIT 3;
114+
```
115+
116+
The query returns:
117+
118+
```text
119+
+----------+-------------+
120+
| customer | total_cents |
121+
| varchar | int32 |
122+
+----------+-------------+
123+
| acme | 12900 |
124+
| globex | 4550 |
125+
| initech | 31875 |
126+
+----------+-------------+
127+
```
128+
129+
Confirm the delivery:
130+
131+
```shell
132+
spice connect status
133+
```
134+
135+
```text
136+
secrets: 1 delivered: PG_PASSWORD
137+
```
138+
139+
Status reports secret names. Values stay in the runtime process.
140+
141+
Three properties of a delivered secret:
142+
143+
- The Spicepod declares no `secrets:` section. Delivered secrets are a built-in store, so the same Spicepod runs unchanged as a managed app and on a self-hosted instance.
144+
- A local value wins. The built-in store has the lowest precedence, so an `env` or `.env.local` value for `PG_PASSWORD` overrides the delivered one. Unset it to use the delivered value.
145+
- Only the first delivery applies live. Rotating a secret that a component already resolved requires a restart, like a `runtime` change.
146+
147+
If you deploy the dataset before adding the secret, the runtime names the unresolved reference and that dataset fails to load. Add the secret and deploy again.
148+
149+
## 4. Deploy a change that requires a restart
150+
151+
Add this setting to the project Spicepod:
152+
153+
```yaml
154+
runtime:
155+
task_history:
156+
captured_output: truncated
157+
```
158+
159+
Deploy the Spicepod. The runtime names the sections that need a start, in the terminal running the instance:
160+
161+
```text
162+
INFO Spice Cloud Connect: applied the deployed spicepod (1 datasets, 0 models, 0 catalogs, 1 views); runtime takes effect when this instance next starts
163+
```
164+
165+
The project in Spice Cloud reports the same pending sections. The instance continues to serve queries until you restart it.
166+
167+
## 5. Restart and reconnect
168+
169+
In the first terminal, press `Ctrl-C`. Start the instance again from the same directory:
170+
171+
```shell
172+
spice run
173+
```
174+
175+
You do not need to run `spice connect` again. The existing identity reconnects the instance.
176+
177+
The instance now serves the deployed `runtime` settings, and the deployment reports nothing pending.
178+
179+
> **Warning:** Spice Cloud invalidates the identity if the instance stays offline for more than 30 days. Enroll the instance again to reconnect it.
180+
181+
## 6. Validate the recipe
182+
183+
Run the local checks:
184+
185+
```shell
186+
./validate.sh
187+
```
188+
189+
These checks do not connect to Spice Cloud or use credentials.
190+
191+
## 7. Clean up
192+
193+
Stop the runtime. Remove the Cloud Connect instance and project:
194+
195+
```shell
196+
spice connect remove
197+
```
198+
199+
Stop PostgreSQL and delete its volume:
200+
201+
```shell
202+
docker compose down -v
203+
unset SPICE_DEMO_PG_PASSWORD
204+
```
205+
206+
## Run as a service
207+
208+
To keep the instance running after you close the terminal, see [Cloud Connect as a service](https://spiceai.org/docs/deployment/cloud/cloud-connect/service).
209+
210+
Windows does not support the managed service.
211+
212+
## Learn more
213+
214+
- [Cloud Connect](https://spiceai.org/docs/deployment/cloud/cloud-connect)
215+
- [`spice connect` reference](https://spiceai.org/docs/cli/reference/connect)

‎cloud-connect-dev/docker-compose.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# A local PostgreSQL for the secrets step of the README.
2+
#
3+
# The password is never written here. It is read from the environment, and the
4+
# same value is what you set as the `pg_password` secret in the Spice Cloud
5+
# portal — so the one credential this recipe uses exists in your shell and in
6+
# the portal, and in no file this repository tracks.
7+
services:
8+
postgres:
9+
image: postgres:16-alpine
10+
container_name: cloud-connect-dev-postgres
11+
environment:
12+
POSTGRES_USER: spice_reader
13+
POSTGRES_DB: spice_demo
14+
# `:?` fails the command with this message rather than starting a
15+
# container with an empty password.
16+
POSTGRES_PASSWORD: ${SPICE_DEMO_PG_PASSWORD:?export SPICE_DEMO_PG_PASSWORD first — see step 3 of the README}
17+
ports:
18+
# 55432, not 5432: a development machine often already has a Postgres.
19+
- "55432:5432"
20+
volumes:
21+
- ./init:/docker-entrypoint-initdb.d:ro
22+
healthcheck:
23+
test: ["CMD-SHELL", "pg_isready -U spice_reader -d spice_demo"]
24+
interval: 5s
25+
timeout: 5s
26+
retries: 12

‎cloud-connect-dev/init/01-orders.sql‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
-- Seeded once, on the container's first start.
2+
CREATE TABLE public.orders (
3+
id integer PRIMARY KEY,
4+
customer text NOT NULL,
5+
total_cents integer NOT NULL,
6+
placed_at timestamptz NOT NULL
7+
);
8+
9+
INSERT INTO public.orders (id, customer, total_cents, placed_at) VALUES
10+
(1, 'acme', 12900, '2026-01-04T09:15:00Z'),
11+
(2, 'globex', 4550, '2026-01-04T11:02:00Z'),
12+
(3, 'initech', 31875, '2026-01-05T14:40:00Z'),
13+
(4, 'acme', 990, '2026-01-06T08:05:00Z'),
14+
(5, 'umbrella', 7625, '2026-01-06T16:22:00Z');

‎cloud-connect-dev/spicepod.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
version: v1
2+
kind: Spicepod
3+
name: cloud-connect-dev
4+
5+
datasets:
6+
- from: s3://spiceai-public-datasets/hive_partitioned_data/
7+
name: data
8+
params:
9+
file_format: parquet

0 commit comments

Comments
 (0)