recipe(cloud-connect-dev): demonstrate secrets sync against a local Postgres

The recipe gains a step where the portal delivers the password for a local
Postgres dataset: the Spicepod names `${secrets:pg_password}`, the value is set
once in the project's secrets and arrives with the deployment, and nothing about
the database password is written to the repository or to the instance's config
in plaintext.

It teaches the whole model rather than the happy path — that no `secrets:`
section is needed because the delivered store is built in at the lowest
precedence, that a local env override still wins, that a first delivery resolves
live while a rotation of an already-resolved value waits for a restart, and that
the encrypted local cache is what lets that restart succeed with the gateway
unreachable.

Author: phillipleblanc

cloud-connect-dev/.gitignore

@@ -1,3 +1,6 @@
 # Cloud Connect writes the enrolled instance identity and the cloud-managed
 # Spicepod here. Neither belongs in version control.
 .spice/
+# The demo password lives in the shell, never in a file.
+.env
+.env.local

cloud-connect-dev/README.md

@@ -4,15 +4,16 @@ Works with `v2.2+`
 
 Connect this directory to Spice Cloud with one command, watch a deployment apply to the running instance without a restart, stop it, and reconnect.
 
-The recipe is deliberately minimal: one public dataset, no infrastructure, no credentials on disk. What it demonstrates is the [Cloud Connect](https://spiceai.org/docs/deployment/cloud-connect) lifecycle on the machine you develop on.
+The recipe stays small: one public dataset, one local PostgreSQL in Docker, and no credential written to any file it tracks. What it demonstrates is the [Cloud Connect](https://spiceai.org/docs/deployment/cloud-connect) lifecycle on the machine you develop on, including how a deployment delivers the secrets its configuration needs.
 
 You will:
 
 1. Connect this directory and get a running instance attached to a new Spice Cloud project.
 2. Deploy a change that applies live, with no restart and no serving gap.
-3. Deploy a change that needs a restart, and see exactly which settings are pending.
-4. Stop with `Ctrl-C` and reconnect with `spice run` — the identity survives.
-5. Release the instance and delete its project.
+3. Deliver a secret with a deployment, and query a local PostgreSQL whose password only ever existed in your shell and in the portal.
+4. Deploy a change that needs a restart, and see exactly which settings are pending.
+5. Stop with `Ctrl-C` and reconnect with `spice run` — the identity survives.
+6. Release the instance and delete its project.
 
 ## Prerequisites
 
@@ -24,10 +25,11 @@ You will:
 
 - A Spice Cloud account with the **owner** or **admin** role in at least one organization. Those roles may enroll an instance and create a project; `member` may not.
 - Outbound HTTPS to `api.spice.ai` and to the Cloud Connect gateway.
+- Docker, for the local PostgreSQL in step 3. Steps 1, 2 and 4 onwards do not need it.
 
 No `spice login` beforehand — `spice connect` runs login inline if needed.
 
-**Cost:** a Spice Cloud project on the free tier. Step 6 deletes it.
+**Cost:** a Spice Cloud project on the free tier. Step 7 deletes it, and the database is a local container.
 
 ## 1. Connect
 
@@ -135,7 +137,125 @@ SELECT count(*) FROM recent;
   secrets:     none (the last deployment delivered no secrets)
 ```
 
-## 3. Deploy a change that needs a restart
+## 3. Deliver a secret with a deployment
+
+A deployment carries configuration **and** the secrets that configuration needs. This step connects a local PostgreSQL as a dataset whose password is never written in the Spicepod, never committed, and never typed on this machine after the first line below.
+
+Start the database. The password lives in your shell, and the container is the only thing on this machine that receives it:
+
+```bash
+export SPICE_DEMO_PG_PASSWORD="$(openssl rand -hex 16)"
+docker compose up -d
+```
+
+It seeds one table:
+
+```bash
+docker compose exec postgres psql -U spice_reader -d spice_demo -c 'SELECT count(*) FROM public.orders;'
+```
+
+```console
+ count
+-------
+     5
+(1 row)
+```
+
+Now put that same value in the portal. In the project's **Secrets**, add:
+
+| Name          | Value                                  |
+| ------------- | -------------------------------------- |
+| `pg_password` | the value of `$SPICE_DEMO_PG_PASSWORD` |
+
+Then add the dataset to the project's Spicepod and deploy:
+
+```yaml
+datasets:
+  - from: postgres:public.orders
+    name: orders
+    params:
+      pg_host: localhost
+      pg_port: "55432"
+      pg_db: spice_demo
+      pg_user: spice_reader
+      pg_pass: ${secrets:pg_password}
+      pg_sslmode: disable # a local container, so there is no TLS to verify
+```
+
+The deployment applies live — a dataset is a live section, and a secret this instance did not hold yet is installed the moment it arrives, so the dataset resolves its password as it loads:
+
+```console
+INFO Spice Cloud Connect: the deployment delivered 1 secret(s) this instance did not hold, now resolvable: pg_password
+INFO runtime::secrets_preflight: Secrets: all 1 secret reference(s) resolved.
+INFO runtime::init::dataset: Dataset orders registered (postgres:public.orders), results cache enabled. duration_ms=0
+```
+
+The PID is still the one from step 1. Query the table the portal just wired up:
+
+```bash
+spice sql
+```
+
+```sql
+SELECT customer, total_cents FROM orders ORDER BY id LIMIT 3;
+```
+
+```console
++----------+-------------+
+| customer | total_cents |
+|  varchar |    int32    |
++----------+-------------+
+| acme     | 12900       |
+| globex   | 4550        |
+| initech  | 31875       |
++----------+-------------+
+
+Time: 0.003107 seconds. 3 rows.
+```
+
+Status names what was delivered — the name only:
+
+```bash
+spice connect status
+```
+
+```console
+  deployment:  <path>/cloud-connect-dev/.spice/spicepod-cloud-managed.yml
+  secrets:     1 delivered: pg_password
+```
+
+### Why the Spicepod has no `secrets:` section
+
+`${secrets:pg_password}` resolves without declaring anything, because delivered secrets are a **built-in** store rather than one a Spicepod configures. That is what keeps the same Spicepod portable: it runs unchanged as a managed app and on this self-hosted instance, with no `secrets:` block that would be meaningless on the other.
+
+### A local value still wins
+
+The delivered store is registered at the **lowest** precedence, so any store you configure yourself — `env`, a `.env.local` file, a keyring — keeps its value for the same key. Delivery adds a source; it does not take your override away:
+
+```bash
+# The env store maps ${secrets:pg_password} to PG_PASSWORD, uppercased.
+export PG_PASSWORD="$SPICE_DEMO_PG_PASSWORD"
+```
+
+That is how you keep working against the same Spicepod with no portal round trip. Unset it before continuing, or the delivered value is never the one in use:
+
+```bash
+unset PG_PASSWORD
+```
+
+### If you deploy the dataset before setting the secret
+
+The runtime says exactly what is missing and which stores it searched, and the dataset is the only thing that fails:
+
+```console
+WARN runtime::secrets_preflight: Secrets: 1 of 1 secret reference(s) could not be resolved:
+  ✗ ${ secrets:pg_password }  dataset 'orders' (pg_pass) — not found in [env, cloud]
+ERROR runtime::init::dataset: Error initializing dataset orders. Failed to initialize data connector: Cannot connect to the dataset orders (postgres). Authentication failed.
+```
+
+Set the secret in the portal and deploy again — the instance keeps serving everything else throughout.
+
+## 4. Deploy a change that needs a restart
 
 Some sections are read only when the runtime starts. Add one in the portal — for example a `runtime` setting — and deploy again:
 
@@ -184,9 +304,24 @@ curl -s http://127.0.0.1:8090/v1/cloud-connect/status | jq
 }
 ```
 
-The PID is still unchanged, and queries still work. In the foreground there is no supervisor to restart the instance — that is step 4.
+The PID is still unchanged, and queries still work. In the foreground there is no supervisor to restart the instance — that is step 5.
+
+### A rotated secret is the same shape
+
+The first delivery of `pg_password` in step 3 applied live because nothing had resolved it yet. Rotating it is different: the components holding the old value keep it until they are built again, so a rotation is pending in exactly the way a `runtime` setting is. Change the password in the database, in the portal secret, and deploy:
+
+```bash
+docker compose exec postgres psql -U spice_reader -d spice_demo \
+  -c "ALTER ROLE spice_reader WITH PASSWORD 'the-new-value';"
+```
+
+```console
+  restart:     required for secrets
+```
+
+The instance goes on serving with the connection it already has. The restart in step 5 is what picks the new value up — and it reads it from the encrypted cache in `.spice/`, so it succeeds even with no route to Spice Cloud.
 
-## 4. Stop and reconnect
+## 5. Stop and reconnect
 
 In the foreground terminal, press `Ctrl-C`. The runtime stops. The Cloud identity is **not** released: the instance shows as offline in the portal and `.spice/identity.json` is still on disk.
 
@@ -219,7 +354,7 @@ The `restart:` line is gone.
 
 `spice connect` from this directory does the same thing. It never enrolls a second instance — an existing identity always wins.
 
-## 5. Validate locally
+## 6. Validate locally
 
 `validate.sh` checks the parts of this recipe that need no Spice Cloud account and no credentials. Run it any time:
 
@@ -229,7 +364,7 @@ The `restart:` line is gone.
 
 It asserts that the CLI is new enough, that the project-name default derives from this directory with no random fallback, that `spice connect status` reports a coherent snapshot in both output formats, that a non-interactive `spice connect` refuses instead of hanging, and that no credential or generated identity is staged for commit.
 
-## 6. Clean up
+## 7. Clean up
 
 Stop the runtime first — removal refuses while `spiced` is running — then release the instance:
 
@@ -256,9 +391,17 @@ spice connect status
 Spice Cloud Connect: not connected (<path>/cloud-connect-dev)
 ```
 
+Then stop the database and drop its volume:
+
+```bash
+docker compose down -v
+unset SPICE_DEMO_PG_PASSWORD
+```
+
 ## Notes
 
 - **Nothing is committed.** `.spice/` is gitignored: the issued identity, the cloud-managed Spicepod, and the delivered-secrets cache all live there. `validate.sh` checks it.
+- **A delivered value is never in plaintext on disk.** The cache under `.spice/` is sealed with a key derived from this instance's identity, and it is what makes a restart work without asking the control plane for the secret again. `spice connect status` reports delivered secret **names**; the values stay inside the runtime process.
 - **`spice connect` needs a terminal.** It is interactive; on a non-interactive stdin it exits and points at `spiced --token <enrollment-key>` for [headless enrollment](https://spiceai.org/docs/deployment/cloud-connect/headless).
 - **Cancelling is safe.** `Esc` or `Ctrl-C` at any prompt is a clean exit. An interrupted enrollment resumes on the next run rather than creating a duplicate instance or project.
 - **This recipe is foreground only.** To keep the instance running across reboots, install the managed service — systemd on Linux, launchd on macOS. See [Cloud Connect as a persistent service](https://spiceai.org/docs/deployment/cloud-connect/service). Windows has no managed service; there, the foreground flow in this recipe is the development story.

cloud-connect-dev/docker-compose.yml

@@ -0,0 +1,26 @@
+# A local PostgreSQL for the secrets step of the README.
+#
+# The password is never written here. It is read from the environment, and the
+# same value is what you set as the `pg_password` secret in the Spice Cloud
+# portal — so the one credential this recipe uses exists in your shell and in
+# the portal, and in no file this repository tracks.
+services:
+  postgres:
+    image: postgres:16-alpine
+    container_name: cloud-connect-dev-postgres
+    environment:
+      POSTGRES_USER: spice_reader
+      POSTGRES_DB: spice_demo
+      # `:?` fails the command with this message rather than starting a
+      # container with an empty password.
+      POSTGRES_PASSWORD: ${SPICE_DEMO_PG_PASSWORD:?export SPICE_DEMO_PG_PASSWORD first — see step 3 of the README}
+    ports:
+      # 55432, not 5432: a development machine often already has a Postgres.
+      - "55432:5432"
+    volumes:
+      - ./init:/docker-entrypoint-initdb.d:ro
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U spice_reader -d spice_demo"]
+      interval: 5s
+      timeout: 5s
+      retries: 12

cloud-connect-dev/init/01-orders.sql

@@ -0,0 +1,14 @@
+-- Seeded once, on the container's first start.
+CREATE TABLE public.orders (
+  id          integer PRIMARY KEY,
+  customer    text        NOT NULL,
+  total_cents integer     NOT NULL,
+  placed_at   timestamptz NOT NULL
+);
+
+INSERT INTO public.orders (id, customer, total_cents, placed_at) VALUES
+  (1, 'acme',    12900, '2026-01-04T09:15:00Z'),
+  (2, 'globex',   4550, '2026-01-04T11:02:00Z'),
+  (3, 'initech', 31875, '2026-01-05T14:40:00Z'),
+  (4, 'acme',      990, '2026-01-06T08:05:00Z'),
+  (5, 'umbrella', 7625, '2026-01-06T16:22:00Z');

cloud-connect-dev/validate.sh

@@ -213,6 +213,64 @@ else
 fi
 echo
 
+# --- The secrets step is internally consistent and carries no credential. ---
+#
+# None of this needs Spice Cloud: it checks that the recipe asks for the
+# password from a delivered secret rather than a literal, that the container
+# refuses to start without one, and that the three places naming the database
+# agree with each other.
+echo "Secrets sync"
+if grep -q '\${secrets:pg_password}' README.md; then
+  ok "the deployed Spicepod resolves the password from \${secrets:pg_password}"
+else
+  no "the README's Spicepod does not use \${secrets:pg_password}"
+fi
+if grep -q 'secrets:' spicepod.yaml; then
+  no "the local spicepod declares a secrets: section" "delivered secrets are a built-in store; declaring one makes the Spicepod unportable"
+else
+  ok "no secrets: section is declared — the delivered store is built in"
+fi
+if grep -qE 'POSTGRES_PASSWORD: \$\{SPICE_DEMO_PG_PASSWORD:\?' docker-compose.yml; then
+  ok "compose reads the password from the environment and fails loudly without it"
+else
+  no "compose does not read POSTGRES_PASSWORD from the environment with a :? guard"
+fi
+if grep -q 'CREATE TABLE public.orders' init/01-orders.sql; then
+  ok "the seed creates the table the README queries"
+else
+  no "init/01-orders.sql does not create public.orders"
+fi
+
+# The published port, the Spicepod's pg_port, and the psql examples must agree,
+# or the reader follows three different databases.
+# Quote-agnostic: a formatter may normalise '55432' to "55432", and a check that
+# reads only one spelling reports a mismatch that does not exist.
+compose_port="$(grep -oE '[0-9]{4,5}:5432' docker-compose.yml | head -1 | cut -d: -f1)"
+readme_port="$(grep -oE 'pg_port: .?[0-9]{4,5}' README.md | head -1 | grep -oE '[0-9]{4,5}')"
+if [ -n "$compose_port" ] && [ "$compose_port" = "$readme_port" ]; then
+  ok "compose publishes $compose_port and the Spicepod connects to it"
+else
+  no "port mismatch: compose publishes '${compose_port:-?}', the README uses '${readme_port:-?}'"
+fi
+
+if command -v docker >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then
+  if SPICE_DEMO_PG_PASSWORD=validate-only docker compose config >/dev/null 2>&1; then
+    ok "docker-compose.yml parses"
+  else
+    no "docker-compose.yml does not parse" "$(SPICE_DEMO_PG_PASSWORD=validate-only docker compose config 2>&1 | head -3)"
+  fi
+  # The guard is the point: an unset password must stop the command, not start
+  # a database that accepts an empty one.
+  if (unset SPICE_DEMO_PG_PASSWORD; docker compose config >/dev/null 2>&1); then
+    no "compose accepts an unset SPICE_DEMO_PG_PASSWORD"
+  else
+    ok "compose refuses to run with SPICE_DEMO_PG_PASSWORD unset"
+  fi
+else
+  printf '  skip docker not available; compose file not parsed\n'
+fi
+echo
+
 printf '%d passed, %d failed\n' "$pass" "$fail"
 if [ "$fail" -eq 0 ]; then
   echo "TEST PASSED"