busbar 0.2.8: make the chart actually boot on busbar 1.5.3

Chart 0.2.7 / appVersion 1.5.3 could not boot by any documented route.
Each defect below was reproduced by running getbusbar/busbar:1.5.3 with the
chart's own rendered config mounted read-only at /etc/busbar/config.yaml as
uid 65532 under a read-only rootfs, then re-verified green the same way and
on a real kind cluster with ct lint + ct install.

1. Manage config.locked / config.overlay (new `configMutability`).
   1.5.3 refuses to boot a mutable config whose overlay backend is not
   writable, and that backend defaults to /etc/busbar/busbar-overlay.json,
   a read-only ConfigMap subPath mount. Default mode is `locked`: a Helm
   release is a GitOps deployment and it is the only mode that is correct
   at replicaCount > 1. `overlay` and `none` are documented alternatives,
   with the ephemerality / replica-divergence tradeoff spelled out.

2. `config` is now required; an empty one fails the render with the minimal
   1.5.x config inline, instead of rendering a listener stub that exits 1
   with "missing field `providers`". Both READMEs no longer claim a bare
   install boots.

3. Every config example is busbar 1.5.x syntax: secret references
   (api_key: { env: VAR }), an identity-providers admin-tokens definition
   referenced by name from auth.admin_auth, and pool members keyed on
   `model:` rather than the retired `target:`.

4. governance.enabled no longer emits the dissolved top-level `governance:`
   block. It renders identity-providers.admin-tokens + auth.admin_auth +
   store. New governance.store.module defaults to `memory` because every
   durable store in 1.5.x is a signed store plugin and the 1.5.3 image
   ships none; a non-memory module without config.plugins fails the render.

5. adminTLS / dataTLS emit 1.5.x secret references. cert_file / key_file /
   client_ca_file are now cert: { file } / key: { file } / client_ca: { file };
   the old fields made every TLS-enabled install exit 1.

Also: the bundled helm test now retries (it could fire before the Service
endpoint propagated and report a healthy gateway as broken), and
ci/overlay-values.yaml covers the overlay mode in ct install.

appVersion stays at 1.5.3.

Author: MattJackson

README.md

@@ -9,9 +9,14 @@ The official Helm chart repository for [busbar](https://getbusbar.com), a Rust L
 ```console
 helm repo add busbar https://getbusbar.github.io/helm-charts
 helm repo update
-helm install my-busbar busbar/busbar
+helm install my-busbar busbar/busbar -f my-values.yaml
 ```
 
+`config` is **required** and has no default. busbar is a gateway in front of upstream LLM
+providers, so there is no meaningful zero-config install: with no `providers` map the binary
+exits 1 at boot. `helm install` with no `config` therefore **fails the render** and prints the
+minimal config you need, rather than handing you a `CrashLoopBackOff`.
+
 See the [chart README](charts/busbar/README.md) for the full values reference and examples.
 
 ## The two-listener model
@@ -23,20 +28,34 @@ busbar always runs **two separate listeners**:
 - **Admin plane** (`admin_listen`, default loopback `127.0.0.1:8081`) — the runtime admin API.
 
 busbar enforces a **boot-guard**: a non-loopback `admin_listen` *refuses to boot* unless the admin
-listener requires **mTLS** (`admin_tls` with a `client_ca_file`) **or** an explicit `admin_insecure:
+listener requires **mTLS** (`admin_tls` with a `client_ca`) **or** an explicit `admin_insecure:
 true` waiver is set.
 
-The chart is secure by default: the admin plane stays on loopback and is **not** exposed, so
-`helm install my-busbar busbar/busbar` always boots. To expose the admin plane (`service.admin.enabled=true`)
+The chart is secure by default: the admin plane stays on loopback and is **not** exposed, so the
+admin boot-guard is always satisfied. To expose the admin plane (`service.admin.enabled=true`)
 you must also enable `adminTLS.enabled=true` (mTLS, ideally via cert-manager) or `adminInsecure=true`,
 or the chart fails the render with a clear message.
 
+## Config mutability
+
+busbar 1.5.3 refuses to boot a *mutable* config (`config.locked: false`, its default) whose overlay
+backend is not writable, and that backend defaults to `busbar-overlay.json` next to the resolved
+`config.yaml`. This chart mounts `config.yaml` from a read-only ConfigMap under
+`readOnlyRootFilesystem`, so the chart manages that setting for you: `configMutability.mode`
+defaults to `locked` (a Helm release is a GitOps deployment; you change config with
+`helm upgrade`). Set `configMutability.mode=overlay` to re-enable admin-API config mutation, and
+read the tradeoff in `values.yaml` first: the default overlay path is pod-local and ephemeral.
+
 ## Governance
 
-With `governance.enabled=true`, busbar owns a single-writer SQLite DB, so the chart deploys a
-**StatefulSet with a PVC and `replicas: 1`**. Horizontal scale of a shared SQLite governance store is
+With `governance.enabled=true`, busbar owns single-writer state, so the chart deploys a
+**StatefulSet with a PVC and `replicas: 1`**. Horizontal scale of a shared single-writer store is
 not supported. Stateless (no-governance) deployments use a Deployment and support the HPA.
 
+busbar 1.5.x moved every durable store behind a signed **store** plugin, and the
+`getbusbar/busbar:1.5.3` image ships no plugin tarballs, so `governance.store.module` defaults to
+`memory` (ephemeral). Naming any other module without also wiring `config.plugins` fails the render.
+
 ## Contributing
 
 Charts live under `charts/`. Pull requests are linted and smoke-tested with

charts/busbar/CHANGELOG.md

@@ -1,5 +1,53 @@
 # Changelog
 
+## 0.2.8
+
+Chart 0.2.7 / appVersion 1.5.3 could not boot by any documented route. Three defects, all
+reproduced by running `getbusbar/busbar:1.5.3` with the chart's own rendered config mounted
+read-only at the chart's mount path as the chart's uid (65532), plus a fourth found on the way.
+
+- **The chart now manages `config.locked` / `config.overlay` (`configMutability`).** busbar 1.5.3
+  refuses to boot a mutable config (`config.locked: false`, its default) whose overlay backend is
+  not writable, and that backend defaults to `busbar-overlay.json` next to the resolved
+  `config.yaml`, i.e. `/etc/busbar/busbar-overlay.json` - a read-only ConfigMap subPath mount under
+  `readOnlyRootFilesystem: true`. Every install, including a correct one, died with
+  `[error] config is mutable (config.locked: false) but the overlay backend ... is not writable`.
+  New `configMutability.mode` defaults to **`locked`**: a Helm release is a GitOps deployment, the
+  ConfigMap is the source of truth, and locked is the only mode that behaves correctly at
+  `replicaCount > 1`. `mode: overlay` re-enables admin-API config mutation against the chart's
+  writable `/tmp` emptyDir, with the ephemerality and replica-divergence tradeoff spelled out in
+  `values.yaml` and the README. `mode: none` hands the keys back to you. `.Values.config.config`
+  always wins over the chart's value.
+- **`config` is now required, and an empty one fails the render.** `helm install my-busbar
+  busbar/busbar` with no values rendered a two-line listener stub that busbar rejects with
+  `[error] config.yaml: invalid YAML: missing field 'providers'`, while both READMEs claimed the
+  defaults "always boot". There is no honest bootable zero-config default for a gateway that
+  proxies upstream providers, so the chart fails fast with the minimal 1.5.x config inline in the
+  error, and both READMEs were corrected.
+- **Every config example is now busbar 1.5.x syntax.** The chart README's "Minimal" example was a
+  1.x config that 1.5.3 hard-rejects (`auth.client_tokens:`, `providers.openai.api_key_env:`).
+  Both READMEs, the `values.yaml` comments and all `ci/` fixtures now use secret references
+  (`api_key: { env: VAR }`), an `identity-providers` admin-tokens definition referenced by name
+  from `auth.admin_auth`, and pool members keyed on `model:` rather than the retired `target:`.
+- **`governance.enabled=true` no longer renders a config busbar refuses to boot.** It emitted a
+  top-level `governance:` block, which 1.5.3 flags as a 1.x marker
+  (`governance: block (dissolved into store / rate_card / per_request_fee / groups / advanced /
+  auth)`). It now renders `identity-providers.admin-tokens` + `auth.admin_auth: [admin-tokens]` +
+  `store:`. New `governance.store.module` defaults to `memory` because every durable store in
+  1.5.x is a signed store plugin and the 1.5.3 image ships none; naming another module without
+  wiring `config.plugins` fails the render with the exact error busbar would have printed.
+- **`adminTLS`/`dataTLS` now emit 1.5.x secret references.** Caught by `ct install` on a real kind
+  cluster: the chart still rendered the 1.4.x plaintext path fields, so every TLS-enabled install
+  died with `[error] config.yaml: invalid YAML: admin_tls: unknown field 'cert_file', expected one
+  of 'cert', 'key', 'client_ca'`. `cert_file`/`key_file`/`client_ca_file` are now
+  `cert: { file: ... }` / `key: { file: ... }` / `client_ca: { file: ... }`.
+- Added `ci/overlay-values.yaml` so `configMutability.mode=overlay` is covered by `ct install`.
+- The bundled `helm test` now retries. It could fire before the data Service endpoint had
+  propagated and fail instantly with `curl: (7) ... Could not connect to server`, reporting a
+  healthy gateway as broken.
+
+`appVersion` stays at `1.5.3`.
+
 ## 0.2.3
 
 Fixes an Artifact Hub scan failure (`image not found (package busbar:0.2.2)`) and a real

charts/busbar/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: busbar
 description: A production-grade Helm chart for busbar, a Rust LLM gateway with a data plane and a separate, loopback-by-default, mTLS-gated admin plane.
 type: application
-version: 0.2.7
+version: 0.2.8
 appVersion: "1.5.3"
 kubeVersion: ">=1.24.0-0"
 home: https://getbusbar.com

charts/busbar/README.md

@@ -18,26 +18,82 @@ guard — see [Admin plane](#admin-plane) below.
 ```console
 helm repo add busbar https://getbusbar.github.io/helm-charts
 helm repo update
-helm install my-busbar busbar/busbar
+helm install my-busbar busbar/busbar -f my-values.yaml
 ```
 
-The defaults render a secure, bootable deployment: the admin plane stays on loopback and is not
-exposed, so the gateway always boots. Supply a provider key and a client token to make it useful:
+`config` is **required and has no default.** busbar is a gateway in front of upstream LLM
+providers, and it exits 1 at boot without a `providers` map
+(`[error] config.yaml: invalid YAML: missing field 'providers'`), so there is no honest
+zero-config default to ship: any "bootable" one would mean inventing credentials for an upstream
+nobody configured. Installing with no `config` therefore **fails the render** and prints the
+minimal config you need, instead of giving you a `CrashLoopBackOff`.
+
+Everything else is secure by default: the admin plane stays on loopback and is not exposed, so the
+admin boot-guard is satisfied out of the box.
 
 ```console
-helm install my-busbar busbar/busbar \
-  --set-string secrets.data.OPENAI_API_KEY=sk-... \
-  --set-string secrets.data.BUSBAR_CLIENT_TOKEN=my-client-token \
-  --set 'config.auth.chain[0]=client-tokens' \
-  --set 'config.auth.client_tokens[0]=${BUSBAR_CLIENT_TOKEN}' \
-  --set config.providers.openai.api_key_env=OPENAI_API_KEY \
-  --set config.models.gpt-4o.provider=openai \
-  --set config.models.gpt-4o.max_concurrent=8
+helm install my-busbar busbar/busbar -f - <<'EOF'
+secrets:
+  data:
+    ANTHROPIC_KEY: sk-ant-...
+    BUSBAR_ADMIN_TOKEN: a-long-random-admin-token
+config:
+  identity-providers:
+    admin-tokens:
+      module: admin-tokens
+      token: { env: BUSBAR_ADMIN_TOKEN }
+  auth:
+    chain: []
+    admin_auth: [admin-tokens]
+  providers:
+    anthropic:
+      api_key: { env: ANTHROPIC_KEY }
+  models:
+    claude:
+      provider: anthropic
+  pools:
+    default:
+      members:
+        - model: claude
+EOF
 ```
 
 Secrets are injected as environment variables from a Kubernetes Secret and referenced from the
-config with `${VAR}` interpolation. The provider catalog ships inside the image at
-`/etc/busbar/providers.yaml`; only set `providersCatalog` to override it.
+config by **secret reference** (`api_key: { env: ANTHROPIC_KEY }`) - busbar 1.5.x removed the old
+`*_env` string fields. The provider catalog ships inside the image at `/etc/busbar/providers.yaml`;
+only set `providersCatalog` to override it.
+
+`auth.chain: []` is an **open relay** and is for a first boot only; configure a real auth chain
+before you expose the data plane.
+
+## Config mutability
+
+busbar 1.5.3 **refuses to boot** a mutable config (`config.locked: false`, its default) whose
+overlay backend is not writable, and that backend defaults to `busbar-overlay.json` next to the
+resolved `config.yaml` - here, `/etc/busbar/busbar-overlay.json`, which is a read-only ConfigMap
+mount under `readOnlyRootFilesystem: true`:
+
+```
+[error] config is mutable (config.locked: false) but the overlay backend
+'/etc/busbar/busbar-overlay.json' is not writable (is the config directory read-only?).
+```
+
+So the chart manages this for you via `configMutability.mode`:
+
+| mode | renders | meaning |
+|---|---|---|
+| `locked` (default) | `config: { locked: true }` | The ConfigMap is the source of truth. The admin API refuses **config mutations** (it still serves keys, stats, `/config/reload` and the rest); you change config with `helm upgrade`, and `reloadOnConfigChange` rolls the pods. |
+| `overlay` | `config: { locked: false, overlay: { file: <overlayFile> } }` | Admin-API config mutation works again, persisted to `configMutability.overlayFile` (default `/tmp/busbar-overlay.json`, on the chart's emptyDir). |
+| `none` | nothing | You own `config.locked` / `config.overlay` inside `.Values.config`. |
+
+`locked` is the default because a Helm release **is** a GitOps deployment, and because it is the
+only mode that is correct at `replicaCount > 1`. In `overlay` mode with the default path the
+overlay lives on a per-pod `emptyDir`: mutations are lost on restart, on `helm upgrade` and on any
+reschedule, and with more than one replica **each replica keeps its own overlay and the replicas
+will diverge**. Use `overlay` at `replicaCount: 1`, or point `overlayFile` at durable storage that
+every replica mounts read-write.
+
+Anything you set under `.Values.config.config` wins over the chart's value in every mode.
 
 ## Admin plane
 
@@ -52,10 +108,22 @@ If you enable the admin Service without either, the chart **fails the render** w
 
 ## Governance
 
-When `governance.enabled=true`, busbar owns a single-writer SQLite DB — per-replica state. The chart
-switches to a **StatefulSet with a PVC and `replicas: 1`**. **Horizontal scale of a shared SQLite
-governance store is not supported.** Stateless (no governance) deployments use a Deployment and may
-scale out (HPA supported).
+When `governance.enabled=true`, busbar owns single-writer state, so the chart switches to a
+**StatefulSet with a PVC and `replicas: 1`**. **Horizontal scale of a shared single-writer store is
+not supported.** Stateless (no governance) deployments use a Deployment and may scale out (HPA
+supported).
+
+busbar 1.5.x dissolved the old top-level `governance:` config block (it is now a hard 1.x marker
+the binary refuses to boot) into `store:` plus an admin-tokens identity provider referenced from
+`auth.admin_auth`. The chart renders that shape for you.
+
+Every durable store in 1.5.x is a signed **store** plugin, and the `getbusbar/busbar:1.5.3` image
+ships no plugin tarballs (`busbar --list-plugins` prints `no plugin tarballs found`), so
+`governance.store.module` defaults to `memory`: keys, group usage and ledgers are **ephemeral** and
+reset on restart. To get durability, mount the signed store plugin into the pod yourself, set
+`config.plugins.enabled: true` and `config.plugins.dir`, then set `governance.store.module`. The
+chart fails the render if you name a non-`memory` module without wiring the plugin, because busbar
+would otherwise exit 1.
 
 ## Examples
 
@@ -64,37 +132,59 @@ scale out (HPA supported).
 ```yaml
 secrets:
   data:
-    OPENAI_API_KEY: sk-...
-    BUSBAR_CLIENT_TOKEN: my-client-token
+    ANTHROPIC_KEY: sk-ant-...
+    BUSBAR_ADMIN_TOKEN: a-long-random-admin-token
 config:
+  identity-providers:
+    admin-tokens:
+      module: admin-tokens
+      token: { env: BUSBAR_ADMIN_TOKEN }
   auth:
-    chain: ["client-tokens"]
-    client_tokens: ["${BUSBAR_CLIENT_TOKEN}"]
+    chain: []                 # open relay; set a real chain before exposing the data plane
+    admin_auth: [admin-tokens]
   providers:
-    openai:
-      api_key_env: OPENAI_API_KEY
+    anthropic:
+      api_key: { env: ANTHROPIC_KEY }
   models:
-    gpt-4o:
-      provider: openai
-      max_concurrent: 8
+    claude:
+      provider: anthropic
+  pools:
+    default:
+      members:
+        - model: claude       # `model:`, not the retired 1.4.x `target:`
 ```
 
 ### With governance
 
-Governance requires an admin token — put it in `secrets.data` under the key named by
+Governance requires an admin token: put it in `secrets.data` under the key named by
 `governance.adminTokenEnv` (default `BUSBAR_ADMIN_TOKEN`); the chart wires
-`governance.admin_token` for you. (`helm install` fails fast if it is missing.)
+`identity-providers.admin-tokens` and `auth.admin_auth` for you. (`helm install` fails fast if it
+is missing.)
 
 ```yaml
 governance:
   enabled: true
+  store:
+    module: memory            # see the note above about durable store plugins
   dbPath: /var/lib/busbar/governance.db
   persistence:
     size: 5Gi
     storageClass: fast-ssd
 secrets:
   data:
+    ANTHROPIC_KEY: sk-ant-...
     BUSBAR_ADMIN_TOKEN: super-secret-admin-token
+config:
+  providers:
+    anthropic:
+      api_key: { env: ANTHROPIC_KEY }
+  models:
+    claude:
+      provider: anthropic
+  pools:
+    default:
+      members:
+        - model: claude
 ```
 
 ### With ingress (data plane)
@@ -156,14 +246,18 @@ networkPolicy:
 | `nodeSelector` | Node selector | `{}` |
 | `tolerations` | Tolerations | `[]` |
 | `affinity` | Affinity | `{}` |
-| `config` | Structured map rendered to `config.yaml` (chart injects `listen`/`admin_listen`) | `{}` |
+| `config` | **Required.** Structured map rendered to `config.yaml` (chart injects `listen`/`admin_listen`) | `{}` (fails the render) |
+| `configMutability.mode` | `locked` / `overlay` / `none` (see [Config mutability](#config-mutability)) | `locked` |
+| `configMutability.overlayFile` | Overlay backend path, `overlay` mode only. Pod-local and ephemeral by default | `/tmp/busbar-overlay.json` |
 | `providersCatalog` | Optional map rendered to `providers.yaml` (else the image's catalog is used) | `{}` |
 | `existingConfigMap` | Use an existing ConfigMap instead of rendering config | `""` |
 | `secrets.create` | Create a Secret from `secrets.data` | `true` |
 | `secrets.existingSecret` | Use an existing Secret | `""` |
 | `secrets.data` | Key/value secret data injected as env vars | `{}` |
 | `governance.enabled` | Enable governance (StatefulSet + PVC, replicas=1) | `false` |
-| `governance.dbPath` | Governance SQLite DB path | `/var/lib/busbar/governance.db` |
+| `governance.adminTokenEnv` | Secret key holding the admin token | `BUSBAR_ADMIN_TOKEN` |
+| `governance.store.module` | `store:` module. Non-`memory` needs a signed store plugin you supply | `memory` |
+| `governance.dbPath` | Governance DB path (`store.settings.db_path`, non-`memory` only) | `/var/lib/busbar/governance.db` |
 | `governance.persistence.size` | Governance PVC size | `1Gi` |
 | `governance.persistence.storageClass` | Governance PVC storage class | `""` |
 | `governance.persistence.accessMode` | Governance PVC access mode | `ReadWriteOnce` |

charts/busbar/ci/admin-mtls-values.yaml

@@ -1,18 +1,31 @@
 # CI: exposed admin plane with cert-manager mTLS. Requires cert-manager + a
 # ClusterIssuer named "busbar-ca" in the cluster (the workflow installs both).
 config:
+  identity-providers:
+    admin-tokens:
+      module: admin-tokens
+      token:
+        env: BUSBAR_ADMIN_TOKEN
+  auth:
+    chain: []
+    admin_auth: [admin-tokens]
   providers:
     anthropic:
-      api_key_env: ANTHROPIC_KEY
+      api_key:
+        env: ANTHROPIC_KEY
   models:
     claude-sonnet:
       provider: anthropic
       max_concurrent: 20
-      max_requests: -1
+  pools:
+    default:
+      members:
+        - model: claude-sonnet
 secrets:
   create: true
   data:
     ANTHROPIC_KEY: ci-dummy-anthropic-key
+    BUSBAR_ADMIN_TOKEN: ci-dummy-admin-token
 service:
   admin:
     enabled: true