| description | Connect a Kubernetes cluster in your own cloud account to Spice.ai Cloud with the operator's managed mode. |
|---|---|
| icon | cloud |
Bring-Your-Own-Cluster (BYOC) connects a Kubernetes cluster that runs in your own cloud account to Spice.ai Cloud. You install the Spice.ai Kubernetes Operator in your cluster and enable managed mode. Spice.ai Cloud then deploys and observes Spicepods in your cluster, while your data stays in your account.
{% hint style="info" %} BYOC is available on the Enterprise plan. Registering, decommissioning, and issuing enrollment tokens for a cluster requires the organization Admin or Owner role. {% endhint %}
Managed mode is a remote-control channel, not a data channel:
- The operator opens one outbound connection from your cluster to the Spice.ai Cloud gateway. It is a long-lived, bidirectional gRPC stream secured with mutual TLS (mTLS).
- Spice.ai Cloud does not open any inbound connection to your cluster. You do not expose an endpoint, open a firewall port, or share cluster credentials.
- The operator stays the source of truth for the
SpicepodSetandSpicepodClusterresources in your cluster. Spice.ai Cloud sends commands (apply, delete, pause, drain, get status, get logs) over the stream; the operator applies them locally. - Query traffic and data do not flow through the stream. The stream carries only commands, command results, heartbeats, telemetry roll-ups, and metrics.
The operator makes three kinds of outbound connections:
| Purpose | Protocol | Destination | Port |
|---|---|---|---|
| Enrollment (once) | HTTPS POST |
https://api.spice.ai/v1/cloud-connect/enroll |
443 |
| Identity renewal | HTTPS POST |
https://api.spice.ai/v1/cloud-connect/renew |
443 |
| Control stream | gRPC over mTLS | connect.<region>-aws.spiceai.io (assigned during enrollment) |
443 |
The cluster also pulls the operator chart and the Spice runtime images from the AWS Marketplace registry, 709825985650.dkr.ecr.us-east-1.amazonaws.com. Allow egress to all of these hosts on port 443.
The operator gets its identity by enrollment:
- At install, the operator holds only a single-use enrollment token (
spice-enroll-…) that you mint in the Spice.ai Cloud portal. - On first start, the operator generates a private key and a certificate signing request (CSR) locally. It sends the CSR and the token to the enrollment endpoint. The private key never leaves the cluster.
- Spice.ai Cloud returns a signed mTLS certificate, the CA bundle, and the gateway address. The operator stores them in a Secret it owns (
spice-byoc-operator-identityby default). Do not create or edit this Secret yourself. - The operator connects to the gateway with the new identity. Each later restart reuses the stored identity; enrollment happens only once.
The identity certificate is valid for 24 hours. The operator renews it automatically at half-life (about every 12 hours) with a proof-of-possession request, and rotates its keys on every renewal. If renewal fails, a 30-day grace period applies during which renewal still works. Past the grace period, or when Spice.ai Cloud terminally rejects the identity (for example, after you decommission the cluster), the operator stops managed mode and needs a fresh enrollment token.
- A Spice.ai Cloud organization on the Enterprise plan, and the Admin or Owner role in it.
- An AWS account subscribed to the AWS Marketplace Spice.ai Enterprise listing. Accounts that are not subscribed can log in to the registry but the image pull is refused.
- A Kubernetes cluster (1.33+) with storage and IAM already provisioned — Spice.ai Cloud does not create cloud infrastructure in your account.
kubectl,helm(3.x; 3.17+ recommended for versionedoci://chart references), and theawsCLI.- Egress to the endpoints listed above on port 443.
- Operator chart version
1.1.0-operator-helmor later (the first release with BYOC support).
- Open the Spice.ai Cloud portal and go to Clusters → BYOC.
- Select Add a cluster.
- On the Details step, enter:
- Cluster name — for example
production-us-east. - AWS account ID (optional) — the 12-digit account the cluster runs in.
- AWS region — any AWS region. One cluster per AWS account + region pair; multi-region clusters are not yet supported.
- Cluster name — for example
- Select Continue. The portal creates the cluster record and shows the Launch in AWS step with:
- The enrollment token (
spice-enroll-…). - The cluster ID and region.
- A ready-to-run Helm command.
- The enrollment token (
{% hint style="warning" %} Copy the enrollment token now — it cannot be retrieved later. The token is shown once, is single-use, and expires after 1 hour by default. Anyone with the token can register an operator as this cluster, so treat it like a secret. If the token expires or is lost before the operator enrolls, issue a new one from the cluster's Setup tab (this revokes the old token). {% endhint %}
Alternatively, create a cluster and token through the management API: POST /v1/cloud-connect/token with { "kind": "cluster", "name": "...", "region": "..." }, authenticated as an organization admin with the clusters:write scope.
Authenticate Helm to the AWS Marketplace registry, then install the operator with managed mode enabled. Read the token into an environment variable first so it does not enter your shell history:
read -s SPICE_ENROLLMENT_TOKEN # paste the enrollment token
aws ecr get-login-password --region us-east-1 | \
helm registry login --username AWS --password-stdin \
709825985650.dkr.ecr.us-east-1.amazonaws.com
helm upgrade --install spiceai-operator \
oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/spice-ai/spiceai-enterprise-plan \
--version 1.1.0-operator-helm \
--namespace spiceai-operator-system --create-namespace \
--set spice.managedMode.enabled=true \
--set-string spice.managedMode.enrollmentToken="$SPICE_ENROLLMENT_TOKEN"{% hint style="info" %}
A value passed through spice.managedMode.enrollmentToken is persisted in Helm's release history in plaintext. The token is single-use and short-lived, which limits the exposure — but for production, pre-create the Secret yourself and reference it with enrollmentTokenSecret so the token never enters Helm state:
kubectl create namespace spiceai-operator-system
kubectl -n spiceai-operator-system create secret generic spice-enrollment-token \
--from-literal=token="$SPICE_ENROLLMENT_TOKEN"
helm upgrade --install spiceai-operator \
oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/spice-ai/spiceai-enterprise-plan \
--version 1.1.0-operator-helm \
--namespace spiceai-operator-system \
--set spice.managedMode.enabled=true \
--set spice.managedMode.enrollmentTokenSecret=spice-enrollment-token{% endhint %}
When managed mode is disabled (the default), the operator behavior does not change: it opens no outbound stream, and every spice.managedMode.* value is ignored. Existing self-hosted installations are unaffected.
Enrollment and connection normally complete within a few minutes. Check the operator logs:
kubectl -n spiceai-operator-system logs deploy/spiceai-operator | grep -i "managed mode"A healthy first start logs, in order:
Managed mode enrolling over the cloud /enroll endpoint
Managed mode enrollment complete; identity persisted to Secret 'spice-byoc-operator-identity'
Managed mode connected to Spice Cloud gateway
In the portal, the wizard's Connect cluster step polls automatically and reports Cluster is online. You can safely close the page while you wait; the cluster detail page shows registration errors on its Events tab.
The portal shows one of these statuses for each BYOC cluster:
| Status | Meaning |
|---|---|
| Pending | The cluster is registered, but no operator has connected yet. |
| Online | The operator has a live connection to the gateway. |
| Degraded | The operator is reachable, but reports a partial failure (for example, unhealthy runtime instances). |
| Offline | The operator's heartbeat was lost. |
| Decommissioning | Decommission was requested; the cluster is shutting down its connection. |
| Archived | Teardown is complete. The cluster is kept for audit and hidden from the clusters list. |
In managed mode, the operator collects two metric streams from each managed pod and forwards them to Spice.ai Cloud:
- Spice runtime metrics — scraped from each pod's own
/metricsendpoint. - Container CPU and memory — scraped from the
/metrics/resourceendpoint of the kubelet on each node. These values power the resource graphs in the portal.
The kubelet scrape does not use the operator's own credentials. The chart creates a dedicated ServiceAccount, spiceai-operator-kubelet-metrics, whose only permission is get nodes/metrics. The operator mints short-lived tokens for that ServiceAccount and presents them to the kubelets. This keeps the credential that reaches the nodes as narrow as possible.
The kubelet serves /metrics/resource over HTTPS. By default, the operator verifies the kubelet's serving certificate against the cluster CA and refuses to scrape a kubelet it cannot verify. Many clusters — including default EKS, GKE, and AKS clusters — use self-signed kubelet serving certificates that fail this check. To find out which case applies to your cluster, run:
kubectl get csr | grep kubelet-serving- Approved requests appear — your cluster issues kubelet certificates from its own CA. The default configuration works; set neither option below.
- No output — your kubelets self-sign. Container CPU/memory collection needs one of these two options (set at most one):
| Option | Helm value | Behavior |
|---|---|---|
| Supply the kubelet CA (preferred) | spice.managedMode.kubeletCaSecret |
Name of a Secret that holds the CA which signs kubelet serving certificates, under the key ca.crt. The operator verifies against it. |
| Skip verification | spice.managedMode.kubeletInsecureTls |
true disables certificate verification for the kubelet scrape only. |
These options mirror metrics-server's --kubelet-certificate-authority and --kubelet-insecure-tls — it is the same endpoint and the same problem.
{% hint style="warning" %}
With kubeletInsecureTls: true, the scrape presents its metrics-only token to whatever answers on the node's address, without proof it is the kubelet. That token can only read node metrics — nothing else — but prefer kubeletCaSecret wherever the cluster can provide a CA. The operator logs a warning at startup while verification is disabled.
{% endhint %}
If neither option is set and the kubelet certificates cannot be verified, the operator logs the reason at startup and counts each skipped scrape on the metric spiceai_operator_managed_metrics_scrape_total{source="kubelet",result="tls_error"}. Spice runtime metrics keep flowing regardless — only the container CPU/memory graphs stay empty.
All managed-mode values live under spice.managedMode:
| Value | Default | Description |
|---|---|---|
enabled |
false |
Master switch. When false, no outbound stream is opened and every other value here is ignored. |
enrollmentToken |
— | The single-use spice-enroll-… token from the portal. The chart stores it in a Secret for the operator. Persisted in Helm history — see Step 2. |
enrollmentTokenSecret |
— | Name of an existing Secret that holds the enrollment token in its token data key. Use instead of enrollmentToken in production. When the chart provisions the Secret from enrollmentToken, it is named spice-enrollment-token. |
enrollUrl |
https://api.spice.ai/v1/cloud-connect |
Base URL of the enrollment API. Leave empty for Spice.ai Cloud; set only for a self-hosted control plane. |
endpoint |
— | Gateway address override. Not needed for enrollment — the gateway address comes from the enrollment response. Required only with a pre-provisioned identity. |
identitySecret |
spice-byoc-operator-identity |
Name of the Secret the operator creates to persist its enrolled identity. Operator-owned — do not pre-create or edit it. |
secretNamespace |
(operator namespace) | Namespace for the managed-mode Secrets. Leave empty to use the operator's own namespace. |
instanceId |
— | Instance identifier. Normally empty — it is assigned during enrollment. Required only with a pre-provisioned identity. |
mtlsSecret |
— | Pre-provisioned client identity (tls.crt/tls.key) — skips enrollment. Must be set together with caSecret. |
caSecret |
— | Pre-provisioned control-plane CA bundle (ca.crt). Must be set together with mtlsSecret. |
serverCaSecret |
— | Extra serving-CA trust anchor (ca.crt) for a private/self-hosted control plane. Trusted in addition to public roots. |
heartbeatIntervalSeconds |
30 |
Heartbeat cadence. |
telemetryIntervalSeconds |
60 |
Telemetry roll-up cadence. |
metricsIntervalSeconds |
30 |
Managed-instance metrics collection cadence. |
kubeletCaSecret |
— | Secret with the CA that signs kubelet serving certificates (ca.crt). Mutually exclusive with kubeletInsecureTls. |
kubeletInsecureTls |
false |
Skip kubelet certificate verification. Mutually exclusive with kubeletCaSecret. |
The Helm values map to run flags on the operator binary. Each flag also reads a matching SPICEAI_MANAGED_MODE_* environment variable (for example, --managed-mode-enabled reads SPICEAI_MANAGED_MODE_ENABLED).
| Flag | Default | Helm value |
|---|---|---|
--managed-mode-enabled |
false |
enabled |
--managed-mode-enroll-url |
https://api.spice.ai/v1/cloud-connect |
enrollUrl |
--managed-mode-enrollment-token-secret |
— | enrollmentTokenSecret |
--managed-mode-endpoint |
— | endpoint |
--managed-mode-identity-secret |
spice-byoc-operator-identity |
identitySecret |
--managed-mode-secret-namespace |
(operator namespace) | secretNamespace |
--managed-mode-instance-id |
— | instanceId |
--managed-mode-mtls-secret |
— | mtlsSecret |
--managed-mode-ca-secret |
— | caSecret |
--managed-mode-server-ca-path |
— | serverCaSecret (mounted as a file) |
--managed-mode-heartbeat-interval-seconds |
30 |
heartbeatIntervalSeconds |
--managed-mode-telemetry-interval-seconds |
60 |
telemetryIntervalSeconds |
--managed-mode-metrics-interval-seconds |
30 |
metricsIntervalSeconds |
--managed-mode-kubelet-certificate-authority |
— | kubeletCaSecret (mounted as a file) |
--managed-mode-kubelet-insecure-tls |
false |
kubeletInsecureTls |
Enabling managed mode adds these objects, beyond the standard operator installation:
- RBAC — the operator ClusterRole gains
getonnodes(to read each node's kubelet port) and permission to mint tokens for exactly one ServiceAccount. The chart also creates thespiceai-operator-kubelet-metricsServiceAccount with a ClusterRole that grantsget nodes/metricsand nothing else. - Secrets — the operator creates and owns the identity Secret (
spice-byoc-operator-identity). Secrets delivered from Spice.ai Cloud carry the labelspice.ai/managed-mode-secret: "true"; the operator refuses to delete Secrets without that label. - ConfigMap —
spice-managed-mode-pausein the operator's namespace, used as a pause gate (see below). - Namespaces — the operator creates the application namespace before it applies a manifest sent from Spice.ai Cloud, when that namespace does not exist yet.
Secret values sent from Spice.ai Cloud are sealed with two layers of public-key encryption (HPKE): one to a key that persists across restarts, and one to a key that exists only in the operator's memory for the current connection. There is no plaintext path.
Managed mode includes a fail-closed pause gate for the resources that Spice.ai Cloud manages. When you pause the cluster from the portal (Admin → Pause), the operator stops reconciling cloud-managed resources. The gate also starts in the paused state on every boot, until the operator reads the spice-managed-mode-pause ConfigMap.
Only resources applied through the cloud connection are affected. Resources you apply yourself with kubectl or Helm keep reconciling normally, even while the cluster is paused or disconnected.
{% hint style="warning" %} If managed mode is enabled but the operator cannot reach Spice.ai Cloud at boot, cloud-managed resources stay paused until the connection succeeds. Self-applied resources are not affected. {% endhint %}
If you manage certificates out of band (for example, with cert-manager against a self-hosted control plane), you can skip enrollment and supply the identity directly. Set all of these together:
spice:
managedMode:
enabled: true
mtlsSecret: my-operator-identity # kubernetes.io/tls Secret: tls.crt, tls.key
caSecret: my-control-plane-ca # Secret with ca.crt
instanceId: my-instance-id
endpoint: connect.example.com:443With a pre-provisioned identity the operator does not enroll and does not run its renewal loop — your certificate issuer owns rotation. enrollmentToken, enrollmentTokenSecret, and enrollUrl are ignored on this path. For a control plane that serves a certificate from a private CA, additionally set serverCaSecret so enrollment and reconnects trust it; concatenate multiple CAs into one PEM bundle if needed.
-
Deploy — once the cluster is Online, it appears as a deploy target in the Spice.ai Cloud portal. In the new-app flow, choose Bring your own cloud and select the cluster.
-
Rotate the enrollment token — from the cluster's Setup tab (before first enrollment) or Admin tab. Issuing a new token revokes the outstanding one. Tokens can only be rotated before the cluster enrolls.
-
Pause / resume — from the cluster's Admin tab. Pause stops reconciliation of cloud-managed resources; resume restores it.
-
Decommission — from the cluster's Admin tab, with a typed-name confirmation. The cluster moves to Decommissioning, its identity is terminally rejected, and the record is finally Archived. To connect the same Kubernetes cluster again, register a new BYOC cluster and enroll with a fresh token.
-
Re-enroll — to force a fresh enrollment on an existing registration (for example, after restoring the operator from a broken state), delete the identity Secret and restart the operator with a new enrollment token:
kubectl -n spiceai-operator-system delete secret spice-byoc-operator-identity
Search the operator logs for managed mode first — every failure mode below logs a clear message. The cluster's Events tab in the portal shows the same lifecycle events (managed-mode-enrolled, managed-mode-connected, managed-mode-disconnect, managed-mode-renewal-degraded, and others).
| Message contains | Cause and fix |
|---|---|
bootstrapToken was replaced by spice.managedMode.enrollmentToken |
The values use the old pre-release key names. Rename bootstrapToken → enrollmentToken and bootstrapTokenSecret → enrollmentTokenSecret. |
requires either enrollmentToken/enrollmentTokenSecret … or both mtlsSecret and caSecret |
enabled: true was set with no identity source. Provide an enrollment token, or a full pre-provisioned identity. |
mtlsSecret and caSecret must be set together |
Half of a pre-provisioned identity. Set both, or neither. |
instanceId is required with a pre-provisioned identity |
Add instanceId (and endpoint) when using mtlsSecret + caSecret. |
kubeletInsecureTls and kubeletCaSecret are mutually exclusive |
Set at most one of the two kubelet options. |
| Chart pull is refused | The AWS identity is not subscribed to the Marketplace listing, or the ECR login expired. Subscribe, then re-run helm registry login. |
The operator fails hard at startup when the managed-mode configuration is invalid, instead of running in a wedged state. The log names the exact flag:
managed mode failed to start: managed mode configuration invalid: ...
Fix the value named in the message and roll the deployment. If the pod crash-loops with a clap argument-parsing error (for example, unexpected value 'true'), the operator image is older than the chart — align image.tag with the chart version.
| Symptom in logs | Cause | Fix |
|---|---|---|
failed to load managed-mode enrollment token secret '<name>' in namespace '<ns>' |
The Secret does not exist, or secretNamespace points at the wrong namespace. |
Create the Secret in the right namespace, or fix secretNamespace. |
is missing required key 'token' |
The Secret exists but the data key is wrong. | Store the token under the data key token. |
does not start with 'spice-enroll-' / expected exactly 32 characters after 'spice-enroll-' |
The token was mistyped, truncated, or is not an enrollment token. | Copy the token exactly as the portal shows it. |
Enroll endpoint returns 410 |
The token expired (1-hour default lifetime). This is the most common first-run failure. | Issue a new token from the cluster's Setup tab and update the Secret. |
Enroll endpoint returns 409 (consumed) |
The token was already used, or the cluster already enrolled once. | Issue a new token. If a previous operator install enrolled, delete the identity Secret first, or decommission and re-register. |
Enroll endpoint returns 401 |
The token was revoked (a newer token was issued) or is unknown. | Use the most recently issued token. |
Managed mode identity/enrollment failed: …; retrying |
Transient network or control-plane error; the operator retries with backoff. | Check egress to api.spice.ai:443. The operator recovers on its own once reachable. |
| Symptom in logs | Cause and fix |
|---|---|
Managed mode connection failed: …; reconnecting |
Transient. The operator reconnects with backoff (1s doubling to a 60s cap). Persistent repeats usually mean egress to connect.<region>-aws.spiceai.io:443 is blocked. |
Spice Cloud stream closed unexpectedly; reconnecting with backoff |
The cloud side closed the stream (for example, a gateway restart). The operator reconnects automatically. |
connection persistently rejected with an expired identity; … Stopping managed mode |
The cluster was decommissioned, or the identity is past recovery. This is a hard stop. Register/rotate in the portal, delete the identity Secret, and enroll with a fresh token. |
failed to build TLS client: reading server_ca_path |
The serverCaSecret mount is missing or holds invalid PEM. Fix the Secret; the operator deliberately does not fall back to public roots. |
| Symptom in logs | Meaning and action |
|---|---|
Managed mode identity renewal failed: …; backing off |
A single failed attempt. No action needed; the operator retries (5s doubling to a 5-minute cap). |
identity renewal is failing repeatedly; the identity expires if this is not resolved |
Renewal has failed for 3 consecutive attempts or 5 minutes. Check egress to api.spice.ai:443. The log includes expires_in_hours; there is a 30-day grace period after expiry. |
identity renewal recovered |
The alert condition cleared. No action needed. |
renewal terminally rejected … a fresh enrollment is required |
Spice.ai Cloud rejected the identity permanently (decommissioned or past grace). Delete the identity Secret and enroll with a fresh token. |
identity secret … carries mTLS material but a missing or empty 'identifier' |
The identity Secret was edited or partially restored. Delete it to force re-enrollment, as the message says. |
Check the scrape counter on the operator's Prometheus endpoint (see Operator Metrics):
kubectl -n spiceai-operator-system port-forward deploy/spiceai-operator 9090:9090
curl -s http://localhost:9090/metrics | grep managed_metricsresult label on …managed_metrics_scrape_total{source="kubelet"} |
Cause and fix |
|---|---|
tls_error |
The kubelet serving certificate cannot be verified — the common case on EKS/GKE/AKS. Set kubeletCaSecret or kubeletInsecureTls: true (see Kubelet TLS verification). |
no_client |
kubeletCaSecret points at an unreadable or empty CA. The startup log names the reason. |
forbidden |
The token authenticated, but authorization failed — the spiceai-operator-kubelet-metrics ClusterRole or binding is missing. Reinstall or repair the chart RBAC. |
unauthorized |
The kubelet could not authenticate the token at all. Check that kubelet webhook authentication is enabled on the nodes. |
no_scraper_token |
The operator cannot mint a token for spiceai-operator-kubelet-metrics — the ServiceAccount or the serviceaccounts/token grant is missing. Reinstall or repair the chart RBAC. |
missing_identity (on source="discovery") |
Managed pods lack the annotations that identify them to Spice.ai Cloud. Pods deployed from the portal carry them automatically. |
Spice runtime metrics (source="spiced") and kubelet metrics are independent — a kubelet TLS problem never blocks runtime metrics.
The operator never connected. Work through, in order: the enrollment token was copied within its lifetime (see Enrollment fails); the operator pod is running; the logs show Managed mode connected to Spice Cloud gateway; egress on 443 to api.spice.ai and connect.<region>-aws.spiceai.io is allowed.
The heartbeat stopped. Check that the operator pod is running and that egress to the gateway is still allowed. The operator reconnects automatically; the status returns to Online on the next successful heartbeat. If the cluster was decommissioned in the portal, the operator logs a terminal rejection instead — see Connection drops.
- Overview — installation, all Helm values, and CLI flags.
- User Guide — deploying and operating Spicepods with the operator.
- Operator Metrics — the operator's own Prometheus/OTLP telemetry, including managed-mode scrape counters.
- AWS Marketplace — subscribing and authenticating to the registry.