Skip to content

Commit 38d99a7

Browse files
committed
docs(enterprise): clarify BYOC secret encryption boundary and kubelet TLS diagnosis
The HPKE sealing protects secret values in transit only; at-rest protection follows the cluster's Secret encryption and access controls. The kubelet-serving CSR check is inconclusive when CSR objects have been garbage-collected, so lead with the operator's tls_error signal instead.
1 parent 90a9ae8 commit 38d99a7

1 file changed

Lines changed: 10 additions & 9 deletions

File tree

enterprise/kubernetes/byoc.md

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -149,14 +149,9 @@ The kubelet scrape does not use the operator's own credentials. The chart create
149149

150150
### Kubelet TLS verification
151151

152-
The kubelet serves `/metrics/resource` over HTTPS. By default, the operator **verifies** the kubelet's serving certificate against the cluster CA and refuses to scrape a kubelet it cannot verify. Many clusters — including default **EKS, GKE, and AKS** clusters — use self-signed kubelet serving certificates that fail this check. To find out which case applies to your cluster, run:
152+
The kubelet serves `/metrics/resource` over HTTPS. By default, the operator **verifies** the kubelet's serving certificate against the cluster CA and refuses to scrape a kubelet it cannot verify. Many clusters — including default **EKS, GKE, and AKS** clusters — use self-signed kubelet serving certificates that fail this check.
153153

154-
```bash
155-
kubectl get csr | grep kubelet-serving
156-
```
157-
158-
- **Approved requests appear** — your cluster issues kubelet certificates from its own CA. The default configuration works; set neither option below.
159-
- **No output** — your kubelets self-sign. Container CPU/memory collection needs one of these two options (set at most one):
154+
Start with the default (verification enabled) and let the operator tell you whether a change is needed. When the kubelet certificates cannot be verified, the operator logs the reason at startup and counts each skipped scrape on the metric `spiceai_operator_managed_metrics_scrape_total{source="kubelet",result="tls_error"}`. Spice runtime metrics keep flowing regardless — only the container CPU/memory graphs stay empty. A rising `tls_error` count means container CPU/memory collection needs one of these two options (set at most one):
160155

161156
| Option | Helm value | Behavior |
162157
| ----------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
@@ -169,7 +164,13 @@ These options mirror `metrics-server`'s `--kubelet-certificate-authority` and `-
169164
With `kubeletInsecureTls: true`, the scrape presents its metrics-only token to whatever answers on the node's address, without proof it is the kubelet. That token can only read node metrics — nothing else — but prefer `kubeletCaSecret` wherever the cluster can provide a CA. The operator logs a warning at startup while verification is disabled.
170165
{% endhint %}
171166

172-
If neither option is set and the kubelet certificates cannot be verified, the operator logs the reason at startup and counts each skipped scrape on the metric `spiceai_operator_managed_metrics_scrape_total{source="kubelet",result="tls_error"}`. Spice runtime metrics keep flowing regardless — only the container CPU/memory graphs stay empty.
167+
To confirm how your cluster issues kubelet serving certificates, check for approved `kubelet-serving` certificate signing requests:
168+
169+
```bash
170+
kubectl get csr | grep kubelet-serving
171+
```
172+
173+
Approved requests mean the cluster issues kubelet serving certificates from its own CA, and the default configuration works. **No output is inconclusive** — Kubernetes garbage-collects CSR objects, and kubelet certificates can be provisioned without a retained CSR — so do not switch to `kubeletInsecureTls` on that signal alone. Rely on the `tls_error` count above, or inspect the issuer of the certificate a kubelet serves on port `10250`.
173174

174175
## Configuration reference
175176

@@ -227,7 +228,7 @@ Enabling managed mode adds these objects, beyond the standard operator installat
227228
- **ConfigMap**`spice-managed-mode-pause` in the operator's namespace, used as a pause gate (see below).
228229
- **Namespaces** — the operator creates the application namespace before it applies a manifest sent from Spice.ai Cloud, when that namespace does not exist yet.
229230

230-
Secret values sent from Spice.ai Cloud are sealed with two layers of public-key encryption (HPKE): one to a key that persists across restarts, and one to a key that exists only in the operator's memory for the current connection. There is no plaintext path.
231+
Secret values sent from Spice.ai Cloud are sealed with two layers of public-key encryption (HPKE): one to a key that persists across restarts, and one to a key that exists only in the operator's memory for the current connection. There is no plaintext delivery path — the operator rejects a payload that fails either layer. This encryption protects the values in transit only: after decryption, the operator stores them as standard Kubernetes Secrets, so protection at rest depends on your cluster's Secret encryption and access controls.
231232

232233
### Pause behavior
233234

0 commit comments

Comments
 (0)