Provide a Prometheus-consumable system:monitoring client certificate for scraping TCP metrics

[Jakob3xD]

Follow-up to the monitoring discussion in #1171. That issue was the apiserver-kubelet-client ClusterRoleBinding bug (fixed by #1172, now closed); this is the separate "proper monitoring identity" idea that came up in its comments.

Goal: let Prometheus scrape a TCP's control-plane metrics with a dedicated least-privilege client cert, instead of reusing apiserver-kubelet-client.

Prometheus can't consume the KubeconfigGenerator output as-is. A ServiceMonitor/PodMonitor tlsConfig takes the client cert and key as PEM via two references (cert = SecretOrConfigMap, keySecret = SecretKeySelector) and runs pem.Decode on each. The generator writes a single Secret key value holding the whole kubeconfig YAML (with client-certificate-data/client-key-data base64-nested inside), so pem.Decode on it finds no PEM block → tls: failed to find any PEM data in certificate input.

What it needs is PEM under the referenced key(s). That can be the conventional split tls.crt/tls.key, or even a single key holding a combined cert+key PEM referenced by both cert and keySecret (Go's X509KeyPair scans each input for the matching block, so that works too). The one thing it won't do is unwrap the kubeconfig — so cert and key sharing one Secret is fine; the kubeconfig packaging is the problem.

Two relevant points:

• Authorization is already covered. system:monitoring is a built-in ClusterRole with a built-in ClusterRoleBinding to the system:monitoring group (GET on /metrics, /livez, /readyz, …). A client cert with O=system:monitoring is authorized in every tenant out of the box — no RBAC to ship. That's also why it sidesteps the binding the apiserver-kubelet-client identity needs in fix(kubeadm): call AllowAPIServerToAccessKubeletAPI in BootstrapToken added in k8s v1.36.1 #1172.
• Confirmed end to end on a cluster: a KubeconfigGenerator with user=monitoring-agent, groups=[system:monitoring] mints exactly subject=O=system:monitoring, CN=monitoring-agent, signed by the tenant CA. The identity is correct — only the packaging is wrong for Prometheus.

So the only real gap is the cert format. Options for kamaji to emit something Prometheus-ready:

1. Extend KubeconfigGenerator to also write the cert/key as PEM (e.g. tls.crt/tls.key) next to value. Cheapest — the PEM bytes already exist in the controller, and the consumers that loop the secret keys (e.g. extractCertificateFromKubeconfig) tolerate unknown keys. Downside: mixes a kubeconfig and raw PEM in one Secret.
2. A dedicated, opt-in system:monitoring client cert that kamaji generates per TCP, using the same secret/rotation machinery as its existing cert secrets (split .crt/.key, x509-labeled and TCP-owned so the certificate-lifecycle controller rotates it automatically), with the cert minted under O=system:monitoring the way the KubeconfigGenerator already does. To be explicit: this is a new, purpose-built least-privilege cert — not a reuse of apiserver-kubelet-client, which is the overprivileged identity we want to move away from.

I'd lean toward option 2 for a clean built-in story, but I'm happy with 1 if you'd rather keep it inside the generator. Which direction do you prefer?

One operational note either way: the KubeconfigGenerator is an opt-in separate deployment today (kubeconfigGenerator.enabled=false), so the generator-based path means operators run an extra component.

[prometherion]

I'd lean toward option 2

I'm interested about your perspective, mostly because I'm for option no.1 and extend the KubeconfigGenerator since I'm not very satisfied by it, since it's mostly unutilized although it could serve such a features.

I'm biased, I know, but I'll br happy to extend it and offer on the same Secret the related cert and private key.

[Jakob3xD]

Fair points — and I agree that the KubeconfigGenerator being underutilized is a good reason to give it more to do. Let me push back gently on the one axis I think actually matters here.

My hesitation isn't about the generator's internals — rotation is handled either way. The generator's Secret is already labeled kamaji.clastix.io/certificate_lifecycle_controller=kubeconfig, so the CertificateLifecycle controller checks NotAfter and re-enqueues it before expiry, the same path the regular TCP certs use. So whichever option we pick, the cert rotates for free.

What gives me pause is coupling. The KubeconfigGenerator is opt-in and runs as a separate Deployment (kubeconfigGenerator.enabled=false). Scraping control-plane metrics, on the other hand, is close to a baseline expectation for a TCP — it's a first-class workflow in the monitoring guide. Gating that credential behind "also enable and run the generator" ties an essentially-default capability to an otherwise-unrelated opt-in component: an operator who just wants Prometheus to scrape their tenants' CPs would have to stand up the generator too.

That's why I'd still lean toward a dedicated system:monitoring cert that rides the existing per-TCP cert/secret machinery — the same secrets the TCP already produces and the lifecycle controller already rotates — so it's available without an extra component, and could even be on by default. It's still least-privilege (O=system:monitoring, authorized out of the box, no RBAC to ship); it just isn't bound to the generator's enablement.

If you'd still rather keep it in the generator, I won't block on it — it solves the format problem just as well. I just wanted to make the "metrics is effectively a default, so its credential probably shouldn't require a second component" case explicitly before we commit. WDYT?