admin.conf unusable during cluster creation due to race between TCP reconciler and PhaseClusterAdminRBAC

[HarshavardhanK]

Summary

Since Kamaji adopted kubeadm v1.36 (edge-26.2.5+, following KEP-2305), the admin.conf client certificate is generated with O=kubeadm:cluster-admins instead of the previous O=system:masters. This creates a race condition that makes the {tcp}-admin-kubeconfig secret temporarily unusable after a tenant cluster is created or its certificates are rotated.

Root cause

The TCP reconciler writes the new admin.conf (with O=kubeadm:cluster-admins) to the management cluster secret. A consumer of this secret trying to authenticate against the tenant cluster then needs the kubeadm:cluster-admins → cluster-admin ClusterRoleBinding to exist inside that tenant. This CRB is created by PhaseClusterAdminRBAC in the soot manager — a separate controller loop that starts after the control plane becomes reachable, asynchronously relative to the TCP reconciler.

In the window between the TCP reconciler writing admin.conf and the soot manager completing PhaseClusterAdminRBAC, any caller using admin.conf to perform privileged operations in the tenant receives:

secrets is forbidden: User "kubernetes-admin" cannot create resource "secrets"
  in API group "" in the namespace "kube-system"

The cert authenticates successfully (HTTP 403, not 401) — the issue is purely that O=kubeadm:cluster-admins requires a CRB that does not yet exist.

Why this wasn't an issue before

Before kubeadm v1.29 / KEP-2305, admin.conf used O=system:masters, which is a Kubernetes API-server internal short-circuit that bypasses RBAC entirely. No ClusterRoleBinding was required.

Relevant files

• internal/resources/kubeconfig.go — KubeconfigResource.mutate calls kubeadm.CreateKubeconfig(AdminKubeConfigFileName, ...), which in kubeadm v1.36 emits O=kubeadm:cluster-admins.
• internal/resources/kubeadm_phases.go — PhaseClusterAdminRBAC calls EnsureAdminClusterRoleBinding to create the CRB. Runs in the soot manager.
• controllers/soot/manager.go — starts the soot manager, including PhaseClusterAdminRBAC. Runs after TCP reconciler.

Proposed fix

When generating admin.conf, substitute SuperAdminKubeConfigFileName as the cert configuration argument so the cert carries O=system:masters (same as super-admin.conf). The secret key is still named admin.conf; only the embedded cert's Organization field changes. This makes admin.conf unconditionally privileged without depending on ClusterRoleBinding ordering.

kubeconfigFileName := r.KubeConfigFileName
if kubeconfigFileName == kubeadmconstants.AdminKubeConfigFileName {
    kubeconfigFileName = kubeadmconstants.SuperAdminKubeConfigFileName
}
kubeconfig, kcErr := kubeadm.CreateKubeconfig(kubeconfigFileName, crtKeyPair, config)

An alternative fix is to make PhaseClusterAdminRBAC a hard prerequisite before the TCP reconciler marks admin.conf as ready, or to add the admin-kubeconfig secret as a watch trigger for PhaseClusterAdminRBAC.

A PR with the proposed fix is attached.

[HarshavardhanK]

Correction on the proposed fix

The PR I submitted (#1168) was wrong and I've closed it. Reverting admin.conf to O=system:masters undoes the revocability guarantee KEP-2305 was designed to provide, and re-introduces a non-revocable, RBAC-invisible credential into a stored per-tenant secret. That is a security regression, not a bug fix.

The correct fix targets the ordering gap, not the O= field.

The real issue is that PhaseClusterAdminRBAC runs asynchronously in the soot manager, after the TCP reconciler has already written the new admin.conf cert. The two approaches worth evaluating:

Option A — inline CRB creation in the TCP reconciler

After writing admin.conf (with O=kubeadm:cluster-admins), run EnsureAdminClusterRoleBinding directly in the TCP reconcile path using the CA the reconciler already holds to mint a short-lived super-admin-equivalent credential. This is exactly what kubeadm does at bootstrap and closes the window completely.

Option B — gate admin.conf ready-status on PhaseClusterAdminRBAC

Mark the admin-kubeconfig resource's status as not-ready until PhaseClusterAdminRBAC has confirmed the CRB exists. Callers that check TCP readiness before using admin.conf would naturally wait. This is more idiomatic to Kamaji's resource/status model but requires callers to respect the readiness signal.

Both options keep O=kubeadm:cluster-admins and preserve the revocability property. Happy to contribute a PR for whichever direction the maintainers prefer.