security: disable anonymous auth on kube-apiserver (MK8S-187)#4900
security: disable anonymous auth on kube-apiserver (MK8S-187)#4900g-carre wants to merge 4 commits intodevelopment/132.0from
Conversation
kube-apiserver was running with the default --anonymous-auth=true, letting unauthenticated callers reach endpoints bound to system:public-info-viewer (e.g. /version, /healthz) and disclose Kubernetes/Go versions usable for further attack planning. Set --anonymous-auth=false so the system:anonymous user cannot authenticate at all, which neutralises the default kubeadm-managed ClusterRoleBindings granting access to system:unauthenticated. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Bake the security advisory's diagnostic into the post-deploy BDD suite so a regression on --anonymous-auth is caught by CI: hit the advisory's exact endpoint (/api/kubernetes/version through the control-plane ingress, which proxies to kube-apiserver) without credentials and expect a 401 Unauthorized. Reuses existing 'perform a request on ... on control-plane Ingress' and 'the server returns ... with message ...' steps; no new step code. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Hello g-carre,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
|
/create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
…K8S-187) Disabling --anonymous-auth in commit f39d88d caused bootstrap to hang on the salt http.wait_for_successful_query probes that hit /healthz on kube-apiserver: the unauthenticated GET now returns 401, the state times out after 5 minutes, and every dependent orchestration step fails with "One or more requisite failed". Pass the apiserver-kubelet client cert and key (CN kube-apiserver-kubelet-client, O system:masters) to all seven affected probes -- the direct one on https://<host>:6443/healthz in kubernetes/apiserver/installed.sls, and the six going through the local apiserver-proxy on https://127.0.0.1:7443/healthz in the bootstrap, deploy_node, upgrade and downgrade orchestrations. The salt-master Pod already mounts /etc/kubernetes/pki, and the cert exists on every master before kube-apiserver starts, so no extra plumbing is needed. Expose apiserver-kubelet-client.key as certificates.client.files['apiserver-kubelet'].key in defaults.yaml and switch existing literal references in apiserver/installed.sls and apiserver/certs/kubelet-client.sls to it, to keep the path in a single place. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
There was a problem hiding this comment.
Pull request overview
This PR hardens the Kubernetes control plane by restricting kube-apiserver anonymous authentication to health endpoints only, while ensuring Salt orchestration health probes and post-install tests continue to function.
Changes:
- Add a kube-apiserver
AuthenticationConfigurationto allow anonymous access only to/livez,/readyz, and/healthz, and wire it into the apiserver static pod manifest. - Update Salt
http.wait_for_successful_queryprobes (bootstrap/deploy/upgrade/downgrade and apiserver install readiness checks) to use the existing apiserver-kubelet client certificate. - Extend post-install authentication BDD tests to assert
/api/kubernetes/versionis rejected anonymously while health endpoints remain accessible (anonymous via Ingress; authenticated directly to apiserver).
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/post/steps/test_authentication.py | Adds new pytest-bdd scenario bindings and a step to perform mTLS-authenticated requests directly against the apiserver. |
| tests/post/features/authentication.feature | Adds scenarios covering anonymous rejection of /version, anonymous allowance of health endpoints via Ingress, and authenticated health endpoint access directly. |
| salt/metalk8s/orchestrate/upgrade/init.sls | Authenticates apiserver availability checks during upgrade using the apiserver-kubelet client cert. |
| salt/metalk8s/orchestrate/downgrade/init.sls | Authenticates apiserver availability checks during downgrade using the apiserver-kubelet client cert. |
| salt/metalk8s/orchestrate/deploy_node.sls | Authenticates apiserver availability checks before/after highstate using the apiserver-kubelet client cert. |
| salt/metalk8s/orchestrate/bootstrap/pre-upgrade.sls | Authenticates the local proxy apiserver availability check prior to master upgrade. |
| salt/metalk8s/orchestrate/bootstrap/pre-downgrade.sls | Authenticates the local proxy apiserver availability check prior to master downgrade. |
| salt/metalk8s/orchestrate/bootstrap/init.sls | Authenticates bootstrap apiserver availability checks using the apiserver-kubelet client cert. |
| salt/metalk8s/kubernetes/apiserver/installed.sls | Mounts and passes the new authentication config into kube-apiserver; switches kubelet client key path to the centralized defaults map; authenticates readiness probe. |
| salt/metalk8s/kubernetes/apiserver/init.sls | Includes the new authnconfig sub-state in the apiserver state bundle. |
| salt/metalk8s/kubernetes/apiserver/certs/kubelet-client.sls | Switches the kubelet-client private key path to the centralized defaults map. |
| salt/metalk8s/kubernetes/apiserver/authnconfig.sls | New state generating /etc/kubernetes/authentication-config.yaml restricting anonymous auth to health endpoints only. |
| salt/metalk8s/defaults.yaml | Adds the apiserver-kubelet client key path into the certificates.client.files map for centralized reuse. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
61d346a to
15c0da2
Compare
History mismatchMerge commit #1b0256add37d05f04c884aa2f6a8b0f5531c24ed on the integration branch It is likely due to a rebase of the branch Please use the The following options are set: create_integration_branches |
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
15c0da2 to
10e4ed5
Compare
History mismatchMerge commit #68f2a0394cd43770668080f6bc58eaa70f2ac9b6 on the integration branch It is likely due to a rebase of the branch Please use the The following options are set: create_integration_branches |
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
10e4ed5 to
6132e5c
Compare
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
3 times, most recently
from
904e998 to
1cb9435
Compare
…8S-187) Switch kube-apiserver from --anonymous-auth=false to an AuthenticationConfiguration that allows anonymous access to /livez, /readyz and /healthz only. This restores kubelet livenessProbe, startupProbe and readinessProbe (plain httpGet, no way to attach credentials) while every other endpoint -- /version, /api/*, the discovery surface -- still requires authentication, so the original goal of MK8S-187 (neutralising the kubeadm system:public-info-viewer binding to system:anonymous) is preserved. The previous --anonymous-auth=false approach broke kubelet probes: unauthenticated GETs to /livez and /readyz returned 401, and after startupProbe.failureThreshold * periodSeconds (~250s) kubelet would have killed and restarted the apiserver in a permanent crash-loop. The salt http.wait_for_successful_query timeout on /healthz (fixed in 68189f9 by attaching a client cert) was the visible symptom; the kubelet-probe failure is the underlying one and cannot be fixed with client certs because httpGet probes do not support TLS client auth. Generate /etc/kubernetes/authentication-config.yaml from a sibling salt state (mirroring cryptconfig.sls), mount it into the static pod as a File volume, and wire --authentication-config to it. --anonymous-auth is dropped because it is mutually exclusive with the anonymous block in AuthenticationConfiguration. Move the OIDC issuer configuration from --oidc-* CLI flags to the AuthenticationConfiguration `jwt:` array. K8s 1.32 rejects the two mechanisms together (pkg/kubeapiserver/options/authentication.go: "authentication-config file and oidc-* flags are mutually exclusive"); a first apiserver start can succeed before the control-plane Ingress endpoint is known (no --oidc-* flags emitted), but the subsequent "Reconfigure control plane Ingress" pass populates oidc_config in the pillar, re-renders the manifest with both flag sets, and the apiserver crash-loops on startup. The same five fields the legacy code consumed (issuerURL, clientID, CAFile, usernameClaim, groupsClaim) feed the new jwt issuer block. The CA PEM is read from the salt mine entry `ingress_ca_b64` for the default Dex / Ingress path (avoiding any state-graph ordering hazard against the on-disk file), and falls back to salt['file.read'] for a pillar-supplied OIDC override pointing at a different CA path. When the username claim is `email`, a `claimValidationRules` CEL expression `claims.?email_verified.orValue(true)` is added to reproduce the implicit `email_verified == true` guard that the legacy --oidc-username-claim=email flag used to apply automatically. ARTESCA configures Keycloak via the same OIDC pillar override, so its reconfiguration path is unaffected. Relies on the AnonymousAuthConfigurableEndpoints and StructuredAuthenticationConfiguration feature gates, both beta and on-by-default in Kubernetes 1.32 (per pkg/features/versioned_kube_features.go in release-1.32) -- no extra --feature-gates flag needed for the version metalk8s pins. Extend the post-install authentication feature with two scenario outlines that lock the contract in place: anonymous access to /livez, /readyz, /healthz returns 200 'ok' (via the control-plane Ingress, same path as the existing "rejects anonymous" test), and authenticated access (admin client cert from the kubeconfig) to the same three paths also returns 200 'ok'. The pre-existing /version 401 scenario still passes because /version is not in the allowed anonymous path list. Signed-off-by: Guillaume Carre <guillaume.carre@scality.com>
g-carre
force-pushed
the
bugfix/MK8S-187-disable-kube-apiserver-anonymous-auth
branch
from
1cb9435 to
250a7d0
Compare
|
/reset |
Reset completeI have successfully deleted this pull request's integration branches. The following options are set: create_integration_branches |
Integration data createdI have created the integration data for the additional destination branches.
The following branches will NOT be impacted:
You can set option The following options are set: create_integration_branches |
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
Peer approvals must include at least 1 approval from the following list: The following options are set: create_integration_branches |
Component: salt, tests
Context:
A security assessment flagged that the ARTESCA Kubernetes cluster permits anonymous access to the API server. With kube-apiserver running on its default
--anonymous-auth=true, unauthenticated callers can reach endpoints bound to the kubeadm-managedsystem:public-info-viewerClusterRoleBinding (e.g./version,/healthz) and disclose Kubernetes / Go versions usable for further attack planning.Reproducer (from the advisory):
curl -k https://<INSTANCE_IP>:8443/api/kubernetes/versionreturns a JSON body withgitVersion/goVersioninstead of401.Summary:
AuthenticationConfiguration(/etc/kubernetes/authentication-config.yaml, generated from a new sibling salt state mirroringcryptconfig.sls) that scopes anonymous access to/livez,/readyz,/healthzonly.--anonymous-auth=falsewas tried first but is incompatible with kubelethttpGetprobes (which cannot carry credentials) and would put the apiserver in a startup-probe crash-loop after ~250s; it is also mutually exclusive with theAuthenticationConfiguration.anonymousblock, so it is dropped.--oidc-*CLI flags intoAuthenticationConfiguration.jwt. K8s 1.32 rejects the two mechanisms together (pkg/kubeapiserver/options/authentication.go: "authentication-config file and oidc- flags are mutually exclusive"*). A first apiserver start can succeed before the control-plane Ingress endpoint is known (no--oidc-*flags emitted), but the subsequent Reconfigure control plane Ingress pass populatesoidc_config, re-renders the manifest with both flag sets, and the apiserver crash-loops on startup. The same five fields the legacy code consumed (issuerURL,clientID,CAFile,usernameClaim,groupsClaim) feed the newjwtissuer block. The CA PEM is read from the salt mine entryingress_ca_b64for the default Dex / Ingress path (avoiding any state-graph ordering hazard), with asalt['file.read']fallback for a pillar-supplied OIDC override pointing at a different CA path. When the username claim isemail, aclaimValidationRulesCEL expressionclaims.?email_verified.orValue(true)is added to reproduce the implicitemail_verified == trueguard that--oidc-username-claim=emailused to apply automatically. ARTESCA configures Keycloak via the same OIDC pillar override (installer/artesca_installer/salt/artesca/base/keycloak/configured.sls), so its reconfiguration path is unaffected.AnonymousAuthConfigurableEndpointsandStructuredAuthenticationConfigurationfeature gates, both beta and on-by-default in Kubernetes 1.32 (perpkg/features/versioned_kube_features.goinrelease-1.32). No--feature-gatesflag needed for the version metalk8s pins.http.wait_for_successful_queryprobes that hit/healthzon the apiserver (one direct, six through the local apiserver-proxy:7443raw stream) using the existingapiserver-kubelet-client.{crt,key}(CNkube-apiserver-kubelet-client, groupsystem:masters). Defence-in-depth: those probes still succeed even though/healthzis now anonymous-allowed, and the cert pattern matches what the etcd state already does.apiserver-kubelet-client.keyascertificates.client.files['apiserver-kubelet'].keyinsalt/metalk8s/defaults.yamland switch the existing literal references inapiserver/installed.slsandapiserver/certs/kubelet-client.slsto it, to keep the path in a single place.tests/post/features/authentication.feature:GET /api/kubernetes/version→401 Unauthorized. Still passes —/versionis not in the allowed anonymous list.GET /api/kubernetes/{livez,readyz,healthz}via the control-plane Ingress →200 ok.GET /{livez,readyz,healthz}directly on the API server (admin client cert pulled from the kubeconfig) →200 ok.Acceptance criteria:
kubectl, salt, kubelet → apiserver, and the bootstrap/post-install flow continue to work end-to-end (single-node and multi-node e2e green).curl -k https://<INSTANCE_IP>:8443/api/kubernetes/versionreturns401, bodyUnauthorized.curl -k https://<INSTANCE_IP>:8443/api/kubernetes/livez(and/readyz,/healthz) returns200, bodyok.kube-apiserver …scenarios intests/post/features/authentication.feature.Follow-ups:
apiVersionfromapiserver.config.k8s.io/v1beta1tov1once metalk8s pins Kubernetes ≥ 1.34 (no urgency — v1beta1 still served through 1.36+).See: MK8S-187