nodes are unable to join cluster

[jorhett]

When using a configuration very similar to your example -- except supplying our own bootstrap token of course, the nodes fail to come online because they can't retrieve the CA token.

Notice: /Stage[main]/K8s::Common/File[/root/.kube]/ensure: created
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Remove broken CA]/returns: executed successfully
Info: /Stage[main]/K8s::Node::Kubelet/Exec[Remove broken CA]: Scheduling refresh of Exec[Retrieve K8s CA]
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: E0422 22:52:02.097623 1309935 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: E0422 22:52:02.102260 1309935 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: E0422 22:52:02.106345 1309935 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: E0422 22:52:02.110465 1309935 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: E0422 22:52:02.115065 1309935 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: Error from server (Forbidden): unknown
Error: 'kubectl --server='https://example-k8s.example.com:6443' --username=anonymous --insecure-skip-tls-verify=true           get --namespace=kube-system configmap cluster-info --output=jsonpath={.data.ca} > '/var/lib/kubelet/pki/ca.pem'' returned 1 instead of one of [0]
Error: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]/returns: change from 'notrun' to ['0'] failed: 'kubectl --server='https://example-k8s.example.com:6443' --username=anonymous --insecure-skip-tls-verify=true           get --namespace=kube-system configmap cluster-info --output=jsonpath={.data.ca} > '/var/lib/kubelet/pki/ca.pem'' returned 1 instead of one of [0]
Notice: /Stage[main]/K8s::Node::Kubelet/Exec[Retrieve K8s CA]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/K8s::Node::Kubelet/Kubeconfig[/srv/kubernetes/bootstrap-kubelet.kubeconf]: Dependency Exec[Retrieve K8s CA] has failures: true

Testing manually from that node confirms this query won't work:

# kubectl --server='https://example-k8s.example.com:6443' --username=anonymous --insecure-skip-tls-verify=true           get --namespace=kube-system configmap cluster-info -oyaml
E0422 22:53:19.658375 1310295 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
E0422 22:53:19.662981 1310295 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
E0422 22:53:19.667140 1310295 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
E0422 22:53:19.671012 1310295 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
E0422 22:53:19.674882 1310295 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"
Error from server (Forbidden): unknown

We aren't disabling anonymous auth, and I do see anonymous authentication is enabled on the running apiserver process... I suspect something has changed in kubernetes 1.32 which means that the very manual setup used in k8s::server::resources is likely out of date?

[jorhett]

Adding -v10 to the query I see that it's trying to query /api before it tries to get the configmap.

I0423 00:32:45.295774 1346609 round_trippers.go:473] curl -v -XGET  -H "User-Agent: kubectl/v1.32.3 (linux/amd64) kubernetes/32cc146" -H "Authorization: Basic <masked>" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://example-k8s.example.com:6443/api?timeout=32s'
I0423 00:32:45.296419 1346609 round_trippers.go:502] HTTP Trace: DNS Lookup for example-k8s.example.com resolved to [{10.22.20.17 }]
I0423 00:32:45.297055 1346609 round_trippers.go:517] HTTP Trace: Dial to tcp:10.22.20.17:6443 succeed
I0423 00:32:45.299918 1346609 round_trippers.go:560] GET https://example-k8s.example.com:6443/api?timeout=32s 403 Forbidden in 4 milliseconds
I0423 00:32:45.299930 1346609 round_trippers.go:577] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 1 ms ServerProcessing 0 ms Duration 4 ms
I0423 00:32:45.299935 1346609 round_trippers.go:584] Response Headers:
I0423 00:32:45.299940 1346609 round_trippers.go:587]     Content-Length: 188
I0423 00:32:45.299944 1346609 round_trippers.go:587]     Date: Wed, 23 Apr 2025 07:32:45 GMT
I0423 00:32:45.299948 1346609 round_trippers.go:587]     Audit-Id: af9d77ce-ccaf-4afb-a8df-05d08fd6f967
I0423 00:32:45.299952 1346609 round_trippers.go:587]     Cache-Control: no-cache, private
I0423 00:32:45.299955 1346609 round_trippers.go:587]     Content-Type: application/json
I0423 00:32:45.299958 1346609 round_trippers.go:587]     X-Content-Type-Options: nosniff
I0423 00:32:45.299962 1346609 round_trippers.go:587]     X-Kubernetes-Pf-Flowschema-Uid: cea52002-a3fe-441d-b224-a17731270d93
I0423 00:32:45.299965 1346609 round_trippers.go:587]     X-Kubernetes-Pf-Prioritylevel-Uid: 4955aaad-8a63-49b8-b6c0-08f58f725998
I0423 00:32:45.300169 1346609 discovery_client.go:253] "Response Body" body=<
	{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}
 >
E0423 00:32:45.300335 1346609 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: unknown"

[jorhett]

Copying the ~/.kube/config file from the server node works fine with this command, but this is failing to work with the bootstrap token.