[e2e] Error: "tls: failed to verify certificate: x509: certificate signed by unknown authority"

[peppi-lotta]

Which jobs are flaking?

periodic-cluster-api-e2e-mink8s-release-1-11
periodic-cluster-api-e2e-latestk8s-main
periodic-cluster-api-e2e-release-1-11

Which tests are flaking?

• capi-e2e [It] When following the Cluster API quick-start with dualstack and ipv4 primary [IPv6] Should create a workload cluster [IPv6]
• capi-e2e [It] When following the Cluster API quick-start Should create a workload cluster
• capi-e2e [It] When following the Cluster API quick-start with v1beta1 ClusterClass [ClusterClass] Should create a workload cluster [ClusterClass]
• When testing Cluster API working on self-hosted clusters using ClusterClass with a HA control plane [ClusterClass] Should pivot the bootstrap cluster to a self-hosted cluster [ClusterClass]
• capi-e2e [It] When testing Cluster API working on self-hosted clusters using ClusterClass [ClusterClass] Should pivot the bootstrap cluster to a self-hosted cluster [ClusterClass]

Since when has it been flaking?

Jan 23 2026

Testgrid link

https://storage.googleapis.com/k8s-triage/index.html?text=tls%3A%20failed%20to%20verify%20certificate&job=.*cluster-api&xjob=.*-provider-

Reason for failure (if possible)

Failure to call webhook because of tls: failed to verify certificate: x509: certificate signed by unknown authority

Anything else we need to know?

First occurrence is Jan 23 2026 but the error peaked Jan 28 and Feb 3 with multiple errors per day

I noticed cert-manager bump was merged around the time the first peak happened
#13279
#13278
#13277

Label(s) to be applied

/kind flake
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[peppi-lotta]

@tsj-30 as discussed in the CI-meeting today, you could take a look at this one. For example check if local reproduction is possible and is there a clear error message in the logs :)

[tsj-30]

Hey @peppi-lotta ,
I tried to repro the issue in my local, wasn't able to repro it.
Will try to do RCA and update here.
Thanks. 😄

[tsj-30]

RCA for https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-e2e-release-1-11/2023010815933681664

Failure summary:

• The failing test was the self-hosted ClusterClass scenario.
• Worker MachineDeployment self-hosted-mh0n08-md-0-mkrch never became available (0/1 replicas), so the test timed out waiting for a worker node.

Evidence:

• build-log.txt: MachineDeployment self-hosted-mh0n08-md-0-mkrch: 0 available replicas, at least 1 required.
• DockerMachine status for self-hosted-mh0n08-md-0-mkrch-xd6qk-msq6f: BootstrapExecSucceeded=False, message: Failed to exec DockerMachine bootstrap.
• CAPD manager log shows kubeadm join failed and timed out while waiting for kubelet health:
  unable to start the kubelet service: [exit status 1]
operation cancelled: context deadline exceeded.
• Worker journal.log shows repeated kubelet startup failure:
  ERROR: this script needs /sys/fs/cgroup/cgroup.procs to be empty (for writing the top-level cgroup.subtree_control).

Conclusion:

• This is a worker bootstrap failure in CAPD/kind due to kubelet/cgroup initialization failure, which prevented node registration and caused MachineDeployment availability timeout.