Error E0803 logged 5 times a second

[alinjie]

Hi!

I am running a 3 node k3s cluster on version v1.24.3+k3s1 and running journalctl -u k3s I am seeing error E0803 logged multiple times a second. Everything seems to be working fine, but I am not sure how to debug this as I'm not able to find any documentation on this error or anything simmilar on Google.

Any suggestions as to how to debug this or what the issue might be?

Thanks in advance.

W0803 14:32:19.774344   23790 reflector.go:324] k8s.io/client-go@v1.24.3-k3s1/tools/cache/reflector.go:167: failed to list *v1.PartialObjectMetadata: Internal error occurred: error resolving resource
E0803 14:32:19.774371   23790 reflector.go:138] k8s.io/client-go@v1.24.3-k3s1/tools/cache/reflector.go:167: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: Internal error occurred: error resolving resource
E0803 14:32:20.540323   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:20.601745   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:20.686137   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:20.711658   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:20.738200   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:21.537665   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:21.599496   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:21.684775   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:21.710555   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:21.735449   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:22.538012   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:22.597148   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:22.687949   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:22.712828   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:22.738101   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:23.536702   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:23.602118   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:23.689882   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:23.713777   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block
E0803 14:32:23.740633   23790 customresource_handler.go:309] unable to load root certificates: unable to parse bytes as PEM block

[brandond]

That's not anything specific to K3s. It appears to be coming from something within the Kubernetes core code, probably due to something you've deployed. Have you deployed anything that's registered a webhook of some sort?

[alinjie]

Yes, I've deployed both the Prometheus Operator and Vault (with the agent injector). Prometheus deploys both validating and mutating webhooks and Vault I believe only uses mutating.

[brandond]

I would check the TLS config for both of those. Sounds like one of them has a mangled cert.

[alinjie]

Thank you. I will check this tomorrow and update!

[alinjie]

For anyone potentially experiencing the same problem;

The issue was that I seemed to have some left over CRDs, presumably from installing and uninstalling Rancher, that had invalid caBundle values as they were all the value of Cg==.

I ran the following in order to identify them

CRDS=$(kubectl crd -o name)
for CRD in $CRDS; do if [[ $(kubectl get $CRD -A -o yaml | grep caBundle) == *"Cg=="* ]]; then echo $CRD; fi; done

They all had finalizers within the .metadata.finalizers array, which I removed. After removing the finalizers and deleting the CRDs, the issue was resolved.

Thank you @brandond for pointing me in the right direction!