Want to understand the behavior of k3s restart

[BhautikChudasama]

Issue

I recently set up a ValidatingWebhook for the node resource and the create operation. However, every time I restart the K3s server, my webhook receives a CREATE operation request for server only. Can you help me understand this behavior, @brandond? thanks :)

[brandond]

This is just how the kubelet works. It will send a CREATE request, and the apiserver will respond with a conflict error because it already exists, containing the existing resource. This is more efficient than doing a GET, and then sending a CREATE if the resource does not exist.

This is a common pattern in Kubernetes.

[BhautikChudasama]

Got it! 🫡 But why doesn't the server go into the NotReady state even when my webhook rejects the request, If i reproduce for worker it went to NotReady state after restart.

[brandond]

I don't know, you'd probably want to enable audit logging on the apiserver and see what all is going on with your webhook and the node resources other than that one request.

[BhautikChudasama]

I can see 400 request in audit log server for verb create, I can see patch request as well for server.

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"820995f8-bf4a-4f08-b921-81b8c17448a5","stage":"ResponseComplete","requestURI":"/api/v1/nodes","verb":"create","user":{"username":"system:node:server","groups":["system:nodes","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"k3s/v1.31.5+k3s1 (linux/arm64) kubernetes/56ec5dd","objectRef":{"resource":"nodes","name":"server",
"apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"admission webhook \"webhook.xyz\" denied the request without explanation","code":400},"requestObject"....

Thanks i will deep dive.