`MutatingWebhookConfiguration` can lock you out

[Rudi3]

When running a netbird server and operator on the same cluster, the CNI plugin (calico in my case) may fail to start because of a webhook.
Because the webhook is calling the operator, this may also be a problem when just running the operator on said cluster.
This has been tested on a single node cluster - not sure if the behaviour would be different with multiple nodes.

From calico-kube-controllers:

Error creating: Internal error occurred: failed calling webhook "mpod-v1.netbird.io": failed to call webhook: Post "https://netbird-operator-kubernetes-operator-webhook-service.netbird-operator.svc:443/mutate--v1-pod?timeout=10s": dial tcp XXX.XXX.XXX.XXX:443: connect: connection refused

As I understand, because the CNI can't start, reaching the operator will be impossible.

kubectl edit MutatingWebhookConfiguration and setting failurePolicy: Ignore temporarily fixed the problem for me - but it may not be acceptable to set it to Ignore permanently.

[phillebaba]

I have experienced this myself once and limiting namespaces is not really enough. I have looked at two different options. We can either use match conditions to filter out pods without the injector annotations.

The other alternative is to switch over to using mutating admission policies which also supports match conditions but also does not depend on the controller running. For me this is the preferable future solution as we would no longer need to manage any webhooks.

I think option 1 is best short term until mutating admission policies become GA in many versions.

[Rudi3]

Yes, the match condition for the webhook looks like an easy fix for the time being. It seem to be a bit too early to switch to mutating admission policies.

Maybe this could work as well? I'm assuming that the label netbird.io/setup-key is present in all use-cases...

matchExpressions:
    - { key: netbird.io/setup-key, operator: Exists }

If this works, it could go into

kubernetes-operator/helm/kubernetes-operator/values.yaml

Line 20 in 3efacfd

namespaceSelectors: []

as well, but it likely should be default behaviour.