POD not getting sidecar even though injection is requested

[borajuanjo]

What's going on?

PODs are not getting sidecars, even though the injection is requested. Funnily I got it to work once on a training cluster, and then I deleted the cluster and I can't get it to work again.

Expected Behavior

PODs should get created with sidecar.

Reproducer

I just went through the documentation step by step, and I can't identify what I'm doing wrong, or what I did differently that one time I got it to work.

Here are some logs for when the debian-debug POD gets deployed, but no sidecar.

10.64.4.1 - - [23/Mar/2022:15:16:18 +0000] "GET /health HTTP/2.0" 200 12 "" "kube-probe/1.21"
I0323 15:16:21.450826       1 webhook.go:510] AdmissionReview for Kind=/v1, Kind=Pod, Namespace=default Name= () UID=37184454-a6e7-4f35-be04-8eeaedf85265 patchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 4b17f21d-590c-4d37-acf2-5096af5e70cd [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
I0323 15:16:21.450862       1 webhook.go:174] Pod / annotation injector.tumblr.com/request=test1 requesting sidecar config test1:latest
I0323 15:16:21.450961       1 webhook.go:548] AdmissionResponse: patch=[{"op":"add","path":"/spec/containers/0/env","value":[{"name":"HELLO","value":"world"}]},{"op":"add","path":"/spec/containers/0/env/-","value":{"name":"TEST","value":"test_that"}},{"op":"add","path":"/spec/containers/0/volumeMounts/-","value":{"name":"test-vol","mountPath":"/tmp/test"}},{"op":"add","path":"/spec/containers/-","value":{"name":"sidecar-nginx","image":"nginx:1.12.2","ports":[{"containerPort":80}],"env":[{"name":"ENV_IN_SIDECAR","value":"test-in-sidecar"},{"name":"HELLO","value":"world"},{"name":"TEST","value":"test_that"}],"resources":{},"volumeMounts":[{"name":"test-vol","mountPath":"/tmp/test"}],"imagePullPolicy":"IfNotPresent"}},{"op":"add","path":"/spec/volumes/-","value":{"name":"test-vol","configMap":{"name":"test-config"}}},{"op":"add","path":"/metadata/annotations/injector.tumblr.com~1status","value":"injected"}]
I0323 15:16:21.451004       1 webhook.go:626] Ready to write reponse ...
10.64.3.7 - - [23/Mar/2022:15:16:21 +0000] "POST /mutate?timeout=10s HTTP/1.1" 200 1237 "" "kube-apiserver-admission"
10.64.4.1 - - [23/Mar/2022:15:16:28 +0000] "GET /health HTTP/2.0" 200 12 "" "kube-probe/1.21"

Here's the debian-debug POD with no sidecar.

$ kubectl get po | grep debian

debian-debug                                             1/1     Running   0          55m

Version Deets

• Kubernetes Version:

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.9-dispatcher", GitCommit:"2a8027f41d28b788b001389f3091c245cd0a9a60", GitTreeState:"clean", BuildDate:"2022-01-21T20:31:13Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.9-gke.1002", GitCommit:"f87f9d952767b966e72a4bd75afea25dea187bbf", GitTreeState:"clean", BuildDate:"2022-02-25T18:12:32Z", GoVersion:"go1.16.12b7", Compiler:"gc", Platform:"linux/amd64"}

• k8s-sidecar-injector Version: latest (as of March 23rd 2022)

[wiebeck]

Very same here. Running on Azure Kubernetes with K8s v1.23.5

[karolmalyszko]

Got the same issue on AWS EKS v1.21, but made it to work. Looks like main problem lies in MutatingWebhookConfiguration. In original example manifests, webhook config had

• apiVersion: admissionregistration.k8s.io/v1beta1

which was refused by K8s in my case, had to modify it to

• apiVersion: admissionregistration.k8s.io/v1

To conform with K8s requirements for this API version, I had to add additional keys for sideEffects: and admissionReviewVersions: (K8s docs here).

Two WAG's followed :: first one for sideEffects where i put 'None', second one for admissionReviewVersions where i started with 'v1'. This led to successful deployment of MutatingWebhookConfiguration, but the sidecars were not being added to test pods / deployments, same as for both of you. Search through codebase got me to webhook.go line 613 ::
admissionReview := v1beta1.AdmissionReview{}

Change of admissionReviewVersions from v1 to v1beta1 got the problem solved, sidecar started as expected.
Here's a gist with example mutating-webhook-configuration.yaml

Hope this helps.

[mindovermiles262]

So helpful! Thank you! I also had this issue.

I've taken your image updates and pushed it to docker hub if anyone else would like to use it. mindovermiles262/sidecar-injector

Change the deployment.yaml image to:

spec:
  template:
    spec:
      containers:
      - name: "k8s-sidecar-injector"
        [ ... ]
        # image: tumblr/k8s-sidecar-injector:latest   <-- Comment this line out, and update
        image: mindovermiles262/sidecar-injector

Note, this image works with the updated mutating webhook configuration where the apiVersion: admissionregistration.k8s.io/v1 and webhooks.sideEffects: None