[SWCK]Injector don't inject agent

[zhuhao22]

I have a question about java agent injector not working, error log prompt does not have configmap 'skywalking-swck-java-agent-configmap'. How can i fix it?
My system is ctyunos，so i recompiled source package in arm64 environment and made the image of swck-operator.
This is deployment:
operator-bundle.zip

Log information of swck operator after enabled automatic injection agent:
2023-06-13T12:51:11.287Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2023-06-13T12:51:11.289Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=OAPServer", "path": "/mutate-operator-skywalking-apache-org-v1alpha1-oapserver"}
2023-06-13T12:51:11.289Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-operator-skywalking-apache-org-v1alpha1-oapserver"}
2023-06-13T12:51:11.289Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=OAPServer", "path": "/validate-operator-skywalking-apache-org-v1alpha1-oapserver"}
2023-06-13T12:51:11.289Z INFO controller-runtime.webhook registering webhook {"path": "/validate-operator-skywalking-apache-org-v1alpha1-oapserver"}
2023-06-13T12:51:11.289Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=UI", "path": "/mutate-operator-skywalking-apache-org-v1alpha1-ui"}
2023-06-13T12:51:11.289Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-operator-skywalking-apache-org-v1alpha1-ui"}
2023-06-13T12:51:11.289Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=UI", "path": "/validate-operator-skywalking-apache-org-v1alpha1-ui"}
2023-06-13T12:51:11.289Z INFO controller-runtime.webhook registering webhook {"path": "/validate-operator-skywalking-apache-org-v1alpha1-ui"}
2023-06-13T12:51:11.289Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=Fetcher", "path": "/mutate-operator-skywalking-apache-org-v1alpha1-fetcher"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-operator-skywalking-apache-org-v1alpha1-fetcher"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=Fetcher", "path": "/validate-operator-skywalking-apache-org-v1alpha1-fetcher"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/validate-operator-skywalking-apache-org-v1alpha1-fetcher"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=Storage", "path": "/mutate-operator-skywalking-apache-org-v1alpha1-storage"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-operator-skywalking-apache-org-v1alpha1-storage"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=Storage", "path": "/validate-operator-skywalking-apache-org-v1alpha1-storage"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/validate-operator-skywalking-apache-org-v1alpha1-storage"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=JavaAgent", "path": "/mutate-operator-skywalking-apache-org-v1alpha1-javaagent"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-operator-skywalking-apache-org-v1alpha1-javaagent"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=JavaAgent", "path": "/validate-operator-skywalking-apache-org-v1alpha1-javaagent"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/validate-operator-skywalking-apache-org-v1alpha1-javaagent"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=Satellite", "path": "/mutate-operator-skywalking-apache-org-v1alpha1-satellite"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-operator-skywalking-apache-org-v1alpha1-satellite"}
2023-06-13T12:51:11.290Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "operator.skywalking.apache.org/v1alpha1, Kind=Satellite", "path": "/validate-operator-skywalking-apache-org-v1alpha1-satellite"}
2023-06-13T12:51:11.290Z INFO controller-runtime.webhook registering webhook {"path": "/validate-operator-skywalking-apache-org-v1alpha1-satellite"}
2023-06-13T12:51:11.331Z INFO setup registering /mutate-v1-pod webhook
2023-06-13T12:51:11.331Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-v1-pod"}
2023-06-13T12:51:11.331Z INFO setup /mutate-v1-pod webhook is registered
2023-06-13T12:51:11.331Z INFO setup starting manager
I0613 12:51:11.331527 1 leaderelection.go:248] attempting to acquire leader lease istio-system/v1alpha1.swck.skywalking.apache.org...
2023-06-13T12:51:11.331Z INFO controller-runtime.webhook.webhooks starting webhook server
2023-06-13T12:51:11.331Z INFO starting metrics server {"path": "/metrics"}
2023-06-13T12:51:11.331Z INFO controller-runtime.certwatcher Updated current TLS certificate
2023-06-13T12:51:11.332Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 9443}
2023-06-13T12:51:11.332Z INFO controller-runtime.certwatcher Starting certificate watcher
E0613 12:51:11.362657 1 leaderelection.go:334] error initially creating leader election record: leases.coordination.k8s.io "v1alpha1.swck.skywalking.apache.org" already exists
2023-06-13T12:51:29.982Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/mutate-v1-pod", "UID": "b4b1070d-94fe-43fd-85e5-48f0b898bce6", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2023-06-13T12:51:30.032Z INFO injector don't inject agent
2023-06-13T12:51:30.032Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/mutate-v1-pod", "code": 200, "reason": "ok", "UID": "b4b1070d-94fe-43fd-85e5-48f0b898bce6", "allowed": true}
I0613 12:51:30.726628 1 leaderelection.go:258] successfully acquired lease istio-system/v1alpha1.swck.skywalking.apache.org
2023-06-13T12:51:30.726Z INFO controller.oapserver Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "OAPServer", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.726Z DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"istio-system","name":"v1alpha1.swck.skywalking.apache.org","uid":"8e63b75e-3d2a-40a6-912c-c2bb527b94c2","apiVersion":"v1","resourceVersion":"664984429"}, "reason": "LeaderElection", "message": "skywalking-operator-operator-74b5c8845b-82jxj_a8e18185-ae06-427d-9232-63757a651711 became leader"}
2023-06-13T12:51:30.726Z INFO controller.oapserver Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "OAPServer", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.726Z INFO controller.oapserver Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "OAPServer", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.726Z DEBUG events Normal {"object": {"kind":"Lease","namespace":"istio-system","name":"v1alpha1.swck.skywalking.apache.org","uid":"5a77c0e0-df95-4f37-8ff2-b9e8fefc7853","apiVersion":"coordination.k8s.io/v1","resourceVersion":"664984431"}, "reason": "LeaderElection", "message": "skywalking-operator-operator-74b5c8845b-82jxj_a8e18185-ae06-427d-9232-63757a651711 became leader"}
2023-06-13T12:51:30.726Z INFO controller.oapserver Starting Controller {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "OAPServer"}
2023-06-13T12:51:30.727Z INFO controller.pod Starting EventSource {"reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.pod Starting EventSource {"reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.fetcher Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Fetcher", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.pod Starting Controller {"reconciler group": "", "reconciler kind": "Pod"}
2023-06-13T12:51:30.727Z INFO controller.fetcher Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Fetcher", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.fetcher Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Fetcher", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.fetcher Starting Controller {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Fetcher"}
2023-06-13T12:51:30.727Z INFO controller.storage Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Storage", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.storage Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Storage", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.storage Starting Controller {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Storage"}
2023-06-13T12:51:30.727Z INFO controller.satellite Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Satellite", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.satellite Starting Controller {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Satellite"}
2023-06-13T12:51:30.727Z INFO controller.ui Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "UI", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.ui Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "UI", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.ui Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "UI", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.ui Starting EventSource {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "UI", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.ui Starting Controller {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "UI"}
2023-06-13T12:51:30.727Z INFO controller.configmap Starting EventSource {"reconciler group": "", "reconciler kind": "ConfigMap", "source": "kind source: /, Kind="}
2023-06-13T12:51:30.727Z INFO controller.configmap Starting Controller {"reconciler group": "", "reconciler kind": "ConfigMap"}
2023-06-13T12:51:35.432Z INFO controller.configmap Starting workers {"reconciler group": "", "reconciler kind": "ConfigMap", "worker count": 1}
2023-06-13T12:51:35.432Z INFO controller.pod Starting workers {"reconciler group": "", "reconciler kind": "Pod", "worker count": 1}
2023-06-13T12:51:35.432Z INFO controller.oapserver Starting workers {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "OAPServer", "worker count": 1}
2023-06-13T12:51:35.432Z INFO controller.fetcher Starting workers {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Fetcher", "worker count": 1}
2023-06-13T12:51:35.432Z INFO controller.satellite Starting workers {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Satellite", "worker count": 1}
2023-06-13T12:51:35.432Z INFO controller.ui Starting workers {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "UI", "worker count": 1}
2023-06-13T12:51:35.432Z INFO controller.storage Starting workers {"reconciler group": "operator.skywalking.apache.org", "reconciler kind": "Storage", "worker count": 1}
2023-06-14T01:06:01.081Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/mutate-v1-pod", "UID": "f5ad7cda-fb10-495c-8562-b394f2d41ca0", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2023-06-14T01:06:01.083Z ERROR injector can't get default configmap {"error": "ConfigMap "skywalking-swck-java-agent-configmap" not found"}
github.com/apache/skywalking-swck/operator/pkg/operator/injector.(*OverlayPlugins).execute
/home/yangzz/sky/operator/pkg/operator/injector/injection.go:140
github.com/apache/skywalking-swck/operator/pkg/operator/injector.(*OverlayAgent).execute
/home/yangzz/sky/operator/pkg/operator/injector/injection.go:118
github.com/apache/skywalking-swck/operator/pkg/operator/injector.(*OverlaySidecar).execute
/home/yangzz/sky/operator/pkg/operator/injector/injection.go:101
github.com/apache/skywalking-swck/operator/pkg/operator/injector.(*GetStrategy).execute
/home/yangzz/sky/operator/pkg/operator/injector/injection.go:83
github.com/apache/skywalking-swck/operator/pkg/operator/injector.(*InjectProcessData).Run
/home/yangzz/sky/operator/pkg/operator/injector/injection.go:228
github.com/apache/skywalking-swck/operator/pkg/operator/injector.(*JavaagentInjector).Handle
/home/yangzz/sky/operator/pkg/operator/injector/webhook.go:68
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle
/home/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/webhook.go:146
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
/home/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
/home/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
/usr/lib/golang/src/net/http/server.go:2109
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
/home/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
/usr/lib/golang/src/net/http/server.go:2109
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
/home/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
/usr/lib/golang/src/net/http/server.go:2109
net/http.(*ServeMux).ServeHTTP
/usr/lib/golang/src/net/http/server.go:2487
net/http.serverHandler.ServeHTTP
/usr/lib/golang/src/net/http/server.go:2947
net/http.(*conn).serve
/usr/lib/golang/src/net/http/server.go:1991
2023-06-14T01:06:01.087Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/mutate-v1-pod", "code": 200, "reason": "", "UID": "f5ad7cda-fb10-495c-8562-b394f2d41ca0", "allowed": true}
2023-06-14T01:14:55.826Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/mutate-v1-pod", "UID": "525c7be1-79d1-4fd2-b67e-47daec2746d6", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}

[hanahmily]

Which version are you working on? FWIK, the default configmap is deprecated at tall. @dashanji Is it right?

[dashanji]

Actually, the default configmap still exists as the injector will find it via https://github.com/apache/skywalking-swck/blob/06aeb614fca1883b7410ded2a59ecb6614f5ae1d/operator/pkg/operator/injector/injection.go#L182-L185.

@zhuhao22 It should be a bug as the injector will find the default configmap in the hardcoded namespace skywalking-swck-system. You can set the namespace as skywalking-swck-system to work around it.

[zhuhao22]

Thanks, i applied operator in default namespace and then injector worked normally. But i have new question, i applied the operator with helm that namespace as skywalking and it also worked.

[dashanji]

@zhuhao22 I'm not sure it's caused by whether you have permission to create configmap in the namespace. Whatever, as discussed with @hanahmily before, I will remove it to reduce the confusion, thanks for the report.

[dashanji]

I just opened an issue to fix it. FYI, #10931