bug: crashloopbackoff for "unknown" watches in 8.5.x #260
Description
Seeing crashes using 8.5.x version: kube-mgmt:8.5.8
kubectl get pods
NAME READY STATUS RESTARTS AGE
opa-79fcc56cdd-4q5l7 2/3 CrashLoopBackOff 6 (8s ago) 9m48s
opa-79fcc56cdd-hd96z 2/3 CrashLoopBackOff 5 (95s ago) 8m45s
kubectl logs opa-79fcc56cdd-nj79h -c kube-mgmt
time="2024-08-26T17:48:12Z" level=info msg="Policy/data ConfigMap processor connected to K8s: namespaces=[]"
E0826 17:48:12.424657 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
time="2024-08-26T17:48:12Z" level=info msg="Initial informer sync for v1/serviceaccounts completed, took 100.110118ms"
time="2024-08-26T17:48:12Z" level=info msg="Syncing v1/serviceaccounts."
time="2024-08-26T17:48:12Z" level=info msg="Initial informer sync for v1/namespaces completed, took 100.104679ms"
time="2024-08-26T17:48:12Z" level=info msg="Syncing v1/namespaces."
time="2024-08-26T17:48:12Z" level=info msg="Initial informer sync for v1/nodes completed, took 100.047158ms"
time="2024-08-26T17:48:12Z" level=info msg="Syncing v1/nodes."
time="2024-08-26T17:48:12Z" level=info msg="Loaded 133 resources of kind v1/serviceaccounts into OPA. Took 39.533653ms"
time="2024-08-26T17:48:12Z" level=info msg="Loaded 36 resources of kind v1/namespaces into OPA. Took 40.737732ms"
time="2024-08-26T17:48:12Z" level=info msg="Loaded 16 resources of kind v1/nodes into OPA. Took 47.807383ms"
E0826 17:48:13.397307 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 17:48:16.332353 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 17:48:21.992081 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 17:48:30.780212 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 17:48:55.304518 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknownUsing these args:
- args:
- --replicate-cluster=v1/namespaces
- --replicate-cluster=v1/nodes
- --replicate-cluster=v1/serviceaccounts
- --policies=opa
- --require-policy-label=true
- --opa-url=https://127.0.0.1:8443/v1
- --opa-allow-insecure=true
We see the same kind of errors in 8.4.0 but no crashes.
kubectl logs opa-c7886947f-kbg6b -c kube-mgmt
time="2024-08-26T16:30:44Z" level=info msg="Policy/data ConfigMap processor connected to K8s: namespaces=[opa]"
E0826 16:30:44.465353 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/admission.rego, err=code invalid_parameter: error(s) occurred while compiling module(s)"
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/containers.rego, err=<nil>"
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/helpers.rego, err=<nil>"
time="2024-08-26T16:30:44Z" level=info msg="Initial informer sync for v1/namespaces completed, took 101.038623ms"
time="2024-08-26T16:30:44Z" level=info msg="Syncing v1/namespaces."
time="2024-08-26T16:30:44Z" level=info msg="Initial informer sync for v1/serviceaccounts completed, took 101.080936ms"
time="2024-08-26T16:30:44Z" level=info msg="Syncing v1/serviceaccounts."
time="2024-08-26T16:30:44Z" level=info msg="Initial informer sync for v1/nodes completed, took 100.889071ms"
time="2024-08-26T16:30:44Z" level=info msg="Syncing v1/nodes."
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/image.rego, err=<nil>"
time="2024-08-26T16:30:44Z" level=info msg="Loaded 36 resources of kind v1/namespaces into OPA. Took 32.966228ms"
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/istio.rego, err=<nil>"
time="2024-08-26T16:30:44Z" level=info msg="Loaded 17 resources of kind v1/nodes into OPA. Took 65.188539ms"
time="2024-08-26T16:30:44Z" level=info msg="Loaded 133 resources of kind v1/serviceaccounts into OPA. Took 119.451253ms"
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/main.rego, err=code invalid_parameter: error(s) occurred while compiling module(s)"
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies/privileges.rego, err=<nil>"
time="2024-08-26T16:30:44Z" level=info msg="Added policy opa/policies-preprod-artifactory-override/allow_artifactory_preprod.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/admission.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/containers.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/helpers.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/image.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/istio.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/main.rego, err=<nil>"
time="2024-08-26T16:30:45Z" level=info msg="Added policy opa/policies/privileges.rego, err=<nil>"
E0826 16:30:45.569381 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 16:30:48.419411 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 16:30:53.343317 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 16:31:03.178708 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 16:31:17.614971 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 16:32:00.389407 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
E0826 16:32:45.024632 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
Opa RBAC:
kubectl auth can-i --as=system:serviceaccount:opa:opa-sa --list
Warning: the list may be incomplete: webhook authorizer does not support user rule resolution
Resources Non-Resource URLs Resource Names Verbs
workflowtaskresults.argoproj.io [] [] [create delete deletecollection get list patch update watch]
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
nodes [] [] [get list patch watch]
configmaps [] [] [get list watch update patch]
bindings [] [] [get list watch]
endpoints [] [] [get list watch]
events [] [] [get list watch]
limitranges [] [] [get list watch]
namespaces/status [] [] [get list watch]
namespaces [] [] [get list watch]
persistentvolumeclaims/status [] [] [get list watch]
persistentvolumeclaims [] [] [get list watch]
pods/log [] [] [get list watch]
pods/status [] [] [get list watch]
pods [] [] [get list watch]
replicationcontrollers/scale [] [] [get list watch]
replicationcontrollers/status [] [] [get list watch]
replicationcontrollers [] [] [get list watch]
resourcequotas/status [] [] [get list watch]
resourcequotas [] [] [get list watch]
serviceaccounts [] [] [get list watch]
services/status [] [] [get list watch]
services [] [] [get list watch]
controllerrevisions.apps [] [] [get list watch]
daemonsets.apps/status [] [] [get list watch]
daemonsets.apps [] [] [get list watch]
deployments.apps/scale [] [] [get list watch]
deployments.apps/status [] [] [get list watch]
deployments.apps [] [] [get list watch]
replicasets.apps/scale [] [] [get list watch]
replicasets.apps/status [] [] [get list watch]
replicasets.apps [] [] [get list watch]
statefulsets.apps/scale [] [] [get list watch]
statefulsets.apps/status [] [] [get list watch]
statefulsets.apps [] [] [get list watch]
clusterworkflowtemplates.argoproj.io/finalizers [] [] [get list watch]
clusterworkflowtemplates.argoproj.io [] [] [get list watch]
cronworkflows.argoproj.io/finalizers [] [] [get list watch]
cronworkflows.argoproj.io [] [] [get list watch]
workflowartifactgctasks.argoproj.io [] [] [get list watch]
workfloweventbindings.argoproj.io/finalizers [] [] [get list watch]
workfloweventbindings.argoproj.io [] [] [get list watch]
workflows.argoproj.io/finalizers [] [] [get list watch]
workflows.argoproj.io [] [] [get list watch]
workflowtasksets.argoproj.io/finalizers [] [] [get list watch]
workflowtasksets.argoproj.io [] [] [get list watch]
workflowtemplates.argoproj.io/finalizers [] [] [get list watch]
workflowtemplates.argoproj.io [] [] [get list watch]
horizontalpodautoscalers.autoscaling/status [] [] [get list watch]
horizontalpodautoscalers.autoscaling [] [] [get list watch]
cronjobs.batch/status [] [] [get list watch]
cronjobs.batch [] [] [get list watch]
jobs.batch/status [] [] [get list watch]
jobs.batch [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
daemonsets.extensions/status [] [] [get list watch]
daemonsets.extensions [] [] [get list watch]
deployments.extensions/scale [] [] [get list watch]
deployments.extensions/status [] [] [get list watch]
deployments.extensions [] [] [get list watch]
ingresses.extensions/status [] [] [get list watch]
ingresses.extensions [] [] [get list watch]
networkpolicies.extensions [] [] [get list watch]
replicasets.extensions/scale [] [] [get list watch]
replicasets.extensions/status [] [] [get list watch]
replicasets.extensions [] [] [get list watch]
replicationcontrollers.extensions/scale [] [] [get list watch]
nodes.metrics.k8s.io [] [] [get list watch]
pods.metrics.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io/status [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
networkpolicies.networking.k8s.io [] [] [get list watch]
poddisruptionbudgets.policy/status [] [] [get list watch]
poddisruptionbudgets.policy [] [] [get list watch]
[/.well-known/openid-configuration/] [] [get]
[/.well-known/openid-configuration] [] [get]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/openid/v1/jwks/] [] [get]
[/openid/v1/jwks] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]
Not sure what this watch is?
E0826 17:48:12.424657 1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: unknown
Also, see this difference in the logs from 8.5.x vs. 8.4.x and 8.x:
8.5.x
time="2024-08-26T17:48:12Z" level=info msg="Policy/data ConfigMap processor connected to K8s: namespaces=[]"
8.4.x and below
time="2024-08-26T16:30:44Z" level=info msg="Policy/data ConfigMap processor connected to K8s: namespaces=[opa]"