[nr-k8s-otel-collector] chore: exclude container overlay path in OKE (#2114)

kondracek-nr · web-flow · commit cff10aaac5ab · 2026-02-11T11:52:36.000-08:00
#### What this PR does / why we need it:
Deploying the nr-k8s-otel-collector chart to an OKE cluster will result
in a functional DaemonSet, but the DaemonSet logs will include the
following log on each scrape interval:
```
2026-02-10T19:11:45.349Z	error	scraperhelper@v0.142.0/obs_metrics.go:61	Error scraping metrics	{"resource": {"service.instance.id": "0423df01-041e-4e36-aaa0-284cd7dba542", "service.name": "nrdot-collector-k8s", "service.version": "1.8.0"}, "otelcol.component.id": "hostmetrics", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "failed to read usage at /hostfs/var/lib/containers/storage/overlay: permission denied"}
go.opentelemetry.io/collector/scraper/scraperhelper.wrapObsMetrics.func1
	go.opentelemetry.io/collector/scraper/scraperhelper@v0.142.0/obs_metrics.go:61
go.opentelemetry.io/collector/scraper.ScrapeMetricsFunc.ScrapeMetrics
	go.opentelemetry.io/collector/scraper@v0.142.0/metrics.go:24
go.opentelemetry.io/collector/scraper/scraperhelper.scrapeMetrics
	go.opentelemetry.io/collector/scraper/scraperhelper@v0.142.0/controller.go:256
go.opentelemetry.io/collector/scraper/scraperhelper.NewMetricsController.func1
	go.opentelemetry.io/collector/scraper/scraperhelper@v0.142.0/controller.go:228
go.opentelemetry.io/collector/scraper/scraperhelper.(*controller[...]).startScraping.func1
	go.opentelemetry.io/collector/scraper/scraperhelper@v0.142.0/controller.go:171
```

**This error also occurs in the vanilla opentelemetry collector chart
when deployed to OKE**. OKE uses the CRI-O container runtime, and the
nonroot user cannot access this particular path. Excluding the mount
path when the user is running as nonroot will prevent the error from
appearing in logs. If the user chooses to run as root or in privileged
mode, the path will be allowed.


#### Which issue this PR fixes
*(optional, in `fixes #&lt;issue number&gt;(, fixes #&lt;issue_number&gt;, ...)`
format, will close that issue when PR gets merged)*
  - fixes #

#### Special notes for your reviewer:

#### Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove
unrelated fields.]
- [x] Chart Version bumped
- [x] Title of the PR starts with chart name (e.g. `[mychartname]`)

# Release Notes to Publish (nr-k8s-otel-collector)
If this PR contains changes in `nr-k8s-otel-collector`, please complete
the following section. All other charts should ignore this section.

&lt;!--BEGIN-RELEASE-NOTES--&gt;
## 🚀 What's Changed
* Excludes the var/lib/containers/storage/overlay path from hostmetrics
receiver when running as nonroot, which was throwing permission denied
errors in OKE.
&lt;!--END-RELEASE-NOTES--&gt;
diff --git a/charts/nr-k8s-otel-collector/Chart.yaml b/charts/nr-k8s-otel-collector/Chart.yaml
@@ -17,7 +17,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.10.4
+version: 0.10.5
 
 
 dependencies:
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/clusterrole.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/clusterrole.yaml
@@ -9,7 +9,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 rules:
   - apiGroups:
     - ""
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/clusterrolebinding.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/clusterrolebinding.yaml
@@ -9,7 +9,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 subjects:
   - kind: ServiceAccount
     name: nr-k8s-otel-collector
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/daemonset-configmap.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/daemonset-configmap.yaml
@@ -10,7 +10,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 data:
   daemonset-config.yaml: |
     receivers:
@@ -43,6 +43,12 @@ data:
             metrics:
               system.filesystem.utilization:
                 enabled: true
+            # Exclude container storage overlay mount to avoid permission errors when running as non-root
+            # This is only relevant for CRI-O container runtime (used by OKE, OpenShift, etc.)
+            exclude_mount_points:
+              mount_points:
+                - ^/var/lib/containers/storage/overlay$
+              match_type: regexp
           disk:
             metrics:
               system.disk.merged:
@@ -607,7 +613,7 @@ data:
             value: <cluser_name>
           - key: "newrelic.chart.version"
             action: upsert
-            value: 0.10.4
+            value: 0.10.5
           - key: newrelic.entity.type
             action: upsert
             value: "k8s"
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/daemonset.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/daemonset.yaml
@@ -10,7 +10,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 spec:
   selector:
     matchLabels:
@@ -24,7 +24,7 @@ spec:
         app.kubernetes.io/name: nr-k8s-otel-collector
         component: daemonset
       annotations:
-        checksum/config: d2b3a75ff0824bb1a9ba7c3fadcfed978f9cfbc98ab21cace51a0a4d4aaba8b9
+        checksum/config: 6c5ce1a6f82c834eb8b42383bd1fb5181cea24c7ccdbd725ce105076e2cb1094
     spec:
       serviceAccountName: nr-k8s-otel-collector
       initContainers:
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/deployment-configmap.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/deployment-configmap.yaml
@@ -10,7 +10,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 data:
   deployment-config.yaml: |
     receivers:
@@ -516,7 +516,7 @@ data:
             value: <cluser_name>
           - key: "newrelic.chart.version"
             action: upsert
-            value: 0.10.4
+            value: 0.10.5
           - key: newrelic.entity.type
             action: upsert
             value: "k8s"
@@ -534,7 +534,7 @@ data:
             value: <cluser_name>
           - key: "newrelic.chart.version"
             action: upsert
-            value: 0.10.4
+            value: 0.10.5
 
       transform/events:
         log_statements:
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/deployment.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/deployment.yaml
@@ -10,7 +10,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 spec:
   replicas: 1
   minReadySeconds: 5
@@ -26,7 +26,7 @@ spec:
         app.kubernetes.io/name: nr-k8s-otel-collector
         component: deployment
       annotations:
-        checksum/config: 83286d160925df1d451b4b722c4a301886b5071459be13ed88cf2918d817233d
+        checksum/config: cdcda6b0e1e1481ee0d0ee12c277157e965d9a3c2452b4e275e8c13fb218aa59
     spec:
       serviceAccountName: nr-k8s-otel-collector
       containers:
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/secret.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/secret.yaml
@@ -10,6 +10,6 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 data:
   licenseKey: PE5SX2xpY2Vuc2VLZXk+
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/service.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/service.yaml
@@ -10,7 +10,7 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
 spec:
   type: ClusterIP
   ports:
diff --git a/charts/nr-k8s-otel-collector/examples/k8s/rendered/serviceaccount.yaml b/charts/nr-k8s-otel-collector/examples/k8s/rendered/serviceaccount.yaml
@@ -10,5 +10,5 @@ metadata:
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: nr-k8s-otel-collector
     app.kubernetes.io/version: 1.2.0
-    helm.sh/chart: nr-k8s-otel-collector-0.10.4
+    helm.sh/chart: nr-k8s-otel-collector-0.10.5
   annotations:
diff --git a/charts/nr-k8s-otel-collector/templates/_security_context.tpl b/charts/nr-k8s-otel-collector/templates/_security_context.tpl
@@ -49,3 +49,27 @@ A helper to return the container security context to apply to the daemonset.
   {{end}}
 {{- end -}}
 {{- end -}}
+
+{{- /*
+A helper to determine if the daemonset should exclude the container storage overlay mount point.
+This exclusion is only relevant for the CRI-O container runtime (used by OKE, OpenShift, etc.)
+where non-root users cannot access /var/lib/containers/storage/overlay due to permission restrictions.
+Returns "true" if running as non-root and non-privileged (needs exclusion to avoid permission errors).
+Returns empty string if running as root or privileged (can access the mount point).
+Respects precedence: daemonset.containerSecurityContext > containerSecurityContext > global.containerSecurityContext
+*/ -}}
+{{- define "nrKubernetesOtel.daemonset.excludeContainerStorageOverlay" -}}
+{{- $securityContext := dict -}}
+{{- if .Values.daemonset.containerSecurityContext -}}
+  {{- $securityContext = .Values.daemonset.containerSecurityContext -}}
+{{- else if .Values.containerSecurityContext -}}
+  {{- $securityContext = .Values.containerSecurityContext -}}
+{{- else if .Values.global.containerSecurityContext -}}
+  {{- $securityContext = .Values.global.containerSecurityContext -}}
+{{- end -}}
+{{- $runAsRoot := and (hasKey $securityContext "runAsUser") (eq ($securityContext.runAsUser | int) 0) -}}
+{{- $isPrivileged := $securityContext.privileged -}}
+{{- if not (or $runAsRoot $isPrivileged) -}}
+true
+{{- end -}}
+{{- end -}}
diff --git a/charts/nr-k8s-otel-collector/templates/daemonset-configmap.yaml b/charts/nr-k8s-otel-collector/templates/daemonset-configmap.yaml
@@ -42,6 +42,14 @@ data:
             metrics:
               system.filesystem.utilization:
                 enabled: true
+            {{- if include "nrKubernetesOtel.daemonset.excludeContainerStorageOverlay" . }}
+            # Exclude container storage overlay mount to avoid permission errors when running as non-root
+            # This is only relevant for CRI-O container runtime (used by OKE, OpenShift, etc.)
+            exclude_mount_points:
+              mount_points:
+                - ^/var/lib/containers/storage/overlay$
+              match_type: regexp
+            {{- end }}
           disk:
             metrics:
               system.disk.merged:
diff --git a/charts/nr-k8s-otel-collector/tests/container_storage_overlay_exclusion_test.yaml b/charts/nr-k8s-otel-collector/tests/container_storage_overlay_exclusion_test.yaml
@@ -0,0 +1,185 @@
+suite: containerStorageOverlayExclusion
+templates:
+  - templates/daemonset-configmap.yaml
+release:
+  name: my-release
+  namespace: my-namespace
+tests:
+  - it: excludes container storage overlay mount point when running as non-root (default)
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      daemonset:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: true
+          runAsUser: 1001
+    asserts:
+      - matchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: does not exclude container storage overlay mount point when running as root
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      daemonset:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: false
+          runAsUser: 0
+    asserts:
+      - notMatchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: excludes container storage overlay mount point when runAsNonRoot is false but runAsUser is non-root
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      daemonset:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: false
+          runAsUser: 1001
+    asserts:
+      - matchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: does not exclude container storage overlay mount point when privileged is true
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      daemonset:
+        containerSecurityContext:
+          privileged: true
+          runAsNonRoot: true
+          runAsUser: 1001
+    asserts:
+      - notMatchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: does not exclude container storage overlay mount point when privileged and running as root
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      daemonset:
+        containerSecurityContext:
+          privileged: true
+          runAsNonRoot: false
+          runAsUser: 0
+    asserts:
+      - notMatchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: excludes when using default containerSecurityContext values
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+    asserts:
+      - matchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: uses fallback to top-level containerSecurityContext when daemonset.containerSecurityContext not set
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      containerSecurityContext:
+        privileged: false
+        runAsNonRoot: true
+        runAsUser: 1001
+      daemonset:
+        containerSecurityContext:
+    asserts:
+      - matchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: respects daemonset security context over top-level when running as root
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      containerSecurityContext:
+        privileged: false
+        runAsNonRoot: true
+        runAsUser: 1001
+      daemonset:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: false
+          runAsUser: 0
+    asserts:
+      - notMatchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: uses global containerSecurityContext when daemonset and top-level not set
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      global:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: true
+          runAsUser: 1001
+      daemonset:
+        containerSecurityContext:
+      containerSecurityContext:
+    asserts:
+      - matchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: respects daemonset over global when both set
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      global:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: true
+          runAsUser: 1001
+      daemonset:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: false
+          runAsUser: 0
+    asserts:
+      - notMatchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
+
+  - it: respects top-level over global when both set
+    set:
+      cluster: my-cluster
+      licenseKey: us-whatever
+      global:
+        containerSecurityContext:
+          privileged: false
+          runAsNonRoot: false
+          runAsUser: 0
+      containerSecurityContext:
+        privileged: false
+        runAsNonRoot: true
+        runAsUser: 1001
+      daemonset:
+        containerSecurityContext:
+    asserts:
+      - matchRegex:
+          path: data["daemonset-config.yaml"]
+          pattern: 'exclude_mount_points:\s+mount_points:\s+- \^/var/lib/containers/storage/overlay\$'
+        template: templates/daemonset-configmap.yaml
