node-exporter in CrashLoopBackOff state

[Louispettitt]

What happened?

Hello, I am new to K8s and have been tasked with setting up the monitoring stack for a cluster which is hosted on AWS EKS.

I followed steps in the Quick start and all of the pods ran apart from node-exporter, which gives out the following message when I run kubectl get pods -n monitoring:

• CrashLoopBackOff

I cannot for the life of me work out how to right this issue. I have gone through various issues to see if I can find something similar to help me with the fix but to no avail. I am probably missing something basic here because I am a K8s novice, so any advice that you can give me or if you can point me in the right direction for a fix would be much appreciated!

Im pretty sure we meet the pre-requisite requirements with the authentication and authorisation webhook and that the correct Kube-prom stack is being used (release-0.6) for our cluster.

Did you expect to see some different?

All pods to be working.

How to reproduce it (as minimally and precisely as possible):

Quick start

kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/

Environment

• Prometheus Operator version:

quay.io/prometheus-operator/prometheus-operator:v0.42.1

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

I am not sure how this was created but I can find out if need be.

I know we are using AWS EKS.

• Manifests:

node-exporter-daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/name: node-exporter
    app.kubernetes.io/version: v0.18.1
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: node-exporter
        app.kubernetes.io/version: v0.18.1
    spec:
      containers:
      - args:
        - --web.listen-address=127.0.0.1:9100
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        image: quay.io/prometheus/node-exporter:v0.18.1
        name: node-exporter
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/proc
          name: proc
          readOnly: false
        - mountPath: /host/sys
          name: sys
          readOnly: false
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      - args:
        - --logtostderr
        - --secure-listen-address=[$(IP)]:9100
        - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - --upstream=http://127.0.0.1:9100/
        env:
        - name: IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
        name: kube-rbac-proxy
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: https
        resources:
          limits:
            cpu: 20m
            memory: 40Mi
          requests:
            cpu: 10m
            memory: 20Mi
      hostNetwork: true
      hostPID: true
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: node-exporter
      tolerations:
      - operator: Exists
      volumes:
      - hostPath:
          path: /proc
        name: proc
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /
        name: root
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%

• Prometheus Operator Logs:

kubectl logs node-exporter-gd5s6 -n monitoring kube-rbac-proxy
I1116 15:47:22.595166 17823 main.go:213] Generating self signed cert as no cert is provided
I1116 15:48:47.990746 17823 main.go:243] Starting TCP socket on [192.168.65.3]:9100
I1116 15:48:47.991641 17823 main.go:250] Listening securely on [192.168.65.3]:9100
2020/11/16 15:48:53 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:07 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:22 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:37 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused
2020/11/16 15:49:52 http: proxy error: dial tcp 127.0.0.1:9100: connect: connection refused

kubectl logs node-exporter-gd5s6 -n monitoring node-exporter
failed to try resolving symlinks in path "/var/log/pods/monitoring_node-exporter-gd5s6_dcdde6b2-2f40-4fbb-b1a5-ff388057cbfb/node-exporter/8.log": lstat /var/log/pods/monitoring_node-exporter-gd5s6_dcdde6b2-2f40-4fbb-b1a5-ff388057cbfb/node-exporter/8.log: no such file or directory

kubectl describe pod node-exporter-gd5s6 -n monitoring

Name:         node-exporter-gd5s6
Namespace:    monitoring
Priority:     0
Node:         docker-desktop/192.168.65.3
Start Time:   Mon, 16 Nov 2020 15:47:01 +0000
Labels:       app.kubernetes.io/name=node-exporter
              app.kubernetes.io/version=v0.18.1
              controller-revision-hash=69cc88b55f
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           192.168.65.3
IPs:
  IP:           192.168.65.3
Controlled By:  DaemonSet/node-exporter
Containers:
  node-exporter:
    Container ID:  docker://e552750027a6c7d97457919857f85f41c04863ee76a309e4719f1d5c73752aee
    Image:         quay.io/prometheus/node-exporter:v0.18.1
    Image ID:      docker-pullable://quay.io/prometheus/node-exporter@sha256:a2f29256e53cc3e0b64d7a472512600b2e9410347d53cdc85b49f659c17e02ee
    Port:          <none>
    Host Port:     <none>
    Args:
      --web.listen-address=127.0.0.1:9100
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --no-collector.wifi
      --no-collector.hwmon
      --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      path / is mounted on / but it is not a shared or slave mount
      Exit Code:    128
      Started:      Mon, 16 Nov 2020 17:09:45 +0000
      Finished:     Mon, 16 Nov 2020 17:09:45 +0000
    Ready:          False
    Restart Count:  21
    Limits:
      cpu:     250m
      memory:  180Mi
    Requests:
      cpu:        102m
      memory:     180Mi
    Environment:  <none>
    Mounts:
      /host/proc from proc (rw)
      /host/root from root (ro)
      /host/sys from sys (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from node-exporter-token-l8d7w (ro)
  kube-rbac-proxy:
    Container ID:  docker://c79da379c9d53915a427ae9def06900293957f5c2dcd1614a0eaf17319a6995c
    Image:         quay.io/coreos/kube-rbac-proxy:v0.4.1
    Image ID:      docker-pullable://quay.io/coreos/kube-rbac-proxy@sha256:9d07c391aeb1a9d02eb4343c113ed01825227c70c32b3cae861711f90191b0fd
    Port:          9100/TCP
    Host Port:     9100/TCP
    Args:
      --logtostderr
      --secure-listen-address=[$(IP)]:9100
      --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
      --upstream=http://127.0.0.1:9100/
    State:          Running
      Started:      Mon, 16 Nov 2020 15:47:14 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     20m
      memory:  40Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Environment:
      IP:   (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from node-exporter-token-l8d7w (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
  root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
  node-exporter-token-l8d7w:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  node-exporter-token-l8d7w
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason   Age                   From                     Message
  ----     ------   ----                  ----                     -------
  Warning  BackOff  116s (x383 over 86m)  kubelet, docker-desktop  Back-off restarting failed container

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:

I am using Mac.

docker version
Client: Docker Engine - Community
Cloud integration: 1.0.2
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 16:58:31 2020
OS/Arch: darwin/amd64
Experimental: false

Also, some screen shots for reference from Docker and Prometheus UI:

Ignore that the pod name doesn't match the pods mentioned above as I re-deployed the stack after - error message still the same.

[paulfantom]

I believe you run into kubernetes/kubernetes#61058. For Linux, remounting / with mount --make-rshared / should solve it. I have no idea about darwin.

[Louispettitt]

So I'd need to find and use the equivalent of mount --make-rshared / for darwin?

Screenshot of attempt for reference:

[paulfantom]

Either this or deploy it in VM with Linux. For example in minikube.

[4n3w]

Curious: What side effects would changing the mountPropagation to None have on the NodeExporter? I have some objections from administrators that don't want to allow the mountPropagation to support `HostToContainer, ex:

   - mountPath: /host/root
      mountPropagation: None
      name: root
      readOnly: true

[paulfantom]

Without propagation for /host/root you won't get metrics for filesystem collector. Without propagation for /host/sys and /host/proc you won't get other metrics for node_exporter. In essence, you won't get all metrics for the node but only some metrics from a container.

[gliptak]

I see similar issue using Windows Docker Desktop (built-in Kubernetes). The message was different:

      Message:      path /sys is mounted on /sys but it is not a shared or slave mount
      Exit Code:    128

I used script from https://github.com/luksa/kubectl-plugins/ to run mount --make-rshared /sys to no avail.

#601

[paulfantom]

@gliptak I don't have much knowledge in Windows Docker Desktop, but it seems like you are hitting kubernetes/kubernetes#61058

[gliptak]

@paulfantom thank you for the pointer. I think that is where I found the mount command referenced

[Jamil100]

Was someone able to solve this problem?

[AlaaHamoudah]

There is a workaround to remove the mountPropagation to use the default one through creating a patch for promethous exporter damonset, here is the solution

[paulfantom]

Bear in mind that by disabling mount propagation node-exporter won't be able to collect all data needed for grafana dashboards and for alerts.

[TahoorYaseen]

@Louispettitt I ran into similar issue with docker desktop kubernetes cluster v1.22 and kube-prometheus release 0.9 and did you find the solution for it ?

[yamenkaba]

You can use this as a workaround,

• Find Daemon Set name of the node-exporter
  kubectl get ds -n <namespace>
• Patch it with the following command
  kubectl patch ds monitoring-prometheus-node-exporter --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

[Sounak3007]

thanks

[skymonil]

Thanks
For winodws i changed the second command to
kubectl patch ds prometheus-prometheus-node-exporter --namespace default --type "json" -p '[{"op": "remove", "path" : "/spec/template/spec/containers/0/volumeMounts/2/mountPropagation"}]'

And it ran successfully