How to monitor ssd status smartmon

[runningman84]

Due to recent ssd outages I have to use tools like this to get some early indication of ssd problems:
https://github.com/reefland/smartmon_nvme
How would such tool work in a talos cluster?

[packruler]

Did you ever find any info on this?

[champloo]

If you just want to do a manual check now and again, you can use a debug container to run a command either smartctl or nvme smart-log like so

kubectl debug -n kube-system -it --image alpine node/your_node_name --profile=sysadmin -- sh -c "apk add nvme-cli && nvme smart-log /dev/nvme0"

Note that you need to specify --profile=sysadmin in order for the host drives to be mounted in the debug container.

If you want to incorporate smart data into your monitoring stack I guess you could use https://github.com/prometheus-community/smartctl_exporter or https://github.com/E4-Computer-Engineering/nvme-exporter. But I guess you are asking how you get say smartmontools or nvme-cli onto the host. Not sure, perhaps a DeamonSet that mounts the hostPath disks. I've not tried this.

I've been using the manual method to inspect the drives since I recently I also had ssd problems.

[champloo]

Also worth reading this post about not doing smart checks too frequently rook/rook#12652 (comment)

[champloo]

You got me curious so here is a DeamonSet that will run the smartctl on each node. You could then modify this to save to a file and use the exporters to scrape the data

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: smartctl-runner
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: smartctl-runner
  template:
    metadata:
      labels:
        app: smartctl-runner
    spec:
      hostPID: true
      containers:
      - name: smartctl
        image: alpine:latest
        securityContext:
          privileged: true
        command: ["/bin/sh", "-c"]
        args:
          - |
            apk add --no-cache smartmontools;
            while true; do
              echo "Running smartctl on $(hostname) at $(date)";
              for disk in /dev/sd? /dev/nvme?n?; do
                echo "=== $disk ===";
                smartctl -a "$disk" || echo "Failed to check $disk";
              done;
              sleep 3600;
            done
        volumeMounts:
          - name: dev
            mountPath: /dev
      volumes:
        - name: dev
          hostPath:
            path: /dev
            type: Directory
      tolerations:
        - operator: "Exists"

[runningman84]

The smartctl exporter helm release works just fine:

# yaml-language-server: $schema=https://kubernetes-schemas.pages.dev/helm.toolkit.fluxcd.io/helmrelease_v2.json
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: smartctl-exporter
  namespace: monitoring
spec:
  interval: 60m
  chart:
    spec:
      chart: prometheus-smartctl-exporter
      version: 0.8.0
      sourceRef:
        kind: HelmRepository
        name: prometheus-community-charts
        namespace: flux-system
  install:
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      strategy: rollback
      retries: 3
  values:
    config:
      devices:
        - /dev/nvme0n1
        - /dev/sda
        - /dev/sdb
    serviceMonitor:
      enabled: true
    prometheusRules:
      enabled: true