Merge pull request #1400 from mickem/feature/modernized_docker_2

Modernize CheckDocker and add stats, restarts and disk-usage checks

Author: mickem

docs/samples/CheckDocker_check_docker_desc.md

@@ -0,0 +1,41 @@
+#### About `check_docker`
+
+`check_docker` checks the state of docker containers via the local daemon
+socket (`/var/run/docker.sock` on Linux, the `\\.\pipe\docker_engine` named
+pipe on Windows; configurable via the `endpoint` setting or `host=`). It works
+against any daemon speaking the docker API, including podman's compatibility
+socket.
+
+Three ways to use it:
+
+* `container=<name>` (repeatable) — the "is my container running" mode: only
+  the named containers are checked, and a name the daemon does not know gets
+  the synthetic state `missing`, so it trips the default critical instead of
+  silently disappearing from the listing.
+* No arguments — lists running containers; nothing trips the default
+  thresholds, so this is an inventory-style check.
+* `all=true` — also includes stopped containers (like `docker ps -a`); any
+  non-running container then trips the default critical
+  (`container_state != 'running'`), so combine it with `filter=` on hosts
+  where exited one-shot containers are expected.
+
+Available keywords (for `filter=` / `warning=` / `critical=` / syntax):
+
+| Keyword           | Description                                                                  |
+|-------------------|------------------------------------------------------------------------------|
+| `names`           | Container name(s), comma separated                                           |
+| `container_state` | `created`, `restarting`, `running`, `removing`, `paused`, `exited`, `dead` or `missing` |
+| `status`          | Human readable status, e.g. `Up 3 hours (healthy)`                           |
+| `health`          | Health-check state: `healthy`, `unhealthy`, `starting`, or empty without a health check |
+| `image`           | Image the container was created from                                         |
+| `image_id`        | Id of that image                                                             |
+| `id`              | Container id                                                                 |
+| `command`         | Command the container runs                                                   |
+| `ip`              | First IP address on any attached network                                     |
+| `ports`           | Published/exposed ports, e.g. `0.0.0.0:8080->80/tcp`                         |
+| `labels`          | Container labels as `key=value`, comma separated                             |
+| `created`         | When the container was created (date)                                        |
+
+The daemon endpoint is restricted to a local named pipe / absolute socket path:
+a UNC path in `host=` would make a Windows host authenticate to a remote SMB
+server with the service account, so anything non-local is refused outright.

docs/samples/CheckDocker_check_docker_df_desc.md

@@ -0,0 +1,34 @@
+#### About `check_docker_df`
+
+`check_docker_df` reports docker's disk usage the way `docker system df`
+does — images, container writable layers, volumes and the build cache — plus
+what a prune would reclaim: unused images (their non-shared size), stopped
+containers, unreferenced volumes and idle build cache.
+
+The daemon computes this by walking every image layer, container filesystem
+and volume, so **the request is slow on large hosts** (seconds to minutes).
+The check therefore defaults to six times the module's normal timeout; don't
+schedule it more often than you need it.
+
+Available keywords (for `filter=` / `warning=` / `critical=` / syntax):
+
+| Keyword                   | Description                                                  |
+|---------------------------|--------------------------------------------------------------|
+| `images` / `unused_images`| Number of images / images no container uses                  |
+| `images_size`             | Disk used by images (unique layer sizes)                     |
+| `images_reclaimable`      | Freed by pruning unused images                               |
+| `containers`              | Number of containers (running and stopped)                   |
+| `containers_size`         | Disk used by container writable layers                       |
+| `containers_reclaimable`  | Freed by pruning stopped containers                          |
+| `volumes` / `unused_volumes` | Number of volumes / volumes no container references       |
+| `volumes_size`            | Disk used by volumes                                         |
+| `volumes_reclaimable`     | Freed by pruning unused volumes                              |
+| `build_cache_size`        | Disk used by the build cache                                 |
+| `build_cache_reclaimable` | Freed by pruning the idle build cache                        |
+| `total_size`              | Everything above combined                                    |
+| `total_reclaimable`       | Everything a full prune would free                           |
+| `message`                 | The human readable summary line                              |
+
+All `*_size` / `*_reclaimable` keywords are byte-sized: thresholds take a unit
+(`500M`, `10G`, `1T`); a bare number is rejected. No default thresholds — a
+responding daemon is OK until you add limits.

docs/samples/CheckDocker_check_docker_df_samples.md

@@ -0,0 +1,27 @@
+**Show docker disk usage (like `docker system df`):**
+
+```
+check_docker_df
+OK: images 71 (8.216GB), containers 19 (388KB), volumes 6 (4.517GB), build cache 12.808GB, reclaimable 21.479GB
+```
+
+**Alert when a prune would free a lot of space (size thresholds take a unit):**
+
+```
+check_docker_df "warning=total_reclaimable > 10G" "critical=total_reclaimable > 50G"
+WARNING: images 71 (8.216GB), containers 19 (388KB), volumes 6 (4.517GB), build cache 12.808GB, reclaimable 21.479GB|'docker reclaimable'=23063090871B;10737418240;53687091200
+```
+
+**Alert on unused images piling up:**
+
+```
+check_docker_df "warning=unused_images > 20"
+OK: images 71 (8.216GB), containers 19 (388KB), volumes 6 (4.517GB), build cache 12.808GB, reclaimable 21.479GB|'docker unused images'=63;20;0
+```
+
+**Watch a specific category, e.g. the build cache:**
+
+```
+check_docker_df "warning=build_cache_size > 20G" "detail-syntax=build cache %(build_cache_size) (reclaimable %(build_cache_reclaimable))"
+OK: build cache 13752766549 (reclaimable 13752766549)|'docker build cache'=13752766549B;21474836480;0
+```

docs/samples/CheckDocker_check_docker_info_desc.md

@@ -0,0 +1,23 @@
+#### About `check_docker_info`
+
+`check_docker_info` checks the docker daemon itself: it reads `/info` from the
+local daemon socket and reports the server version, host name and the
+container/image counts. A responding daemon is **OK** unless you threshold the
+counts; an unreachable daemon is **UNKNOWN** with the transport error — which
+makes this the natural "is docker itself healthy" companion to per-container
+`check_docker` checks.
+
+Available keywords (for `filter=` / `warning=` / `critical=` / syntax):
+
+| Keyword      | Description                            |
+|--------------|----------------------------------------|
+| `version`    | Docker server version                  |
+| `name`       | Daemon host name                       |
+| `os`         | Operating system the daemon runs on    |
+| `containers` | Total number of containers             |
+| `running`    | Number of running containers           |
+| `paused`     | Number of paused containers            |
+| `stopped`    | Number of stopped containers           |
+| `images`     | Number of images                       |
+
+Count keywords used in thresholds are emitted as performance data.

docs/samples/CheckDocker_check_docker_info_samples.md

@@ -0,0 +1,20 @@
+**Check that the docker daemon itself is up and responding:**
+
+```
+check_docker_info
+OK: docker 29.5.3 on docker-host: 16 running, 0 paused, 4 stopped containers, 231 images
+```
+
+**Alert when nothing is running (or too much is stopped), with perf data:**
+
+```
+check_docker_info "warning=running < 1" "critical=stopped > 100"
+OK: docker 29.5.3 on docker-host: 16 running, 0 paused, 4 stopped containers, 231 images|'docker-host running'=16;1;0 'docker-host stopped'=4;0;100
+```
+
+**A daemon that is down is clearly reported (UNKNOWN):**
+
+```
+check_docker_info host=/var/run/missing.sock
+Failed to connect to docker daemon at '/var/run/missing.sock': Failed to connect to /var/run/missing.sock: No such file or directory
+```

docs/samples/CheckDocker_check_docker_restarts_desc.md

@@ -0,0 +1,29 @@
+#### About `check_docker_restarts`
+
+`check_docker_restarts` detects crash-looping and OOM-killed containers via
+the inspect endpoint (`RestartCount`, `State.OOMKilled`, `State.StartedAt`,
+`State.ExitCode`). It always includes stopped containers: a crash-looping
+container spends most of its time not-running, and a final crash leaves it
+exited — both must stay visible.
+
+Default thresholds encode the crash-loop signature: **warning** when
+`restart_count > 3 and started < 15m and started >= 0` (many restarts *and* a
+recent start — a container that has been up for a month is fine no matter how
+bumpy its past), **critical** when `oom_killed = 1` (the last exit was an
+out-of-memory kill).
+
+Available keywords (for `filter=` / `warning=` / `critical=` / syntax):
+
+| Keyword           | Description                                                           |
+|-------------------|-----------------------------------------------------------------------|
+| `names`           | Container name(s), comma separated                                    |
+| `image`           | Image the container was created from                                  |
+| `container_state` | `created`, `restarting`, `running`, `removing`, `paused`, `exited`, `dead` |
+| `restart_count`   | Restarts since the container was created                              |
+| `started`         | Seconds since the last start, `-1` if it never started; supports units (`started < 10m`) |
+| `exit_code`       | Exit code of the last exit (0 while running fine)                     |
+| `oom_killed`      | `1` when the last exit was an out-of-memory kill, else `0`            |
+
+`container=<name>` (repeatable) scopes the check to specific containers.
+Note that `restart_count` is cumulative for the container's lifetime; that is
+why the default warning also requires a recent start before it fires.

docs/samples/CheckDocker_check_docker_restarts_samples.md

@@ -0,0 +1,27 @@
+**Detect restart loops (default: WARNING when a container restarted more than 3 times and last started within 15 minutes; OOM kills are CRITICAL):**
+
+```
+check_docker_restarts
+WARNING: crashy-app: 8 restarts, restarting
+```
+
+A stable container is OK no matter how bumpy its distant past:
+
+```
+check_docker_restarts container=app-backend
+OK: app-backend: 0 restarts, running
+```
+
+**An out-of-memory kill is CRITICAL by default:**
+
+```
+check_docker_restarts container=greedy-app
+CRITICAL: greedy-app: 2 restarts, exited
+```
+
+**Custom rules using the keywords (e.g. any restart of a specific container within the last hour):**
+
+```
+check_docker_restarts container=app-backend "warning=restart_count > 0 and started < 1h" "detail-syntax=%(names): %(restart_count) restarts, up %(started)s, exit=%(exit_code) oom=%(oom_killed)"
+OK: app-backend: 0 restarts, up 236s, exit=0 oom=0|'app-backend restarts'=0;0;0
+```

docs/samples/CheckDocker_check_docker_samples.md

@@ -0,0 +1,46 @@
+**Require that specific containers are running (a missing or stopped container is CRITICAL):**
+
+```
+check_docker container=web-frontend
+OK: web-frontend=running
+```
+
+```
+check_docker container=web-frontend container=backup-agent
+CRITICAL: backup-agent=missing
+```
+
+**List all running containers (inventory-style):**
+
+```
+check_docker
+OK: web-frontend=running, database=running
+```
+
+**Include stopped containers (`docker ps -a`) — any non-running container trips the default critical:**
+
+```
+check_docker all=true
+CRITICAL: old-job=exited
+```
+
+**Use the container keywords in the output:**
+
+```
+check_docker container=web-frontend "detail-syntax=%(names): %(image) %(status) ports=%(ports)" "top-syntax=${status}: ${list}"
+OK: web-frontend: nginx:alpine Up 2 hours ports=0.0.0.0:18080->80/tcp,:::18080->80/tcp
+```
+
+**Alert on failing container health checks instead of state:**
+
+```
+check_docker "filter=has_health_check = 1" "warning=health = 'starting'" "critical=health = 'unhealthy'" "detail-syntax=%(names)=%(health)"
+OK: web-frontend=healthy
+```
+
+**A daemon that is down is clearly reported (UNKNOWN):**
+
+```
+check_docker host=/var/run/missing.sock
+Failed to connect to docker daemon at '/var/run/missing.sock': Failed to connect to /var/run/missing.sock: No such file or directory
+```

docs/samples/CheckDocker_check_docker_stats_desc.md

@@ -0,0 +1,27 @@
+#### About `check_docker_stats`
+
+`check_docker_stats` samples per-container resource usage the way
+`docker stats` does: CPU as a percentage of the host (delta of the container's
+CPU time over the host's, scaled by online CPUs) and memory usage with the
+page cache excluded, against the container's memory limit.
+
+Sampling uses the daemon's `stream=false` stats mode, which takes two readings
+about a second apart so the CPU delta is meaningful — expect roughly **one
+second per sampled container**. On hosts with many containers, scope the check
+with `container=<name>` (repeatable) or accept the added latency.
+
+Available keywords (for `filter=` / `warning=` / `critical=` / syntax):
+
+| Keyword        | Description                                                        |
+|----------------|--------------------------------------------------------------------|
+| `names`        | Container name(s), comma separated                                 |
+| `image`        | Image the container was created from                               |
+| `cpu_pct`      | CPU usage in percent of the host (like `docker stats`)             |
+| `memory_used`  | Memory used in bytes, page cache excluded; size units in thresholds (`memory_used > 200M`) |
+| `memory_limit` | Memory limit in bytes (the host's total memory when unlimited)     |
+| `memory_pct`   | Memory usage in percent of the limit                               |
+| `memory`       | Human readable usage, e.g. `45.2MB of 256MB` (display only)        |
+
+Size-typed thresholds need a unit (`1b`, `64k`, `200M`, `1G`); a bare number
+is rejected. No default thresholds: an idle container is OK until you say
+otherwise.

docs/samples/CheckDocker_check_docker_stats_samples.md

@@ -0,0 +1,27 @@
+**Check resource usage of a specific container:**
+
+```
+check_docker_stats container=app-backend
+OK: app-backend: cpu 2%, memory 45.2MB of 256MB (17%)
+```
+
+**Alert on memory pressure or CPU saturation (thresholded keywords become perf data):**
+
+```
+check_docker_stats container=app-backend "warning=memory_pct > 80" "critical=memory_pct > 95"
+OK: app-backend: cpu 2%, memory 45.2MB of 256MB (17%)|'app-backend memory %'=17%;80;95
+```
+
+**Absolute memory thresholds take byte units (`1b`, `64k`, `200M`, `1G`, ...):**
+
+```
+check_docker_stats container=app-backend "critical=memory_used > 200M"
+OK: app-backend: cpu 2%, memory 45.2MB of 256MB (17%)|'app-backend memory'=47401984B;0;209715200
+```
+
+**Sample every running container (about a second per container, so scope on busy hosts):**
+
+```
+check_docker_stats "warning=cpu_pct > 80"
+OK: web-frontend: cpu 1%, memory 12.4MB of 15.35GB (0%), app-backend: cpu 2%, memory 45.2MB of 256MB (17%)
+```