Checkers

This Django app provides a small health-check framework and two management commands you can run locally or in automation.

See Architecture for how this app fits in the pipeline (CHECK stage).

What's included

Models

CheckRun — Log of standalone health check executions
- Status, message, and metrics from the check
- Link to created Alert (if check found an issue)
- Execution timing and correlation ID for tracing

Note: When checks run as part of the pipeline, tracking is handled by apps.orchestration via StageExecution with stage="check".

Available checkers

These checkers are registered in apps/checkers/checkers/__init__.py (CHECKER_REGISTRY):

cpu — CPU usage % (warn ≥ 70, critical ≥ 90)
memory — RAM usage % (warn ≥ 70, critical ≥ 90)
disk — disk usage % by path (warn ≥ 80, critical ≥ 95)
disk_macos — macOS disk analysis: space hogs, old files, cleanup recommendations (warn ≥ 5 GB, critical ≥ 20 GB recoverable). Skips on non-darwin.
disk_linux — Linux disk analysis: apt cache, journal logs, Docker/Snap data, old temp files (warn ≥ 5 GB, critical ≥ 20 GB recoverable). Skips on non-linux.
disk_common — Cross-platform disk analysis: system logs, user caches, temp files, large files in home (warn ≥ 5 GB, critical ≥ 20 GB recoverable).
disk_inodes — Inode usage per filesystem via os.statvfs (cross-platform Unix — Linux + macOS; no sudo). Catches inode exhaustion (many tiny files) that free-space checks miss. Worst path drives status: warn ≥ 80 %, critical ≥ 95 %. Filesystems that don't track inodes (f_files == 0) are skipped; skips as OK where statvfs is unavailable (e.g. Windows).
network — % of hosts reachable via ping (OK ≥ 70%, warning ≥ 50%, else critical)
process — % of named processes running (OK = 100%, warning ≥ 50%, else critical)
raid — Linux software-RAID (mdadm) health from /proc/mdstat (no sudo, no subprocess). Degraded-and-not-rebuilding or inactive array → CRITICAL; actively rebuilding (recovery/resync/reshape) → WARNING; routine check/repair scrubs and healthy arrays → OK. Skips as OK on non-linux or when /proc/mdstat is absent.
disk_temp — Hottest disk temperature via Linux hwmon (psutil.sensors_temperatures(), chips drivetemp/nvme; no sudo). Worst disk drives status: warn ≥ 55 °C, critical ≥ 60 °C. Skips as OK on non-linux or when no disk sensors are present. Requires the drivetemp kernel module (Linux 5.6+) loaded (modprobe drivetemp) for SATA/NVMe temps without root; SMART/smartctl is out of scope.
cpu_temp — Hottest CPU package/core temperature via Linux hwmon (psutil.sensors_temperatures(), chips coretemp/k10temp/zenpower/cpu_thermal; no sudo). Warm sensor drives status: warn ≥ 80 °C, critical ≥ 90 °C. Skips as OK on non-linux or when no CPU sensors are present.
io_strain — Busiest disk IO utilization (% busy time) via psutil.disk_io_counters(perdisk=True), sampling busy_time over an interval (default 1 s; no sudo). Busiest disk drives status: warn ≥ 80 %, critical ≥ 95 %. Metrics also include per-disk throughput (MB/s) and IOPS. Skips as OK on non-linux or when no IO counters / busy_time are available.
listening_ports — Audits the host's LISTENing ports via psutil.net_connections (reads /proc/net; no sudo on Linux). Ports outside settings.LISTENING_PORTS_ALLOWLIST are flagged WARNING; with no allowlist set, only externally-exposed (non-loopback) ports are flagged. Configure via the LISTENING_PORTS_ALLOWLIST env var (comma-separated ports). Linux-gated — skips as OK on non-linux (where net_connections needs root) or if the sockets can't be read.

List them via:

uv run python manage.py check_health --list

Selecting Checkers

Which checkers run is controlled by pipeline definitions. The context node's checker_names config specifies which checkers to include:

{"id": "check_health", "type": "context", "config": {"checker_names": ["cpu", "memory", "disk"]}}

If checker_names is omitted, all registered checkers run.

You can also specify checkers explicitly via the CLI:

uv run python manage.py run_pipeline --checks-only --checkers cpu memory disk

Django Admin

Access the admin interface at /admin/checkers/ to view check run history:

CheckRun — View all check executions (read-only)
- Colored status badges (ok/warning/critical/unknown)
- Duration display
- Link to created alert (if any)
- Filter by status, checker name, hostname
- Search by checker name, hostname, message, trace ID
- Date hierarchy by execution time

Note: Check runs are audit records and cannot be added/edited manually. They are created automatically when checks are run via management commands or the orchestration pipeline.

Django System Checks

In addition to runtime health checkers, this app registers Django system checks that run with python manage.py check. These verify project configuration:

Tag	Check ID	Description
`database`	`checkers.E001`	Database connection error
`migrations`	`checkers.W001`	Pending migrations warning
`security`	`checkers.W010`	DEBUG mode enabled
`security`	`checkers.W011`	Weak SECRET_KEY
`environment`	`checkers.W012`	`.env` file missing
`environment`	`checkers.I003`	Required env var not set (from `.env.sample`, info)
`environment`	`checkers.W017`	Project directory not writable
`pipeline`	`checkers.I001`	Pipeline definition counts (info)
`pipeline`	`checkers.W014`	No active notification channels or empty config
`crontab`	`checkers.W002`	No crontab configured
`crontab`	`checkers.W004`	Health check cron job not found
`crontab`	`checkers.W015`	`cron.log` stale (> 1 hour)
`crontab`	`checkers.W016`	`cron.log` too large (> 50MB)
`database`	`checkers.E003`	Missing django_migrations table (deploy)

Run all system checks:

uv run python manage.py check

Run specific check tags:

uv run python manage.py check --tag database
uv run python manage.py check --tag migrations
uv run python manage.py check --tag security
uv run python manage.py check --tag environment
uv run python manage.py check --tag pipeline
uv run python manage.py check --tag crontab

Run deployment checks (includes additional security/config checks):

uv run python manage.py check --deploy

Preflight Command

The preflight command runs all Django system checks grouped by tag with formatted output:

# Run all checks
uv run python manage.py preflight

# Filter by tag(s)
uv run python manage.py preflight --only security
uv run python manage.py preflight --only security,environment

# JSON output (for CI)
uv run python manage.py preflight --json

System Check Script

For a comprehensive check that includes shell-level pre-Django checks (Python version, uv installed, disk space) plus all Django preflight checks:

bin/check_system.sh                # Full check (shell + Django)
bin/check_system.sh --shell-only   # Only shell-level checks
bin/check_system.sh --django-only  # Only Django preflight

CLI Reference

There are two management commands for running checks. All flags can be passed after aliases too (e.g., sm-check-health --json).

`check_health`

Run all checkers (or a selection) and show a summary.

# Run ALL registered checkers
uv run python manage.py check_health

# Run specific checkers only
uv run python manage.py check_health cpu memory
uv run python manage.py check_health cpu memory disk network process

# List available checkers and exit
uv run python manage.py check_health --list

JSON output

# JSON output (for scripts, cron, piping to jq)
uv run python manage.py check_health --json

# Specific checkers + JSON
uv run python manage.py check_health cpu disk --json

Exit codes for CI/automation

By default: exit 2 if any CRITICAL, 1 if any UNKNOWN, 0 otherwise.

# Exit 1 if ANY check is WARNING or CRITICAL (strictest)
uv run python manage.py check_health --fail-on-warning

# Exit 1 only if ANY check is CRITICAL
uv run python manage.py check_health --fail-on-critical

# CI pipeline example: fail build on critical
uv run python manage.py check_health --fail-on-critical --json

Threshold overrides

Override default warning/critical thresholds for all checkers in this run:

# Lower thresholds (more sensitive)
uv run python manage.py check_health --warning-threshold 60 --critical-threshold 80

# Higher thresholds (less sensitive)
uv run python manage.py check_health --warning-threshold 85 --critical-threshold 98

# Override thresholds for specific checkers only
uv run python manage.py check_health cpu memory --warning-threshold 75 --critical-threshold 95

Checker-specific options

These flags are passed to the relevant checker when it runs:

# Disk: check specific mount points
uv run python manage.py check_health disk --disk-paths / /var /tmp /home

# Network: ping specific hosts
uv run python manage.py check_health network --ping-hosts 8.8.8.8 1.1.1.1 github.com

# Process: verify specific processes are running
uv run python manage.py check_health process --processes nginx gunicorn postgres

Combined examples

# Full CI check: all checkers, JSON, fail on warning
uv run python manage.py check_health --json --fail-on-warning

# Disk + network with custom targets + thresholds
uv run python manage.py check_health disk network \
  --disk-paths / /var/log \
  --ping-hosts 8.8.8.8 google.com \
  --warning-threshold 75 --critical-threshold 90

# Cron job: all checks, JSON, append to log
uv run python manage.py check_health --json >> /var/log/health-checks.log 2>&1

# Quick smoke test: CPU + memory, fail on critical
uv run python manage.py check_health cpu memory --fail-on-critical

Flag reference

Flag	Type	Default	Description
`checkers` (positional)	str...	all	Specific checkers to run (space-separated)
`--list`	flag	—	List available checkers and exit
`--json`	flag	—	Output results as JSON
`--fail-on-warning`	flag	—	Exit 1 if any WARNING or CRITICAL
`--fail-on-critical`	flag	—	Exit 1 only if any CRITICAL
`--warning-threshold`	float	per-checker	Override warning threshold for all checks
`--critical-threshold`	float	per-checker	Override critical threshold for all checks
`--disk-paths`	str...	`/`	Paths to check (disk checker)
`--ping-hosts`	str...	`8.8.8.8 1.1.1.1`	Hosts to ping (network checker)
`--processes`	str...	—	Process names to check (process checker)

`run_check`

Run a single checker with checker-specific options.

# Basic usage
uv run python manage.py run_check cpu
uv run python manage.py run_check memory
uv run python manage.py run_check disk
uv run python manage.py run_check network
uv run python manage.py run_check process

JSON output

uv run python manage.py run_check cpu --json
uv run python manage.py run_check disk --json

Threshold overrides

# Override thresholds for this single check
uv run python manage.py run_check cpu --warning-threshold 80 --critical-threshold 95
uv run python manage.py run_check memory --warning-threshold 75 --critical-threshold 90
uv run python manage.py run_check disk --warning-threshold 85 --critical-threshold 98

CPU checker options

# Default: 5 samples, 1 second apart
uv run python manage.py run_check cpu

# More samples for better accuracy
uv run python manage.py run_check cpu --samples 10

# Faster sampling (0.5s intervals)
uv run python manage.py run_check cpu --sample-interval 0.5

# Quick snapshot (1 sample, no wait)
uv run python manage.py run_check cpu --samples 1 --sample-interval 0

# Per-CPU mode (reports busiest core)
uv run python manage.py run_check cpu --per-cpu

# All CPU options combined
uv run python manage.py run_check cpu --samples 10 --sample-interval 0.5 --per-cpu

# CPU with threshold override + JSON
uv run python manage.py run_check cpu --samples 10 --per-cpu --warning-threshold 80 --critical-threshold 95 --json

Memory checker options

# Default: RAM only
uv run python manage.py run_check memory

# Include swap memory in the check
uv run python manage.py run_check memory --include-swap

# Memory with custom thresholds
uv run python manage.py run_check memory --include-swap --warning-threshold 75 --critical-threshold 90 --json

Disk checker options

# Default: check /
uv run python manage.py run_check disk

# Check specific paths
uv run python manage.py run_check disk --paths /
uv run python manage.py run_check disk --paths / /var /tmp /home
uv run python manage.py run_check disk --paths /var/log /var/lib

# Disk with thresholds + JSON
uv run python manage.py run_check disk --paths / /var/log --warning-threshold 80 --critical-threshold 95 --json

Network checker options

# Default hosts: 8.8.8.8, 1.1.1.1
uv run python manage.py run_check network

# Custom hosts
uv run python manage.py run_check network --hosts 8.8.8.8 1.1.1.1 github.com
uv run python manage.py run_check network --hosts google.com cloudflare.com aws.amazon.com

# Network with JSON
uv run python manage.py run_check network --hosts 8.8.8.8 google.com --json

Process checker options

# Check specific processes
uv run python manage.py run_check process --names nginx
uv run python manage.py run_check process --names nginx postgres gunicorn
uv run python manage.py run_check process --names nginx gunicorn postgres

# Process with JSON
uv run python manage.py run_check process --names nginx postgres --json

Flag reference

Flag	Type	Default	Description
`checker` (positional)	str	required	Checker name (cpu, memory, disk, network, process, etc.)
`--json`	flag	—	Output as JSON
`--warning-threshold`	float	per-checker	Override warning threshold
`--critical-threshold`	float	per-checker	Override critical threshold
`--samples`	int	5	Number of CPU samples (cpu only)
`--sample-interval`	float	1.0	Seconds between CPU samples (cpu only)
`--per-cpu`	flag	—	Per-CPU mode, reports busiest core (cpu only)
`--include-swap`	flag	—	Include swap memory (memory only)
`--paths`	str...	`/`	Disk paths to check (disk only)
`--hosts`	str...	`8.8.8.8 1.1.1.1`	Hosts to ping (network only)
`--names`	str...	—	Process names to check (process only)

Extending

Checkers live in:

apps/checkers/checkers/

To add a new checker:

Inherit from BaseChecker
Return a CheckResult
Register it in CHECKER_REGISTRY

Troubleshooting

`ping` permissions / failures

The network checker uses the system ping binary. In some environments (containers, locked-down CI), ping may be blocked.

Workarounds:

run without the network checker, or
adjust platform/network permissions

Disk path not found

If a disk path doesn’t exist, the disk checker returns UNKNOWN for that path.

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Checkers

What's included

Models

Available checkers

Selecting Checkers

Django Admin

Django System Checks

Preflight Command

System Check Script

CLI Reference

check_health

JSON output

Exit codes for CI/automation

Threshold overrides

Checker-specific options

Combined examples

Flag reference

run_check

JSON output

Threshold overrides

CPU checker options

Memory checker options

Disk checker options

Network checker options

Process checker options

Flag reference

Extending

Troubleshooting

ping permissions / failures

Disk path not found

`check_health`

`run_check`

`ping` permissions / failures