A self-hosted, agentless hardware monitoring dashboard and REST API for small GPU server clusters. Built for NVIDIA DGX Spark (GB10 Grace Blackwell) nodes running vLLM inference workloads, but works with any Linux host reachable over SSH.
Metrics are collected over SSH — nothing needs to be installed or run on the monitored nodes beyond Python 3 with psutil (and nvidia-smi for GPU stats). The server polls every node concurrently, stores 48 hours of history in SQLite, and serves a live dark-themed web dashboard.
- Agentless collection — CPU, RAM, disk, GPU (
nvidia-smi) and power-rail (lm-sensors) metrics gathered via SSH with public-key auth; offline nodes degrade gracefully instead of failing the request - Live dashboard — per-node metric cards, 9 time-series charts (Chart.js), disk usage bars, cluster summary strip, auto-refresh (2–30 s)
- 48-hour history — SQLite (WAL mode) time-series storage with automatic retention pruning and a zoomable chart range (1 minute → 48 hours)
- In-app configuration — add/remove nodes, toggle cards and charts, set poll interval and log level from the settings modal; changes persist to
.envand hot-apply without a restart - Bilingual UI — Turkish / English toggle
- REST API — typed JSON (Pydantic v2) endpoints for live metrics, history and config; optional
X-API-Keyauthentication - Accessible & responsive — keyboard-navigable, SVG icon set,
prefers-reduced-motionsupport, mobile-friendly layout
┌─ monitor host ───────────────────────────────────────────────┐
│ │
│ ┌────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │ asyncssh │────▶│ FastAPI │────▶│ SQLite │ │
│ │ collectors │ │ + Uvicorn │ │ metrics.db │ │
│ └─────┬──────┘ │ │ │ (48h, WAL) │ │
│ │ │ /api/v1/* │ └───────────────┘ │
│ │ │ / (dashboard)│ ▲ │
│ │ └──────────────┘ persist loop (5 s) │
│ [SSH] │
└────────┼─────────────────────────────────────────────────────┘
├──── node 1 (python3 + psutil, nvidia-smi)
└──── node 2 (python3 + psutil, nvidia-smi)
git clone <repo-url> && cd system_check
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtRequires Python 3.10+.
The server connects to nodes with an Ed25519 key (no passwords, no agent forwarding):
# Create a key if you don't have one
ssh-keygen -t ed25519 -C "castle-node-monitor" -f ~/.ssh/id_ed25519
# Copy the public key to every node
ssh-copy-id -i ~/.ssh/id_ed25519.pub <user>@<node-ip>Important: connect to each node manually once (
ssh <user>@<node-ip>) so its fingerprint is added to~/.ssh/known_hosts. Unknown host keys are rejected, and the node will show as offline until this is done.
cp .env.example .envThen either edit .env directly or start the server and add your nodes from the Settings modal in the dashboard — the device list is persisted back to .env automatically.
python main.pyOpen http://<host>:8765/ for the dashboard, or http://<host>:8765/docs for the interactive API docs (Swagger UI).
All settings live in .env (see .env.example):
| Variable | Default | Description |
|---|---|---|
CNM_NODES |
[] |
JSON array of node definitions — managed by the settings UI |
CNM_UI |
{} |
JSON object of dashboard preferences — managed by the settings UI |
SSH_KEY_PATH |
~/.ssh/id_ed25519 |
Private key used for all node connections |
API_PORT |
8765 |
Server port |
API_BIND_HOST |
0.0.0.0 |
Bind address |
SSH_CONNECT_TIMEOUT |
5.0 |
SSH connection timeout (seconds) |
SSH_COMMAND_TIMEOUT |
10.0 |
Per-command timeout (seconds) |
POLL_INTERVAL_SECONDS |
5.0 |
Background snapshot interval |
METRICS_DB_PATH |
XDG default | SQLite path (~/.local/share/castle_node_monitor/metrics.db) |
DASHBOARD_ENABLED |
true |
Set false for headless API-only mode |
API_SECRET_KEY |
(empty) | If set, all endpoints require the X-API-Key header |
A node entry in CNM_NODES looks like:
[{"name": "node1", "label": "Node 1", "host": "10.0.0.10", "username": "ubuntu", "port": 22, "color": "#3b9eff"}]| Endpoint | Description |
|---|---|
GET /api/v1/metrics |
Live metrics for all nodes (also persists a snapshot) |
GET /api/v1/metrics/{node} |
Live metrics for a single node |
GET /api/v1/history?hours=48&node=all |
Time-series history (max 48 h, sub-hour windows supported) |
GET /api/v1/config |
Current node list and UI preferences |
PUT /api/v1/config |
Save configuration (hot-applies, persists to .env) |
GET /api/v1/health |
Liveness probe |
curl http://localhost:8765/api/v1/metrics
curl "http://localhost:8765/api/v1/history?hours=24&node=node1"
# With authentication enabled
curl -H "X-API-Key: <secret>" http://localhost:8765/api/v1/metricsOffline nodes are returned with status: "offline" and an error_message — the request itself never fails.
- SSH access with public-key auth
- Python 3 with
psutil(CPU, RAM, disk, temperatures) nvidia-smi— GPU utilization, VRAM, temperature, power, clocks (optional)lm-sensors— node-level power rails, e.g. DGX Spark SPBM chip (optional)
Anything missing degrades gracefully: the corresponding card simply shows no data.
system_check/
├── main.py # FastAPI app, lifespan, background persist loop
├── requirements.txt
├── .env.example # Configuration template
├── api/
│ ├── models.py # Pydantic response models
│ └── routes.py # REST endpoints
├── core/
│ ├── config.py # Settings (pydantic-settings, .env-backed)
│ ├── config_store.py # Persists settings-UI changes back to .env
│ ├── metrics_db.py # SQLite history storage + retention
│ ├── security.py # SSH key loading / connection hardening
│ └── logging_setup.py
├── monitors/
│ └── remote_monitor.py # Concurrent SSH metric collection + parsers
└── static/
└── index.html # Dashboard (vanilla JS + Chart.js)
Nodes show as offline
- Is the node's fingerprint in
~/.ssh/known_hosts? Connect manually once. - Does key auth work?
ssh -i ~/.ssh/id_ed25519 <user>@<node-ip> - Is
python3(withpsutil) available on the node?
Dashboard returns 404 — check DASHBOARD_ENABLED=true and that static/index.html exists.
403 responses — API_SECRET_KEY is set; send the matching X-API-Key header.
MIT