A workflow-driven self-healing and alerting layer for a home datacenter, built on n8n. Replaces ad-hoc cron jobs with observable, retriable workflows that detect failures, attempt remediation, and escalate to a human only when automation can't fix it.
Cron jobs fail silently. A restart-if-down.sh script that itself never runs is worse than no automation at all — you get the false confidence without the safety net. This project moved every recurring infra task (health checks, backups, config drift detection, service restarts) off parallel cron/ad-hoc scheduling and into n8n, where every run is logged, retriable, and alerts on its own failure — including failure to alert.
n8n (workflow engine)
|
--------------------------------
| | |
Self-heal Config drift Scheduled sync
workflows detection / backup jobs
(restart agent (e.g. 2 AM
on failure) GitHub sync)
| | |
--------------------------------
|
common.py routing script
(shared self-heal logic)
|
--------------------------------
| | |
Telegram Telegram Obsidian log
Hub 1 Hub 2 archivist
(urgent) (advisory) workflow
Design principle: two-tier alerting, not one noisy firehose. Anything that needs eyes-on-now (a failed restart, a push failure) goes to the urgent hub. Anything informational — a daily summary, a drift report — goes to the advisory hub as a digest. This keeps the urgent channel meaningful instead of trained into "just another notification."
Hub 1 — Urgent: real-time, needs-action alerts. Triggers: self-heal restart attempt, GitHub sync push failure.
Hub 2 — Advisory: non-urgent digests. Triggers: daily infra summary, drift reports.
Splitting these was deliberate: a single alert bot that fires on both a failed backup and a routine daily digest trains you to ignore it. Separating urgency tiers keeps Hub 1 actionable.
Shared Python script that workflows call into for self-heal logic — the actual "detect it's down, attempt a fix, report the outcome" behavior lives here rather than being duplicated per-workflow. Routes outcomes through Hub 1 regardless of which workflow triggered it, so there's one consistent alert format for anything self-heal-related.
Workflow that checks running configuration against expected state and flags divergence — catches the "someone changed something by hand and forgot to commit it" class of problem before it becomes an incident.
Appends daily Hub 1/Hub 2 digest content to a running Markdown log (infra-changes-log.md). Turns ephemeral Telegram alerts into a searchable, permanent record instead of scrolling history in a chat app.
Includes a 2 AM repository sync workflow with built-in push-failure alerting — the current live implementation is a WikiDocs sync workflow that escalates to Hub 1 if the push path fails, rather than failing silently overnight.
This replaced a prior architecture of parallel cron jobs and ad-hoc internal scheduling (including Wazuh, CrowdSec, and QNAP-related scheduled tasks). crontab-ui was fully decommissioned once the equivalent workflows were live in n8n. The motivation: cron gives you a scheduled execution with no retry logic, no structured alerting, and no visibility into whether it actually ran — n8n gives you all three for free.
This first commit seeds the public repo with README.md at the root. The next export pass is intended to add a workflows/ directory for sanitized n8n workflow JSON, a scripts/ directory containing a sanitized common.py, and a docs/ directory containing the alert-architecture write-up. Sanitized for public release — bot tokens, chat IDs, workflow IDs, and internal hostnames are placeholders or removed. Workflow logic will be preserved; secrets and internal identifiers will not.
n8n, Python, Telegram Bot API, Obsidian, GitHub Actions, cron (retired)