Prometheus + Grafana + n8n Telegram Alerting

Containerized monitoring stack for two Docker hosts using Prometheus, Grafana, node_exporter, cAdvisor, and n8n-driven Telegram alerts.

This repository is sanitized for public use. Replace the placeholders in .env.example and the n8n workflow before deployment. Do not commit .env or exported workflow files containing real bot tokens, chat IDs, hostnames, or private IPs.

Architecture

• Host A runs the core monitoring stack:
  • Prometheus with 90-day TSDB retention
  • Grafana OSS
  • host-level node_exporter
  • cAdvisor for Docker/container metrics
• Host B runs a lightweight monitoring-agent stack:
  • host-level node_exporter
  • cAdvisor
• Prometheus scrapes both hosts every 15 seconds.
• An example application scrape target is included for Qdrant-style /metrics endpoints. Replace it with your own app target or remove it.
• Grafana imports community dashboards for host and container visibility.
• n8n polls Prometheus /api/v1/query every 5 minutes, deduplicates alerts for 60 minutes using workflow static data, and routes messages to Telegram.

Design decision: n8n instead of Alertmanager/Blackbox

This stack intentionally keeps the alert path simple:

• No Alertmanager.
• No Blackbox exporter.
• n8n owns the polling, dedupe, formatting, and Telegram routing.

That trades some native Prometheus alerting features for a workflow users can inspect and modify in n8n. It also makes it easy to reuse an existing Telegram bot/chat without creating another bot.

Alert rules

The included n8n workflow implements five Prometheus queries:

1. Host down
   • Query: up == 0
   • Requires two consecutive 5-minute polls before alerting.
2. Disk available below threshold
   • Default threshold: < 15% available.
   • Excludes ephemeral/container filesystems such as tmpfs, overlay, squashfs, ramfs, devtmpfs, nsfs, and fuse mounts.
3. Memory usage above 90%
   • Uses node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes.
4. Sustained CPU above 90% over 10 minutes
   • Uses idle CPU rate over a 10-minute window.
5. Container restarted within 15 minutes
   • Uses changes(container_start_time_seconds{name!=""}[15m]) > 0 from cAdvisor.

Alert message format:

🚨 Prometheus Alert
Host: <host>
Rule: <rule>
Value: <value>
Time: <ISO timestamp>

Grafana dashboards

Import these dashboard IDs and bind them to the Prometheus datasource:

• 1860 — Node Exporter Full
• 14282 — cAdvisor exporter

Repository layout

compose/host-a/docker-compose.yml        # Prometheus + Grafana + exporters
compose/host-b/docker-compose.yml        # exporter-only agent stack
prometheus/prometheus.yml                # sanitized scrape config
n8n/prometheus-alerts.workflow.json      # sanitized n8n workflow export
.env.example                             # required variables and placeholders
.gitignore                               # keeps secrets out from commit one

Setup

1. Clone the repo on Host A and Host B.

2. Copy .env.example to .env and replace placeholders.

3. On Host A, edit or render prometheus/prometheus.yml so target placeholders resolve to your hosts.

   • Prometheus does not automatically expand shell environment variables inside YAML in all configurations.
   • A common pattern is to keep prometheus.yml as a template and render it with envsubst before starting Prometheus.

4. Start Host A:

   cd compose/host-a
   docker compose up -d

5. Start Host B:

   cd compose/host-b
   docker compose up -d

6. Verify Prometheus targets:

   curl http://localhost:${PROMETHEUS_PORT:-9090}/api/v1/targets

7. Configure Grafana:

   • URL: http://<host-a>:${GRAFANA_PORT:-3000}
   • Add Prometheus datasource: http://prometheus:9090 if Grafana and Prometheus share the same Compose network.
   • Import dashboards 1860 and 14282.

8. Import the n8n workflow:

   n8n import:workflow --input=n8n/prometheus-alerts.workflow.json --activeState=false
   n8n update:workflow --id=<imported-id> --active=true

   Depending on your n8n version/deployment mode, you may need to restart n8n so the scheduler picks up the newly activated workflow.

n8n configuration notes

The workflow is sanitized. Replace these placeholders before use:

• {{TELEGRAM_BOT_TOKEN}}
• {{TELEGRAM_CHAT_ID}}
• PROMETHEUS_URL / $env.PROMETHEUS_URL

If your n8n container is not on the same Docker network as Prometheus, do not use the Compose service name prometheus. Use a URL reachable from the n8n container.

Lessons learned

• Bind-mounted config permissions matter. Prometheus runs as a non-root user; parent directories and prometheus.yml must be traversable/readable by that user. A readable file inside an unreadable directory is still unreadable. Linux, refreshingly, does not care about intent.
• Host port collisions happen. cAdvisor defaults often collide with existing services. Pick the nearest free port and document the deviation.
• Grafana first-run state matters. If the Grafana data volume already exists, changing GF_SECURITY_ADMIN_PASSWORD in Compose may not reset the admin password. Use the Grafana CLI to reset it deliberately.
• n8n scheduler pickup may require restart. CLI activation can mark a workflow active in the database while the running process has not loaded the new schedule yet.
• n8n CLI execution is not the same as scheduler execution. A Schedule Trigger workflow may not run through n8n execute unless it also has an Execute Workflow Trigger. Test scheduler behavior with scheduler-driven runs.
• Public repos need a first-push audit. Sanitize before commit one. Git history is permanent enough to punish optimism.

Security

Before publishing changes, run:

gitleaks detect --no-git
# plus explicit private-IP and chat-ID greps for your environment

Do not push real IPs, bot tokens, chat IDs, passwords, API keys, or exported workflow instance IDs.