Skip to content

Commit 8db9e07

Browse files
AndresQuijanoAndresQuijano
committed
feat: add hot wallet low liquidity alerts
1 parent 2f209d3 commit 8db9e07

File tree

5 files changed

+382
-45
lines changed

5 files changed

+382
-45
lines changed

‎docker-compose/monitoring/scripts/README.md‎

Lines changed: 61 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,23 @@ This directory contains the alert rules for the Liquidity Provider Server (LPS)
99
- `alerts/pegin-out-of-liquidity.json` - Alerts when PegIn operations run out of Bitcoin liquidity
1010
- `alerts/pegout-out-of-liquidity.json` - Alerts when PegOut operations run out of BTC liquidity
1111
- `alerts/lps-penalization.json` - Alerts when the LPS has been penalized
12+
- `alerts/hot-wallet-low-liquidity-warning.json` - Alerts when hot wallet liquidity drops below warning threshold
13+
- `alerts/hot-wallet-low-liquidity-critical.json` - Alerts when hot wallet liquidity is critically low
14+
15+
### Custom Contact Points (in contact-points/ subdirectory)
16+
- `contact-points/low-liquidity.json` - Custom email format for low liquidity alerts (used by both warning and critical alerts)
1217

1318
### Import Script
14-
- `import-alerts.sh` - Script to import alert rules into any Grafana instance
19+
- `import-alerts.sh` - Script to import alert rules, contact points, and notification policies into any Grafana instance
1520

1621
## Usage
1722

23+
### Prerequisites
24+
- `ALERT_RECIPIENT_EMAIL` environment variable must be set (e.g., via `source .env.regtest`)
25+
- Grafana with unified alerting enabled
26+
- Loki datasource configured (default UID: `loki-uid`, customizable via script parameter)
27+
- `curl` and `python3` available
28+
1829
### Import Alerts to Grafana
1930

2031
```bash
@@ -42,18 +53,24 @@ cd docker-compose/monitoring/scripts
4253
4. `folder_uid` - Folder UID for alerts (default: LPS)
4354
5. `datasource_uid` - Loki datasource UID (default: loki-uid)
4455

56+
### Environment Variables
57+
- `ALERT_RECIPIENT_EMAIL` (**required**) - Recipient email for alert notifications. Read from the environment (set in `.env` files).
58+
4559
## Directory Structure
4660

4761
```
48-
docker-compose/monitoring/
49-
├── scripts/
50-
│ ├── import-alerts.sh # Main import script
51-
│ └── README.md # This documentation
52-
└── alerts/
53-
├── node-eclipse-detection.json # Eclipse attack alert
54-
├── pegin-out-of-liquidity.json # PegIn liquidity alert
55-
├── pegout-out-of-liquidity.json # PegOut liquidity alert
56-
└── lps-penalization.json # LPS penalization alert
62+
docker-compose/monitoring/scripts/
63+
├── import-alerts.sh # Main import script
64+
├── README.md # This documentation
65+
├── alerts/
66+
│ ├── node-eclipse-detection.json # Eclipse attack alert
67+
│ ├── pegin-out-of-liquidity.json # PegIn liquidity alert
68+
│ ├── pegout-out-of-liquidity.json # PegOut liquidity alert
69+
│ ├── lps-penalization.json # LPS penalization alert
70+
│ ├── hot-wallet-low-liquidity-warning.json # Hot wallet low liquidity warning (regex extraction)
71+
│ └── hot-wallet-low-liquidity-critical.json # Hot wallet critical low liquidity (regex extraction)
72+
└── contact-points/
73+
└── low-liquidity.json # Custom email format for low liquidity alerts
5774
```
5875

5976
## Alert Details
@@ -74,6 +91,18 @@ docker-compose/monitoring/
7491
- **Trigger**: When log contains "Alert! - Subject: LPS has been penalized"
7592
- **Purpose**: Alerts when the Liquidity Provider has been penalized for failing to fulfill quote commitments
7693

94+
### Hot Wallet Low Liquidity Warning Alert
95+
- **Trigger**: When log contains "Alert! - Subject: Hot wallet: Low liquidity, refill recommended"
96+
- **Purpose**: Alerts when the hot wallet liquidity is below the warning threshold
97+
- **Dynamic extraction**: Uses LogQL `regexp` to extract `network`, `current`, and `threshold` from the log body and includes them in the notification via `{{ $labels.xxx }}`
98+
- **Custom contact point**: Routed to `lps-email-low-liquidity` via `__contact_point__` for a tailored email format
99+
100+
### Hot Wallet Critical Low Liquidity Alert
101+
- **Trigger**: When log contains "Alert! - Subject: Hot wallet: Critical low liquidity, refill required"
102+
- **Purpose**: Alerts when the hot wallet liquidity is critically low and an immediate refill is required
103+
- **Dynamic extraction**: Same as the warning alert -- extracts `network`, `current`, and `threshold` via `regexp`
104+
- **Custom contact point**: Shares `lps-email-low-liquidity` with the warning alert via `__contact_point__`
105+
77106
## Configuration Details
78107

79108
### Alert Rule Settings
@@ -91,19 +120,33 @@ The alert JSON files use `"datasourceUid": "loki-uid"` by default.
91120

92121
The import script automatically replaces the datasource UID if you specify a different one via the `datasource_uid` parameter, making it portable across different Grafana instances.
93122

94-
## Requirements
123+
### Contact Points and Notification Policy
95124

96-
- Grafana with unified alerting enabled
97-
- Loki datasource configured (default UID: `loki-uid`, customizable via script parameter)
98-
- curl command available
99-
- jq for JSON processing (optional, for verification)
125+
In Grafana, a **contact point** bundles together the delivery channel (email, Slack, etc.), the recipient address, and the message template (subject, body format). In our setup the recipient is always the same (`ALERT_RECIPIENT_EMAIL`), so custom contact points are used solely to provide different **email formats** per alert type -- not different recipients.
126+
127+
The import script automatically:
128+
- Creates a default `lps-email` contact point for all alerts, using the `ALERT_RECIPIENT_EMAIL` environment variable as the recipient
129+
- Imports custom contact points from the `contact-points/` directory (e.g., `lps-email-low-liquidity` with a tailored subject/message format)
130+
- Builds child routes from alert rules that declare a `__contact_point__` field, routing them to the named contact point
131+
- Configures the notification policy with:
132+
- A root route that sends all alerts to `lps-email` (default Grafana email format)
133+
- Child routes that match specific alerts to their custom contact points (matched by `alertname`)
134+
- Sets notification timing: `group_wait: 10s`, `group_interval: 1m`, `repeat_interval: 5m`
135+
136+
### Routing Alerts to a Custom Contact Point
137+
138+
To give an alert a custom email format:
139+
1. Create a contact point JSON in `contact-points/` with `__ALERT_EMAIL__` as a placeholder for the recipient address (or reuse an existing one)
140+
2. In the alert rule JSON (in `alerts/`), add a `"__contact_point__": "<contact-point-name>"` field referencing the contact point's `name`
141+
3. The script will strip `__contact_point__` before sending to Grafana and automatically create a child route matching the alert to that contact point
142+
4. Multiple alerts can share the same contact point by referencing the same name
100143

101144
## Notes
102145

103146
- Rules are created in the specified folder (default: LPS)
104147
- Script automatically creates the folder if it doesn't exist
105-
- Duplicate rules are skipped (no error)
148+
- Duplicate rules and contact points are skipped (no error)
106149
- Uses Grafana Provisioning API for reliable imports
107-
- Script looks for JSON files in the `alerts/` subdirectory relative to its location
150+
- Script looks for JSON files in the `alerts/` and `contact-points/` subdirectories relative to its location
108151
- Rules created using this script cannot be edited in the Grafana UI
109-
- To avoid a second email being sent once the rule is resolved, the contact point must be changed to "Disable resolved message" in Grafana UI
152+
- Custom contact points have `disableResolveMessage` set to prevent a second email when the alert resolves

‎docker-compose/monitoring/scripts/alerts/hot-wallet-low-liquidity-critical.json‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
{
2+
"uid": "hotwalletlowliqcrit",
3+
"title": "Hot Wallet Critical Low Liquidity Alert",
4+
"condition": "C",
5+
"data": [
6+
{
7+
"refId": "A",
8+
"queryType": "instant",
9+
"relativeTimeRange": {
10+
"from": 120,
11+
"to": 0
12+
},
13+
"datasourceUid": "loki-uid",
14+
"model": {
15+
"datasource": {
16+
"type": "loki",
17+
"uid": "loki-uid"
18+
},
19+
"editorMode": "code",
20+
"expr": "sum by (network, current, threshold) (count_over_time({service=\"lps\"} |~ \"Alert! - Subject: Hot wallet: Critical low liquidity, refill required\" | regexp \"Network: (?P<network>\\\\w+) \\\\| Current: (?P<current>[\\\\d.]+) \\\\| Threshold: (?P<threshold>[\\\\d.]+)\" [1m]))",
21+
"instant": true,
22+
"intervalMs": 1000,
23+
"maxDataPoints": 43200,
24+
"queryType": "instant",
25+
"refId": "A"
26+
}
27+
},
28+
{
29+
"refId": "C",
30+
"datasourceUid": "__expr__",
31+
"model": {
32+
"conditions": [
33+
{
34+
"evaluator": {
35+
"params": [0],
36+
"type": "gt"
37+
},
38+
"operator": {
39+
"type": "and"
40+
},
41+
"query": {
42+
"params": ["C"]
43+
},
44+
"reducer": {
45+
"params": [],
46+
"type": "last"
47+
},
48+
"type": "query"
49+
}
50+
],
51+
"datasource": {
52+
"type": "__expr__",
53+
"uid": "__expr__"
54+
},
55+
"expression": "A",
56+
"intervalMs": 1000,
57+
"maxDataPoints": 43200,
58+
"refId": "C",
59+
"type": "threshold"
60+
}
61+
}
62+
],
63+
"noDataState": "OK",
64+
"execErrState": "OK",
65+
"for": "0s",
66+
"annotations": {
67+
"description": "Hot wallet liquidity is critically low and an immediate refill is required",
68+
"summary": "[CRITICAL] Hot Wallet Low Liquidity - Network: {{ $labels.network }} | Current: {{ $labels.current }} | Recommended: {{ $labels.threshold }}"
69+
},
70+
"labels": {},
71+
"folderUID": "LPS",
72+
"ruleGroup": "10s",
73+
"intervalSeconds": 10,
74+
"__contact_point__": "lps-email-low-liquidity"
75+
}

‎docker-compose/monitoring/scripts/alerts/hot-wallet-low-liquidity-warning.json‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
{
2+
"uid": "hotwalletlowliqwarn",
3+
"title": "Hot Wallet Low Liquidity Warning Alert",
4+
"condition": "C",
5+
"data": [
6+
{
7+
"refId": "A",
8+
"queryType": "instant",
9+
"relativeTimeRange": {
10+
"from": 120,
11+
"to": 0
12+
},
13+
"datasourceUid": "loki-uid",
14+
"model": {
15+
"datasource": {
16+
"type": "loki",
17+
"uid": "loki-uid"
18+
},
19+
"editorMode": "code",
20+
"expr": "sum by (network, current, threshold) (count_over_time({service=\"lps\"} |~ \"Alert! - Subject: Hot wallet: Low liquidity, refill recommended\" | regexp \"Network: (?P<network>\\\\w+) \\\\| Current: (?P<current>[\\\\d.]+) \\\\| Threshold: (?P<threshold>[\\\\d.]+)\" [1m]))",
21+
"instant": true,
22+
"intervalMs": 1000,
23+
"maxDataPoints": 43200,
24+
"queryType": "instant",
25+
"refId": "A"
26+
}
27+
},
28+
{
29+
"refId": "C",
30+
"datasourceUid": "__expr__",
31+
"model": {
32+
"conditions": [
33+
{
34+
"evaluator": {
35+
"params": [0],
36+
"type": "gt"
37+
},
38+
"operator": {
39+
"type": "and"
40+
},
41+
"query": {
42+
"params": ["C"]
43+
},
44+
"reducer": {
45+
"params": [],
46+
"type": "last"
47+
},
48+
"type": "query"
49+
}
50+
],
51+
"datasource": {
52+
"type": "__expr__",
53+
"uid": "__expr__"
54+
},
55+
"expression": "A",
56+
"intervalMs": 1000,
57+
"maxDataPoints": 43200,
58+
"refId": "C",
59+
"type": "threshold"
60+
}
61+
}
62+
],
63+
"noDataState": "OK",
64+
"execErrState": "OK",
65+
"for": "0s",
66+
"annotations": {
67+
"description": "Hot wallet liquidity is below the warning threshold and a refill is recommended",
68+
"summary": "[WARNING] Hot Wallet Low Liquidity - Network: {{ $labels.network }} | Current: {{ $labels.current }} | Recommended: {{ $labels.threshold }}"
69+
},
70+
"labels": {},
71+
"folderUID": "LPS",
72+
"ruleGroup": "10s",
73+
"intervalSeconds": 10,
74+
"__contact_point__": "lps-email-low-liquidity"
75+
}

‎docker-compose/monitoring/scripts/contact-points/low-liquidity.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"name": "lps-email-low-liquidity",
3+
"type": "email",
4+
"disableResolveMessage": true,
5+
"settings": {
6+
"addresses": "__ALERT_EMAIL__",
7+
"singleEmail": true,
8+
"subject": "[{{ index .GroupLabels \"grafana_folder\" }}] {{ index .GroupLabels \"alertname\" }}",
9+
"message": "{{ range .Alerts }}{{ .Annotations.summary }}\n\n{{ .Annotations.description }}\n\nNetwork: {{ .Labels.network }}\nCurrent: {{ .Labels.current }}\nThreshold: {{ .Labels.threshold }}\n{{ end }}"
10+
}
11+
}

0 commit comments

Comments
 (0)