feat: add hot wallet low liquidity alerts

[AndresQuijano]

What

Modify scripts and json files to include two new alerts and use custom format for the mail notifications of two of the alerts of hot/cold wallet feature:

1. Low liquidity warning
2. Extreme low liquidity warning

Why

This is a POC of the research done by dev ops team to extract information from the body of the log message so we can send more personalized messages including precise information in the alert email.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change
• Documentation update
• Refactoring (no functional changes, no api changes)
• Performance improvement
• Test updates
• Security fix
• Deployment/Infrastructure changes

Affected part of the project

• Management UI / API
• PegIn flow
• PegOut flow
• Utility scripts
• Configuration files
• Metrics and alerting

Related Issues

Jira ticket

How to test

Run the local env
Execute the script to create the alerts in Grafana:
export ALERT_RECIPIENT_EMAIL=test@iovlabs.org && bash docker-compose/monitoring/scripts/import-alerts.sh

Go to Grafana and check that everything is there (use admin/test as credentials):
http://localhost:3000/alerting/list
Six alerts should be created.

There are two ways to test the functionality once the alerts are created:

The first option is to use the approach described in #925 but this time running the LPS in Docker. This approach test the full integration between LPS and monitoring stack.

The second option or probably a complementary one is to directly insert a message in the log as the LPS would do when the event/alert happens:

For warning low liquidity
echo "time=\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" level=info msg=\"Alert! - Subject: Hot wallet: Low liquidity, refill recommended | Recipients: test@iovlabs.org | Body: Network: BTC | Current: 2.499869500000000000 | Threshold: 3.000000000000000000\"" >> docker-compose/local/volumes/lps/logs/lps.log

For critical low liquidity
echo "time=\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" level=info msg=\"Alert! - Subject: Hot wallet: Critical low liquidity, refill required | Recipients: test@iovlabs.org | Body: Network: BTC | Current: 0.5 | Threshold: 3.000000000000000000\"" >> docker-compose/local/volumes/lps/logs/lps.log

Go to Grafana and check that the alerts are in state "Firing".

Then go to Mailhog
http://localhost:8025/#

and check the format of the notifications.

Screenshots

Critical:

Warning:

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Files

[Copilot]

Pull request overview

This PR extends the monitoring/alerting provisioning scripts to add hot wallet low-liquidity alerts and to support custom email formatting by provisioning Grafana contact points and notification policy routing.

Changes:

• Added two new Grafana alert rules: hot wallet low liquidity (warning + critical) with LogQL regexp extraction.
• Added a custom email contact point template for low-liquidity alerts.
• Enhanced the import script to provision contact points and configure a notification policy (routing specific alerts to custom contact points).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
docker-compose/monitoring/scripts/import-alerts.sh	Imports alerts and now also provisions contact points + notification policy routing using __contact_point__ metadata and ALERT_RECIPIENT_EMAIL.
docker-compose/monitoring/scripts/contact-points/low-liquidity.json	Adds a custom email template (subject/body) for low-liquidity alerts.
docker-compose/monitoring/scripts/alerts/hot-wallet-low-liquidity-warning.json	New warning alert rule using LogQL regexp capture groups for notification details.
docker-compose/monitoring/scripts/alerts/hot-wallet-low-liquidity-critical.json	New critical alert rule using LogQL regexp capture groups for notification details.
docker-compose/monitoring/scripts/README.md	Documents the new alerts, contact points, and updated import behavior/prereqs.

---

You can also share your feedback on Copilot code review. Take the survey.