monitoring

Lumenqraph Monitoring

This directory contains Prometheus alert rules and Grafana dashboards for monitoring Lumenqraph deployments.

Files

• prometheus_alerts.yml - Prometheus alert rules for indexer health, error rates, and lag
• grafana_dashboard.json - Grafana dashboard visualizing key metrics

Quick Start

Using Docker Compose

Add to your docker-compose.yml:

prometheus:
  image: prom/prometheus:latest
  ports:
    - "9090:9090"
  volumes:
    - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
    - ./monitoring/prometheus_alerts.yml:/etc/prometheus/prometheus_alerts.yml
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'

grafana:
  image: grafana/grafana:latest
  ports:
    - "3000:3000"
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=admin
  volumes:
    - grafana_data:/var/lib/grafana
  depends_on:
    - prometheus

volumes:
  grafana_data:

Prometheus Configuration

Create a monitoring/prometheus.yml:

global:
  scrape_interval: 30s
  evaluation_interval: 30s
  external_labels:
    cluster: 'production'

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  - 'prometheus_alerts.yml'

scrape_configs:
  - job_name: 'lumenqraph-api'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/metrics'

  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Importing the Grafana Dashboard

1. Open Grafana (default: http://localhost:3000, admin/admin)

2. Add Prometheus data source:

   • Click "Configuration" → "Data Sources" → "Add data source"
   • Select "Prometheus"
   • URL: http://prometheus:9090
   • Click "Save & Test"

3. Import dashboard:

   • Click "+" (Create) → "Import"
   • Upload grafana_dashboard.json or paste the JSON directly
   • Select the Prometheus data source
   • Click "Import"

Metrics Overview

Indexer Metrics

• lumenqraph_indexer_lag_ledgers - Ledgers behind chain tip (lower is better)
• lumenqraph_indexer_last_processed_ledger - Most recent ledger processed
• lumenqraph_indexer_chain_tip_ledger - Latest ledger on chain
• lumenqraph_indexer_ingested_total - Total events ingested (cumulative)
• lumenqraph_indexer_errors_total - Total indexer poll-cycle errors (cumulative)

API Metrics

• lumenqraph_api_requests_total - Total HTTP requests served (cumulative)

Data Metrics

• lumenqraph_events_total - Total events stored in database

Alert Rules

Critical Alerts

• IndexerStalled - No events ingested in 5+ minutes but lag exists
• IndexerLagCritical - Lag > 500 ledgers for 2+ minutes

Warning Alerts

• IndexerLagHigh - Lag > 100 ledgers for 5+ minutes
• IndexerErrorRateHigh - Error rate > 1% over 5 minutes
• LargeLagGrowth - Lag grew > 1000 ledgers/hour
• IngestRateLow - Processing < 1 event/sec for 10+ minutes
• APINoRequests - No API requests for 5+ minutes

Customization

Adjusting Alert Thresholds

Edit prometheus_alerts.yml to adjust:

• Lag thresholds (currently 100/500 ledgers)
• Duration before firing (currently 2-10 minutes)
• Error rate thresholds (currently 1%)

Adding More Panels

The Grafana dashboard can be extended by:

1. Opening it in edit mode in Grafana
2. Adding new panels with additional metrics
3. Exporting the updated JSON back to this directory

Production Recommendations

1. Set up AlertManager for alert routing/aggregation
2. Configure persistent storage for Prometheus data
3. Use external Grafana instance for high availability
4. Set up log aggregation (ELK/Loki) for detailed debugging
5. Monitor Prometheus itself with additional scrape jobs
6. Establish on-call rotation for critical alerts

Integration with CI/CD

Store these files in your repository and:

• Deploy Prometheus with the alert rules via your deployment pipeline
• Use Grafana provisioning API to auto-import the dashboard
• Version control all monitoring configurations

Troubleshooting

Metrics not appearing in Grafana?

• Verify Prometheus can reach /metrics endpoint
• Check Prometheus targets at http://prometheus:9090/targets

Alerts not firing?

• Check Prometheus rule evaluation at http://prometheus:9090/alerts
• Verify alert thresholds match your environment

Dashboard showing no data?

• Ensure Prometheus data source is configured correctly
• Check time range selection in dashboard
• Verify metrics are being scraped (check Prometheus targets)