Implement monitoring and metric

[GideonBature]

Overview

This PR implements a comprehensive monitoring and metrics system for the backend application, providing observability through Prometheus-compatible metrics, enhanced health checks, and structured logging.

Changes

🔍 Core Metrics System

• New MetricsService (backend/src/service/metrics_service.rs)
  • Centralized metrics collection using Prometheus client library
  • Tracks key application metrics: request count, error rates, latency, active connections
  • Configurable alert thresholds with severity levels (Warning, Critical)
  • Thread-safe implementation using lazy_static and atomic operations

🏥 Enhanced Health Checks (backend/src/http/health.rs)

• Liveness probe (/health/live) - Simple process alive check for Kubernetes
• Readiness probe (/health/ready) - Database connectivity and detailed health status
  • Returns HTTP 503 when service is not ready (database disconnected)
  • Includes database pool metrics (pool size, available connections)
  • Tracks service uptime
• Basic health check (/health/health) - Lightweight endpoint for load balancers

📈 Metrics Endpoints (backend/src/http/metrics.rs)

• /metrics - Prometheus text exposition format for scraping
• /metrics/json - JSON format with:
  • uptime (seconds)
  • requestCount
  • errorRate (percentage)
  • activeConnections
  • dbPoolConnections
• /metrics/alerts - Active alerts based on configured thresholds

🔧 Middleware Updates (backend/src/middleware/metrics.rs)

• Request tracking middleware now uses MetricsService
• Records request duration, status codes, and error rates
• Tracks active connection counts
• Enhanced structured logging with additional context fields

📝 Telemetry Improvements (backend/src/telemetry.rs)

• JSON logging support - Set LOG_FORMAT=json for structured logs
• Configurable log filters and formats
• Better support for log aggregation systems (Datadog, Elasticsearch)
• Utility macros for operation spans and structured events

API Routes Added

GET /health/live          → Liveness probe
GET /health/ready         → Readiness probe  
GET /metrics              → Prometheus metrics
GET /metrics/json         → JSON metrics
GET /metrics/alerts       → Alert status

Key Features

✅ Prometheus-compatible metrics export
✅ Kubernetes-ready health probes
✅ Alert system with configurable thresholds
✅ Database connection pool monitoring
✅ Request/response tracking
✅ Structured JSON logging for production
✅ Thread-safe metrics collection

Configuration

The system supports environment-based configuration:

• LOG_FORMAT=json - Enable JSON structured logging
• Default log level: info (configurable via RUST_LOG)

Testing

• Health endpoints return appropriate status codes
• Metrics are properly exported in Prometheus format
• Alert thresholds trigger correctly
• Database health checks detect connectivity issues

Deployment Notes

This implementation is designed for cloud-native deployments:

• Use /health/live for Kubernetes liveness probes
• Use /health/ready for Kubernetes readiness probes
• Configure Prometheus to scrape /metrics endpoint
• Consider setting up alerts based on /metrics/alerts endpoint

Closes #16

[ONEONUORA]

Nice implementation @GideonBature

[ONEONUORA]

@GideonBature Pls fix the conflict

[ONEONUORA]

@GideonBature Pls fix the conflict

[GideonBature]

noted, fixing it now boss!