feat: Add health monitoring server with comprehensive system checks, closes #49

[ricardo-perello]

feat: Add health monitoring server with comprehensive system checks, closes #49

Overview

This PR adds a comprehensive health monitoring server to rindexer that provides real-time monitoring of system components and services.

Features Added

Health Server Module

• New health server module with HTTP endpoint at /health
• Real-time monitoring of database, indexing, and sync status
• Health configuration in manifest with default port 8080
• Integration into all CLI start commands (indexer, graphql, all)

Health Checks Implemented

• Database connectivity and health - Verifies PostgreSQL connection status
• Indexing service status - Monitors active indexing tasks and service state
• Data synchronization status - Checks PostgreSQL tables or CSV files for data presence
• Overall system health - Provides timestamped health status with service breakdown

Dependencies Added

• axum - HTTP server framework for the health endpoint

Configuration

• Health server enabled by default in new project templates
• Backward compatible - existing manifests without health configuration use sensible defaults
• Parallel operation - runs alongside existing services without impacting core indexing functionality

Technical Details

Health Endpoint Response

{
  "status": "healthy",
  "timestamp": "2025-09-04T02:48:40.370651Z",
  "services": {
    "database": "healthy",
    "indexing": "healthy", 
    "sync": "healthy"
  },
  "indexing": {
    "active_tasks": 1,
    "is_running": true
  }
}

Health Server Architecture

• Isolated postgres client - Health server creates its own database connection for monitoring without interfering with main indexer operations
• Non-blocking operation - Runs in separate tokio task alongside GraphQL server
• Graceful error handling - Continues operation even if health checks fail

Bug Fixes

• Fixed database race condition - Resolved issue where health server was interfering with main indexer's database schema setup
• Proper client isolation - Health server now only creates monitoring client without calling setup_postgres()

Examples Updated

• Updated examples to include health server configuration
• Health server runs on port 8080 by default
• Accessible at http://localhost:8080/health

Testing

• ✅ Health server starts successfully alongside indexer
• ✅ Database connectivity monitoring works
• ✅ Indexing status monitoring works
• ✅ No interference with core indexing functionality
• ✅ Backward compatibility maintained

Closes

Health monitoring requirements #49

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
rindexer-documentation	Ready	Preview	Comment	Sep 26, 2025 10:19am

[joshstevens19]

good PR just need some general cleaning up / making it more readable and a few other changes

[ricardo-perello]

Here's the PR text with a small explanation about the SQL query:

---

PR Review Response - All Feedback Addressed

Hi @joshstevens19! Thanks for the detailed feedback. I've addressed all the issues you highlighted. Here's what I've implemented:

Port Conflict Validation

Issue: Need to cross-check GraphQL vs health server ports to prevent conflicts
Solution: Added port conflict validation in start.rs that checks both ports before starting either service and bails out with a clear error message if they match.

Simplified Configuration Architecture

Issue: Health server configuration was clunky with separate manifest section
Solution:

• Moved health configuration to global.health_override_port (always enabled by default)
• Removed separate HealthSettings struct and health.rs manifest file
• Health server now runs on port 8080 by default, configurable via global settings
• Much cleaner and more intuitive configuration

Code Quality Improvements

Issue: Large health_handler function was hard to read
Solution: Refactored into smaller, focused functions:

• build_health_status() - constructs the response object
• check_database_health() - database connectivity check
• check_indexing_health() - indexing service status
• check_sync_health() - data synchronization status
• determine_overall_status() - calculates overall health

Performance Optimizations

Issue: Database queries were inefficient
Solution:

• Replaced COUNT(*) queries with query_one_or_none() for better performance
• Uses EXISTS logic instead of counting all rows
• Health server now uses initialize_database() function for proper client setup

Database Sync Health Check: The sync health check uses an optimized SQL query that checks for the existence of user data tables while filtering out system tables:

SELECT 1 FROM information_schema.tables 
WHERE table_schema NOT IN ('information_schema', 'pg_catalog', 'rindexer_internal') 
AND table_name NOT LIKE 'latest_block' 
AND table_name NOT LIKE '%_last_known_%' 
AND table_name NOT LIKE '%_last_run_%' 
LIMIT 1

This efficiently determines if any meaningful user data exists (returns 'healthy') or if the indexer hasn't synced any events yet (returns 'no_data').

Better Integration

Issue: Health server wasn't properly integrated with existing systems
Solution:

• Health server follows indexer lifecycle (only runs when indexer is running)
• Uses existing initialize_database() function instead of custom postgres setup
• Proper error handling and graceful degradation
• No interference with core indexing functionality

The health server is now production-ready with a clean, maintainable codebase that follows all the feedback you provided. All the architectural concerns have been addressed while maintaining the core functionality.

[joshstevens19]

2 small comments and also need to add docs about health etc