Health endpoint checks all backends sequentially — latency grows linearly with backend count

[aryanmehrotra]

Problem

container/health.go calls HealthCheck() on SQL, Redis, PubSub, and all registered external datasources one after another. Each check has its own timeout (typically 1s). With 4+ backends configured, worst-case response time for the health endpoint is 4+ seconds if backends are slow or timing out.

These checks are independent of each other — there's no reason they can't run concurrently.

Additionally, there's no caching. Every hit to the health endpoint re-checks every backend from scratch. If something is polling health every 5 seconds (which is common for k8s liveness/readiness probes), that's a lot of redundant backend pinging.

Suggestion

1. Run health checks concurrently using errgroup or a similar pattern. Response time becomes the max of individual checks rather than the sum.
2. Consider a short TTL cache (e.g., 5-10 seconds) for health results, especially for liveness probes that don't need real-time accuracy.

Where to look

• container/health.go — the Health() method that sequences all checks
• datasource/redis/health.go, datasource/sql/health.go — individual health check implementations

[KushalP47]

@aryanmehrotra I want to pick to pick this up, can you assign it to me?