Model health check support

[dotmobo]

Hi

Support of health check to monitore all served models through a monitoring app like Centreon.

See : https://docs.litellm.ai/docs/proxy/health

Regards

Morgan

[leoguillaume]

Context - why is this issue relevant?

Users of OpenGateLLM need to be informed when models become unreachable, degraded, or overloaded in order to implement fallback strategies toward alternative models and maintain service reliability.

Today, clients have no standardized way to determine whether a model is healthy, saturated, or unavailable. This makes it difficult to build resilient applications and automatic failover mechanisms.

Objective – what is the aim of this issue?

The goal of this issue is to implement a health check endpoint allowing users to retrieve real-time status information about OpenGateLLM models.

This endpoint should expose a normalized service health indicator for each model based on runtime metrics and load conditions.

Expected outcomes – what do we expect at the end of this issue (concrete outcomes)?

Outcomes

• Creation of a dedicated health check endpoint for models.

• Exposure of a health status indicator for each model with the following states:

  1. healthy → the service is operating normally
  2. degraded → the service is overloaded or experiencing degraded performance
  3. unavailable → the service cannot process requests

• Definition of dynamic thresholds based on load-testing metrics such as:

  • Time To First Token (TTFT)
  • End-to-end latency
  • Number of concurrent requests
  • Error rate

• Automatic and dynamic state transitions based on observed runtime metrics.

• Ability to observe and validate status transitions during load-testing scenarios.

Acceptance criteria

• A new endpoint exposes the health status of every available model.
• Health statuses are computed dynamically from runtime and load-test metrics.
• Thresholds defining healthy, degraded, and unavailable states are configurable and documented.
• Service states change automatically when metrics cross defined thresholds.
• Load-testing scenarios can reliably trigger and demonstrate each health state.
• The endpoint response format is stable and documented for client-side integrations and fallback orchestration.