Monitoring Health of Unleash Deployed In a Private Cloud Environtment

[msanssouci]

Hello!

This discussion is related to issue 9965.

We have containerized Unleash and deployed to a private cloud environment connected to Postgres. We have configured a health check for /health and the health check does return 200. However, if Postgres becomes unavailable for any reason, Unleash still reports healthy. As described in the related issue, I would expect the health check to return 500 if Unleash's infrastructure dependencies are not reachable. Based on the documentation update resulting from the issue, it appears Postgres connectivity is intentionally not included in the health check. Is including Postgres in the health check something you would consider adding?

Also, I am interesting in learning best practices for monitoring the health of Unleash in a private cloud environment.

Thanks!

[gastonfournier]

Hi @msanssouci it comes with the trade-off of consuming a DB connection to check it (e.g. running SELECT 1;). We've been considering splitting readiness (application is ready to serve traffic) from health (application is up). This might help with this case. WDYT?

[msanssouci]

@gastonfournier IMO, the tradeoff of consuming a DB connection is worth it. The readiness check of your application should be an aggregate of your ability to serve traffic and the health of the infra you depend on. I have tested the case where postgres is unavailable on startup confirming that Unleash exits correctly.

I am hoping the readiness check logic can be applied to your liveliness check as well. Given that the readiness/liveliness are generally always the same, we leverage a single health endpoint for containers deployed in our private cloud environment.

Hope that helps!

Mitch

[gastonfournier]

Hi @msanssouci thanks for the insights. I do believe that readiness and health checks should be different: https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes. In our hosting we don't want to adopt this readiness check because we have different monitors and alerts for the database. It doesn't give us additional value to also check the database health inside Unleash itself.

Despite that, having a different readiness check is something we've been willing to do, and we can always allow you to opt in for the database check inside Unleash. So we've added this issue to keep track of this request: #10742

[msanssouci]

Thank you @gastonfournier! Should I mark your response as the answer to close the discussion?

[gastonfournier]

Sure, and feel free to add yourself as a watcher of the issue, so whenever we get that out, you'll be notified