More elaborate monitoring capabilities

[asgeirn]

I got bitten by a issue during OAuth login the other day, and actually spent quite some time before I was able to sort out the issue; the underlying PVC had reached capacity and the database writes for the OAuth callback failed / timed out.

This got me thinking of the (somewhat) clunky probes setup in the Helm chart as well -- if you have only OAuth and Apikey enabled, you're setting yourself up for a Catch-22 since the probes will always fail, meaning you can never log in and create a new ApiKey, even if you try to use the REST API to create one.

So: I was thinking of a more elaborate probe endpoint, something along the concepts of the Spring Actuator, where you have a separate endpoint with the possibility to report more detailed health status. In addition the UI could report storage consumption as well.

What I had in mind was:

1. Add a health endpoint, probably /health/liveness, /health/readiness, and /health/startup (or the two first ones collapsed into /health)
2. Separate authentication or even no authentication
3. Probes for overall health; disk usage, system load average, GC health
4. Some means for extensions to add probes

What do you think?

[rchincha]

https://kubernetes.io/docs/reference/using-api/health-checks/

^ these have been a community ask for a while.

[asgeirn]

Ok, so perhaps incorporate sigs.k8s.io/controller-runtime/pkg/healthz and github.com/shirou/gopsutil/v4 to set up monitoring. Should it be implemented as an extension?

[rchincha]

The answer to this question may be the question "would folks run a minimal zot in k8s". If yes, then it should just be a route. Else, it can be an extension. I suspect it is the former.

[rchincha]

@asgeirn let us know if you plan to work on a PR for this.

[rchincha]

#2487

[asgeirn]

@rchincha yes, I'll get started on a PR for this. I think I'll go for a bare-bones version first, it can be extended later if more functionality is needed.

[asgeirn]

#3228