Skip to content

Expand the health check endpoint #237

Description

@lcox74

UQCSbot has a single HTTP endpoint at / whose entire purpose is responding 200 OK with the plain text UQCSbot is running. It exists because Azure App Service (which I've been told is how this is deployed) ignores Docker's built-in HEALTHCHECK and insists on its own HTTP endpoint to poke, so we hand it the smallest possible thing that would make it stop complaining.

A quick trip down memory lane

We deployed a completely rewritten Dockerfile, did a quick lgtm over the diff, and merged with confidence. Around 15 minutes later the bot still hadn't restarted; for reference, the last deploy a few days ago took less than a minute to restart.

@JamesDearlove (infra master) opened the Azure portal to find out why. In order:

  • The portal had changed again, and nothing was where it was last week.
  • Logs? What logs?
  • There was no sign of which image it was even running after manual deploy.
  • Copilot?

So @peclarke (lgtm master who was begging for bed), Jimmy and I sat there in a call while filling the #bot-testing channel with banter to lure the bot into functioning. The collective stress on that call could have put a therapist in hospital. The health endpoint was working, but was useless to us.

Proposal

Change / to return a JSON object with enough information to actually debug a deploy. This shouldn't break the Azure deployment ™️. Here's the proposed object:

{
    "build": {
        "git_sha": "xxxxxxxxxxxxxxxxxxx",
        "time": "2026-05-23T10:10:00Z"
    },
    "discord": {
        "ready": true
    },
    "status": "lgtm",
    "uptime_seconds": 12345
}

The shape and fields are up for discussion. We could add a database.ready flag, discord server metadata, or other constants. But the fields in the build object are the ones that will really help us. It exists so that next time, we can answer "which build is alive, and is this Paul's fault?" instantly instead of chaotic Azure portal archaeology.

Implementation notes

Here are a couple of notes that might help anyone working on this or anyone who has opinions on how this should work.

  • Build metadata: the git commit sha and build time can be Docker build ARGs in the Dockerfile, baked into the image and read at runtime as ENV vars.
  • Inject in CI, Defaults for local: real values come from the CI pipeline at build time. Compose/dev won't have those, so we want fallback defaults like dev or unknown rather than just crashing or returning blank strings.
  • Discord status: this can be pulled from the bot client at request time, don't cache it.

Open questions

A couple of other things I would like some opinions from infra are:

  1. Expand /, or add a new endpoint? This is really about backwards compatibility. We could just leave the default endpoint and add a /livez or /readyz with the JSON response. But it just depends on what you want the / to actually do.

  2. A degraded status? Would you want to have the status change if the discord client is cooked? Does Azure support any alerts to make this even useful to begin with?

  3. DB Check? We could add the database.ready which could do a SELECT 1 on the DB client to check if the database is even working. The caveat is that it will make a DB request on every health check, which is probably fine as it shouldn't completely butcher the connections under a health check attack.

  4. Logging? Each health check will log, there aren't many other logs done in the bot from what I've seen on my little dev environment. So should we drop the health endpoint from the web access logs so we don't completely fill the logging up with health checks?


The point of all this: the health endpoint graduates from confirming that a process is running to telling us which build is running, whether Discord bot client is functional, and how long it's been up. So the next deploy that goes sideways is a 30s look instead of liquor cabinet abuse.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions