Better graceful shutdown for highly avaliable services?

[bittrance]

When quarkus.graceful.timeout is set, Quarkus starts responding to requests on existing persistent HTTP connections with status code 503 and closing the connection. This is not optimal when Quarkus is used to implement REST APIs (e.g. via Spring Boot) where clients tend to make requests using a pool of persistent connections towards the API. In particular, when running a highly available service, there will be other (newly started) instances of the service available which could process the request. In that scenario, closing idle connections will allow load balancers to direct the request to a ready instance. How can I override this behavior? In my case, I would like to process the request as normal and then close the connection as per https://www.rfc-editor.org/rfc/rfc9112.html#name-tear-down.

[cescoffier]

This is a good question. We want to improve the graceful shutdown (not only for HTTP). @ozangunalp has been looking into it, but no action has been decided yet. I feel this will become a working group.

[bittrance]

We want to improve the graceful shutdown (not only for HTTP)

I think there are two (types of) questions at play here:

1. shutting down components in a reasonable order, hopefully traversing a DAG backwards,
2. the procedure by which a particular component unwinds its ongoing work,

I suppose that the current 503 is because the HTTP server does not know that it is still safe to process requests, but I think that in a large fraction of all cases, the developer can know that it is safe and would thus like to control how unwinding is done? I'm only asking about 2. and can live with the Quarkus current shortcomings which seem to relate mostly to 1?

Specifically about unwinding persistent HTTP connections, there are some more details in my recent blog post on the subject https://bittrance.github.io/posts/graceful-shutdown-for-rest-apis/ . In short, I have found two reasonable graceful strategies related to REST and similar:

1. closing the connection immediately (respecting in-flight requests). This prioritizes speedy shutdown, but requires that the shutdown logic is able to iterate over the connection pool, preferably including connection state.
2. awaiting additional requests so that it can be processed as normal and responded to with Connection: close. This can be done by a normal middleware which has access only to the current connection (plus a timeout to put an upper bound on shutdown time).

I think a framework that takes it HTTP seriously will need to provide both strategies. When you have a bursty low-frequency load, you really want 1 and when you round-robin 10k req/s, you really want 2.

[cescoffier]

io.quarkus.vertx.http.runtime.filters.GracefulShutdownFilter waits for all inflight requests to be handled. We do not close the connections; we just track requests. The reason is that with multiplexed connections, it's safer (HTTP/2, HTTP/3).

About 2 - you can do that already using the shutdown event with a priority: https://quarkus.io/guides/lifecycle#graceful-shutdown.

[cescoffier]

BTW, you can still implement the "close" in a route if necessary.