Working Group Proposal - Graceful Shutdown

[cescoffier]

Target: Quarkus 4
V.0.2

Objective

Make Quarkus shutdown truly graceful by providing two complementary capabilities: (1) a mechanism for extensions to block incoming traffic and drain in-flight work to reach a quiescent state, integrated with readiness health checks, and (2) a well-ordered, phased shutdown API that gives extension developers clear control over when and how their shutdown logic executes.

The Problem

No Quiescence Mechanism

When a Quarkus application receives a shutdown signal, there is no standard way for extensions to transition gracefully from "serving" to "draining" to "stopped." The core issue is the lack of a quiescence protocol, a coordinated mechanism where:

1. The application signals it is no longer ready to accept new work (readiness probe goes unhealthy)
2. Incoming traffic is blocked at the ingress layer (load balancer, Kubernetes, service mesh)
3. In-flight requests, transactions, and background work are given time to complete
4. Only once the application reaches a quiescent state does actual resource teardown begin

Today, each extension that wants to participate in graceful shutdown must implement this pattern ad hoc. Most don't. The result is interrupted HTTP requests, half-processed messages, aborted scheduled jobs, and data loss during rolling deployments.

The readiness health check (/q/health/ready, SmallRye Health, Kubernetes readiness probes) is the natural signal to orchestrators that an instance should stop receiving traffic, but the shutdown sequence does not integrate with it in a first-class way. Extensions cannot easily declare "I am draining, mark me as not ready" and have that reflected in the health check automatically.

Confusing and Under-Documented Shutdown APIs

Quarkus currently has three separate shutdown API layers with unclear boundaries:

• Primordial API (ShutdownContext): addShutdownTask(Runnable) (reverse order) and addLastShutdownTask(Runnable), used via ShutdownContextBuildItem at build time. The addLastShutdownTask() name is universally disliked (at least we reached a consensus).

• Graceful Shutdown API (ShutdownListener): preShutdown(ShutdownNotification) and shutdown(ShutdownNotification) with done() callback, used via ShutdownListenerBuildItem. Also exposes @ShutdownDelayInitiated which maps to preShutdown.

• CDI layer: @Shutdown (fires ShutdownEvent before ArC shutdown), @BeforeDestroyed(ApplicationScoped.class) (fired when ArC shuts down), @PreDestroy on beans/interceptors.

There is no documentation explaining when to use which API. The actual shutdown sequence is only discoverable by reading the source code:

1. Graceful shutdown phase: ShutdownListener#preShutdown() → delay wait (if quarkus.shutdown.delay-enabled=true, duration = quarkus.shutdown.delay) → ShutdownListener#shutdown() → timeout wait

2. Shutdown tasks phase: ShutdownEvent fired → ArC shutdown (@BeforeDestroyed(ApplicationScoped.class), @PreDestroy)

3. Last shutdown tasks phase: Tasks registered via addLastShutdownTask()

No Cross-Extension Ordering

There is no way to express that the shutdown logic from extension A should execute before or after the logic from extension B. Extensions shut down in an undefined order, which causes real issues; for example, a scheduler extension may try to complete a job that requires a datasource that has already been closed.

No Debug/Tracing Tools

There is no way to log, trace, or inspect the shutdown sequence at runtime. When shutdown hangs or misbehaves, developers have no tooling to identify which task is blocking or how tasks are ordered.

Limited Extension Coverage

Many extensions do not participate in graceful shutdown at all. For example, quarkus-scheduler does not drain running or scheduled jobs before shutdown, which can cause work to be interrupted.

The Proposed Solution

The work is organized around two complementary parts.

Part 1: Pre-Shutdown Phase (Traffic Draining)

Design and implement a standard protocol for extensions to participate in graceful traffic draining, allowing the application to reach a quiescent state before any ordering-sensitive teardown begins. This phase is independent of the dependency-ordered teardown in Part 2 — it runs first and is about stopping new work, not tearing down resources.

Traffic blocking and readiness integration:

• When shutdown is initiated, the application's readiness health check (/q/health/ready) immediately reports DOWN, signaling to Kubernetes / load balancers to stop routing new traffic
• Extensions can register as "drainable" components, when shutdown begins, they stop accepting new work and drain in-flight work
• A configurable drain timeout governs how long the system waits for quiescence before forcing teardown

Extension participation model:

• Extensions declare their draining capability and report when they have reached quiescence (no in-flight work remaining)
• The shutdown sequence waits for all drainable extensions to report quiescence (or for the drain timeout to expire) before proceeding to resource teardown
• This replaces the current ad hoc pattern where extensions individually implement ShutdownListener#preShutdown() with a done() callback

Key integrations:

• HTTP server: stop accepting new connections, finish in-flight requests
• Quarkus Messaging: stop polling for new messages, finish processing in-flight messages
• Scheduler: stop scheduling new jobs, wait for running jobs to complete
• gRPC server: send GOAWAY, drain active RPCs
• WebSocket: send close frames, wait for graceful close

Part 2: Dependency-Ordered Shutdown (and Startup) API

The correct shutdown order is the reverse of the correct startup order. Today, Quarkus has no explicit startup ordering API (only build-time ordering via build items), and the shutdown ordering is ad hoc. This part of the work normalizes both.

Dependency-oriented ordering:

• Design a proper startup ordering API that is dependency-oriented, so extensions can declare what they depend on
• Shutdown ordering is derived automatically as the reverse of the startup dependency graph — no separate shutdown ordering annotations needed
• This is a partial ordering (a DAG), not a linear sequence — independent extensions can shut down concurrently (see concurrent shutdown below)
• The ordering model should be type-based (like build items), not string-based, for consistency with Quarkus conventions

Clear API guidance:

• Document which API to use for each scenario with a decision guide
• Provide clear examples for extension developers and application developers
• Clarify the relationship between the Primordial API, Graceful Shutdown API, and CDI layer

Concurrent shutdown investigation:

• Coordinate with David's work on concurrent shutdown (running independent shutdown tasks in parallel)
• Ensure concurrent execution respects phase boundaries and ordering constraints
• Produce a recommendation on opt-in vs. automatic concurrency for independent tasks

Shutdown tracing and debugging:

• Investigate JFR events for shutdown activities
• Provide a debug mode that logs all shutdown tasks, their phase, ordering, and execution duration
• Surface shutdown activity in Dev UI during development

API naming improvements:

• Improve naming where possible (e.g., addLastShutdownTask()) within the Quarkus 4 breaking-change window

Definition of Done

Part 1: Pre-Shutdown

• Extensions can register as drainable components and signal completion
• Readiness health check automatically reports DOWN when shutdown is initiated
• The pre-shutdown phase waits for all drainable extensions to complete (or drain timeout)
• Graceful shutdown implemented for HTTP server, Reactive Messaging, quarkus-scheduler, gRPC, and WebSocket
• Configurable drain timeout with clear behavior when exceeded

Part 2: Dependency-Ordered Shutdown

• A dependency-oriented startup/shutdown ordering API exists (type-based, not string-based)
• Shutdown order is automatically derived as the reverse of startup order
• Shutdown sequence fully documented in the Quarkus guides with a clear diagram
• Decision guide published: which shutdown API to use for each scenario
• Shutdown activities can be traced/debugged (JFR events or equivalent)
• Concurrent shutdown investigation completed with clear recommendation
• Integration tests demonstrating correct shutdown ordering under load
• Shutdown debug information available in Dev UI

Scope of Work

In Scope

• Pre-shutdown phase: traffic blocking, drain signaling, readiness health check integration
• Startup ordering API (dependency-oriented) — shutdown ordering derived as its reverse
• Implementing graceful drain for key extensions (HTTP, Reactive Messaging, scheduler, gRPC, WebSocket)
• Dependency-ordered shutdown with cross-extension ordering
• API clarification, documentation, and decision guide
• Shutdown debug/tracing tooling (JFR events investigation, debug logging)
• Concurrent shutdown investigation and recommendation
• API naming improvements where feasible in the Quarkus 4 window

Out of Scope

• Merging ShutdownContext and ShutdownListener APIs
• Cluster-wide coordinated shutdown (multi-instance orchestration)
• Graceful startup (separate concern)

Organizing the Work

Communication

• Zulip channel: #wg-graceful-shutdown
• GitHub discussion: TBD (to be created)
• Points of contact:: @mkouba @ozangunalp

Timeline

• Target: Quarkus 4 release cycle

Existing Issues

• #45946
• #31072
• #38119
• #41233
• #52389
• #51327
• #50475
• #44894

[cescoffier]

@ozangunalp @dmlloyd @mkouba could you have a look to the proposal?

[dmlloyd]

It looks OK to me other than the suggested "Phased shutdown API" which AFAICT doesn't connect to anything that was proposed yet. The problem may or may not be solved in the suggested checklist order.

[cescoffier]

@mkouba told me there will be multiple phases :-)

[cescoffier]

Fair point. Part 2 was too prescriptive for this stage. I've reframed it as "design a dependency-oriented ordering model" rather than prescribing specific phases upfront. The concrete structure will emerge from the design work.

[holly-cummins]

Would we expect that in most cases the optimum shutdown order would be the reverse of the optimum startup order? I think we don't have any mechanism for chaining startup events, only build-time events (through build items), but I wonder whether some harmonisation across the three might make sense? That could give sensible defaults (maybe), and also a coherent mental model.

For example, built-order dependencies are done through types, and the proposed shutdown order is done through strings. And unless I'm forgetting something, the startup order doesn't exist. Well, except for an ad hoc API I put in for dev services.

[dmlloyd]

I would strengthen that and say: in all cases, the correct shutdown order is the reverse of the correct startup order. That said, startup and shutdown only strictly needs a partial ordering so it's more of a graph problem than a linear sequence.

But yes part of this effort will be to normalize a proper startup API that is dependency oriented.

When we discussed graceful shutdown the week before last, I think we more or less concluded that the graceful shutdown APIs are not connected to the graph order.

[cescoffier]

Agreed, the correct shutdown order is the reverse of the correct startup order, and the ordering model should be consistent with Quarkus conventions (type-based like build items, not string-based). I've updated the proposal to reflect this:

• Part 2 is now "Dependency-Ordered Shutdown (and Startup) API", the scope explicitly includes designing a proper startup ordering API that is dependency-oriented
• Shutdown ordering is derived automatically as the reverse of the startup dependency graph (a DAG/partial ordering), not declared separately via string annotations
• The pre-shutdown/drain phase (Part 1) remains independent of this graph, it runs first and is about stopping new work, not tearing down resources

The original @ShutdownOrder(after = "datasource") example has been removed in favor of the dependency graph approach.

[ozangunalp]

The word quiescence reminds me of previous discussions...

While it describes well the state we want to put the application in, I always think of it as a two-way road, where it is possible to come back up from a quiescent state. Here, we want to talk more about a "pre-shutdown" phase here, where the only option is to shut down the application. We can keep the word in here in the description, but during implementation and docs, I think pre-shutdown is easier to understand.

Also, here are even more existing issues : #38119, #41233, #52389, #51327, #50475, #44894

[cescoffier]

Good call on terminology. I've renamed the mechanism from "Quiescence" to "Pre-Shutdown Phase" in the proposal, clearer for users since there's no coming back. "Quiescent state" is still used in the problem description where it describes the goal (no in-flight work), but the API/mechanism naming uses "pre-shutdown."

Added all the extra issues to the proposal, thanks: #38119, #41233, #52389, #51327, #50475, #44894.

[mkouba]

Part 1: Pre-Shutdown Phase (Traffic Draining)

If I understand it corretly, all of this can be currently implemented with ShutdownListener API and its config, and it's actually implemented for HTTP and health check (/q/health/ready). Our goal is to revise/document/improve this API and use it in few other extensions, right? If not, do you have some specific ideas of what functionality is missing?

Changes to the CDI specification's @PreDestroy / @BeforeDestroyed semantics

Could you be more specific?

[cescoffier]

About Pre-Shutdown Phase - exactly. A lot more extension would be needed to participate (you mentioned the scheduler that should stop scheduling new tasks)

Regarding CDI, it's out of scope. Basically, we do not change anything to CDI ;-). But I agree it's confusing. I will remove that line.

[cescoffier]

Working group officially started: WG - Graceful Shutdown (view)

[jponge]

Hey folks, something I spotted while doing work on GracefulShutdownFilter{Test} following the bump to Vert.x 4.5.26: we do not pass any delay for HTTP/2.

At the end of sendGoAwayForHttp2 in GracefulShutdownFilter, we call connection.shutdown() whose implementation in Vert.x has a 30s default delay:

default Future<Void> shutdown() {
  return shutdown(30, TimeUnit.SECONDS);
}

[jponge]

#53320