feat(shutdown): wait for background tasks before closing the pool

[bilhokista]

Closes #1126.

Requirement 1 was already done

main.rs already has shutdown_signal() handling both signal::ctrl_c() and Unix SIGTERM, wired into axum::serve(...).with_graceful_shutdown(...). I did not touch it.

Requirement 2 was missing, and there was a bug behind it

The grace window did not exist, and the shutdown sequence had a concrete flaw:

axum::serve(listener, app).with_graceful_shutdown(shutdown_signal()).await?;

drop(shutdown_tx);        // ask background tasks to stop
db_pool.close().await;    // …and immediately close the pool

Signalling and waiting are not the same thing. drop(shutdown_tx) wakes the watchdog and the webhook dispatcher, but nothing waits for them, so db_pool.close() runs while a task may be mid-transaction — exactly the "in-flight database transactions" the issue names.

It could not have waited even if it wanted to: both InactivityWatchdogService::start and WebhookDispatcherService::start discarded the JoinHandle from tokio::spawn. They now return it, main collects all three tasks (the metrics loop included, under its feature flag), and await_background_tasks joins them under a 15-second timeout before the pool closes.

Three decisions worth review

The signal is now sent explicitly. drop(shutdown_tx) happens to wake receivers, but it says nothing about intent and stops working the moment any task holds a sender clone. shutdown_tx.send(true) says what it means. The result is deliberately discarded with a comment, because main still holds a receiver so the send cannot fail — I would rather note that than write an error branch that can never run.

The grace window is bounded, not infinite. Kubernetes and Docker both send SIGKILL 30 seconds after SIGTERM by default, so waiting forever converts a clean exit into a killed one. Fifteen seconds leaves room for an in-flight transaction and still lands well inside that budget. There is a test asserting the constant stays under 30s.

A panicking task does not abandon the others. Each handle is awaited individually and a JoinError is logged rather than propagated, so one bad task cannot cause the rest to be dropped mid-work. Tested.

On timeout the pool closes anyway, with a warning — the alternative is hanging past the orchestrator's patience and being killed, which is strictly worse. The warning matters: it is the trace to look for if a deploy later shows odd data.

Verification

This one I could genuinely execute. I lifted await_background_tasks and SHUTDOWN_GRACE byte-for-byte into a scratch crate with real tokio and ran them with cargo test. All 5 pass:

• an empty task list returns immediately
• three staggered tasks are all awaited — asserted by elapsed time, so returning early would fail rather than silently pass
• a task that outlives the window causes a timeout, and the call still returns promptly
• a panicking task is logged and the sibling is still awaited to completion
• the grace constant sits inside a typical orchestrator kill budget

Honest note: what I could not run is the full cargo test for the real crate (it needs the whole dependency graph and sqlx's database or offline metadata), so the two start() signature changes and the wiring in main are unverified beyond review. They are mechanical — adding a return type and removing a trailing semicolon so the spawn becomes the return expression — but CI is the check.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CrfEY1tvXrbeMDAUzxfuk7