-
Bug ReportVersion├── axum v0.8.7 PlatformLinux 6.14.0-34-generic #34~24.04.1-Ubuntu DescriptionWhen using graceful_shutdown with TcpListener created with from_raw_fd the shutdown hangs until further requests which fail are made. The issue can also be reproduced with a blocking TcpStream created with from_raw_fd instead of TcpListener directly. Short reproduction of the bug: use std::{os::fd::FromRawFd, time::Duration};
use axum::{Router, response::IntoResponse};
use tokio::{net::{TcpListener}, time};
use tracing_subscriber::{EnvFilter, fmt, layer::SubscriberExt, util::SubscriberInitExt};
async fn timeout() -> () {
time::sleep(Duration::from_secs(15)).await;
println!("Graceful shutdown");
}
#[tokio::main]
async fn main() {
let _log = tracing_subscriber::registry()
.with(EnvFilter::from_default_env().add_directive("trace".parse().unwrap()))
.with(
fmt::layer()
.with_ansi(true)
.with_level(true)
.with_target(true),
)
.try_init();
let listener = unsafe { std::net::TcpListener::from_raw_fd(3) };
let tokio_listener = TcpListener::from_std(listener).unwrap();
let request_handler = async |input: String| input.into_response();
let router = Router::new().fallback(request_handler);
axum::serve(tokio_listener, router)
.with_graceful_shutdown(timeout())
.await
.unwrap();
}What happens: What should happen: Another reproduction I managed to make is replacing the TcpListener from the earlier example with let stream = unsafe { std::net::TcpStream::from_raw_fd(3) };
stream.set_nonblocking(false).unwrap(); // Works as intended when true, hangs when false
let socket = tokio::net::TcpSocket::from_std_stream(stream);
socket.set_nodelay(true).unwrap();
let tokio_listener = socket.listen(1024).unwrap(); |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Just to confirm the following makes it work as intended let mut non_blocking: c_int = 1;
let res = unsafe { libc::ioctl(3, libc::FIONBIO, &mut non_blocking) };As I'm not too familiar with axum I am not sure if it could feasibly be fixed so that graceful shutdown works well even if the fd is blocking and if setting it to non-blocking has some other consequences I'm unaware of. |
Beta Was this translation helpful? Give feedback.
-
|
axum expects to be run in a cooperative scheduler, in most cases tokio. Basically, there is a background task which just loops and tries to accept new connections, if it does, it spawns another task to handle that new connection and then listens to new ones. This works well when the accepting operation is non-blocking, if there's no waiting connection, the thread can be reused for other tasks, if you want to cancel it, you just drop it. On the other hand, if you're using a blocking listener, the thread will have to wait there until there is a new connection and then it can return that. Blocking tokio threads is a bad idea (there's been written a lot about that already) but in this case it also means that if you want to shut down the server, the runtime cannot just kill the task, because the scheduler waits before the task cooperatively yields, but the task is blocked on waiting for the new connection. This is expected behavior and you should really provide a non-blocking listener not just for graceful shutdown, but also for better performance with tokio. The only way to "fix" this would be to spawn the accepting task as its own thread and block on that thread. It is possible by implementing a custom |
Beta Was this translation helpful? Give feedback.
-
|
Makes sense, now that I kind of got a hang of the blocking being an issue and looking at for example std::net::TcpListener and tokio::net::TcpListener more carefully I found discussion online about other people accidentally running into this footgun. Also seems that catching this would have been easier if only I was testing socket activation with debug builds instead of release, as tokio apparently debug asserts this, learning how to read would probably help too as now that I'm looking at it |
Beta Was this translation helpful? Give feedback.
axum expects to be run in a cooperative scheduler, in most cases tokio. Basically, there is a background task which just loops and tries to accept new connections, if it does, it spawns another task to handle that new connection and then listens to new ones. This works well when the accepting operation is non-blocking, if there's no waiting connection, the thread can be reused for other tasks, if you want to cancel it, you just drop it.
On the other hand, if you're using a blocking listener, the thread will have to wait there until there is a new connection and then it can return that. Blocking tokio threads is a bad idea (there's been written a lot about that already) but in this case it…