EndpointDriver::drop panics on a poisoned mutex, escalating any accept-loop panic to a process abort

[yyyyy3155550]

Describe the bug

When the endpoint accept loop panics for any reason, the panic poisons the EndpointInner state mutex. During unwinding, impl Drop for EndpointDriver runs and calls self.0.state.lock().unwrap() (noq 1.0.0, noq/src/endpoint.rs:495), which panics again on the PoisonError. A panic while already unwinding through a destructor is a non-unwinding double panic, so the whole process aborts with SIGABRT (exit 134) instead of just losing the endpoint:

panic in a destructor during cleanup.
thread caused non-unwinding panic. aborting.

So any recoverable panic in the EndpointDriver task escalates to a hard crash of the entire host process. This is the same fatal mechanism noted in #465 ("cleanup hits poisoned mutexes and the process aborts"); #465's fix was for a specific first-panic cause, but the underlying Drop-poison-abort path is still present in 1.0.0.

We hit it in production via a relay disconnect, where the first panic is the active_connections -= 1 underflow at noq/src/endpoint.rs:648. That underflow is fixed on main by #717 (merged, but unreleased as of noq-v1.0.0). This issue is specifically about the second panic — the Drop impl aborting the process — which #717 does not address and which would still turn any other accept-loop panic into a full crash.

Version info

• noq 1.0.0 (tag noq-v1.0.0, 2026-06-15)
• iroh 1.0.0 (2026-06-15), used as a long-running receiver/endpoint
• Reproduced on macOS and Windows

Stack trace

thread '<iroh accept loop>' panicked at noq-1.0.0/src/endpoint.rs:648
    attempt to subtract with overflow            // active_connections -= 1 (fixed by #717)

thread '<iroh accept loop>' panicked at noq-1.0.0/src/endpoint.rs:495
    called `Result::unwrap()` on an `Err` value: PoisonError { .. }
    in <noq::endpoint::EndpointDriver as core::ops::drop::Drop>::drop

panic in a destructor during cleanup.
thread caused non-unwinding panic. aborting.

Trigger

A relay-server disconnect during normal operation:

Lost connection to relay server ... peer closed connection without sending TLS close_notify

This drives the connection into draining/teardown, hits the active_connections underflow (first panic, #717), then the poisoned-mutex unwrap() in EndpointDriver::drop (second panic) aborts the process. The Drop behaviour is independent of what caused the first panic, so #717 alone does not prevent the abort for other first-panic causes (cf. #465, #684, iroh#4289).

Expected behavior

A panic in the EndpointDriver task should at worst tear down the endpoint and surface as errors on the noq/iroh APIs (cf. #684), never abort the whole process. The Drop impl should not unwrap() a poisoned lock — e.g. recover via .lock().unwrap_or_else(PoisonError::into_inner), or otherwise guard the destructor so it can never panic.

Related

• fix(proto): Avoid active_connections underflow #717 — fixes the active_connections underflow (the first panic in our trace); merged, awaiting a release + an iroh noq bump.

• Endpoint::handle_event can panic on stale events after Drained (invalid key) #465 — same Drop-poison-abort mechanism, different first panic; fix was acknowledged as a band-aid.

• If the EndpointDriver or ConnectionDriver stops all API calls should result in errors #684 — driver death should surface as API errors rather than crash.

• Failed socket rebind kills noq EndpointDriver iroh#4289 — another route to a dying EndpointDriver.

• This issue was created by a human that thought critically about the issue.

[flub]

I'm not entirely sure what you are asking for. Panics should not happen, but if they do they are designed to terminate the process unless explicitly captured. This being one or two panics doesn't really change this. If anything tokio not propagating panics by default is the odd behaviour, though it is tricky what else to do. In general iroh tries to propagate errors and panics across tasks and threads.