Isolate blast radius for panics during connection processing

[Mark-Simulacrum]

Security issue notifications

If you discover a potential security issue in s2n-quic we ask that you notify
AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

Today, if a panic occurs while processing s2n-quic connections -- perhaps in user-provided code like a Subscriber -- this will kill at least the entire s2n-quic endpoint.

Solution:

s2n-quic should put in place isolation from unwinding at the connection level, such that a failing connection is torn down (cleanly or uncleanly) but does not affect unrelated connection processing. The goal is not to be perfect, but to have a defense-in-depth mechanism that best-effort isolates panics from bringing down the full endpoint. We could consider a secondary isolation layer wrapping a full event loop turn too.

• Does this change what s2n-quic sends over the wire? probably no
• Does this change any public APIs? not directly -- people shouldn't rely on this

Requirements / Acceptance Criteria:

• RFC links: none
• Related Issues: none
• Will the Usage Guide or other documentation need to be updated? Documenting the exit conditions for torn-down connections is probably useful. Ideally we attempt to send a connection close packet or a stateless reset or some other indication to the client that there was a shutdown.
• Testing:
  • Subscriber panics for connection events should lead to a single connection failing, other connections should continue uninterrupted and new connections should be able to start. This is probably the easiest way to get some confidence this works.

Out of scope:

As mentioned above, we won't cover 100% of code and the goal of this is to have something relatively simple that reduces the blast radius for most problems.