Structured concurrency, panic propagation / handling #1436
Description
Right now there are a lot of places in the driver, where we just tokio::task::spawn the test, and then let it run in the background.
This is not good:
- Panics in such tasks are silently ignored, leading to hard to debug errors. As an example, I added a panic in code that runs on connection initialization, and executed the tests. Some tests failed, but many just hanged indefinitely! If we limit the test execution time with ntest::timeout, we will see some output about the panic. This is not enough though - if session panics during setup, we should return an error. If something panics during operation, we should catch it, log it, and either respawn affected tasks or defunct a session - in general do something else than a silent failure.
- Such ignored tasks can run in the background, even when we no longer need them, or after we closed a session. Driver should properly cleanup after itself, not leave resources behind.
How to achieve that?
- Each spawned task should be owned and awaited by some code, and its panics handled.
- When a parent task is dropped, it should at least abort the child task. Tokio's JoinHandle doesn't abort on drop, but we can write a wrapper that does.
Useful functions / structs related to the above:
https://docs.rs/futures/latest/futures/future/trait.FutureExt.html#method.catch_unwind
https://docs.rs/futures/latest/futures/future/trait.FutureExt.html#method.remote_handle (remote handle aborts future on drop, we can write something similar)
https://docs.rs/futures/latest/futures/future/struct.Abortable.html#method.new
https://docs.rs/futures/latest/futures/future/fn.join_all.html
https://docs.rs/tokio/latest/tokio/task/join_set/index.html