Erred futures are never descoped

[crusaderky]

This test hangs on the final while loop:

import weakref

@gen_cluster(client=True)
async def test_descope_erred_future(c, s, a, b):
    f = c.submit(lambda: 1 / 0, key="f")
    ref = weakref.ref(f)
    with pytest.raises(ZeroDivisionError):
        await f
    del f
    while ref():
        await asyncio.sleep(0.01)

Additionally, the task never disappears from s.tasks.

Replacing with pytest.raises(ZeroDivisionError): await f with

try:
    await f
except ZeroDivisionError:
    pass

does not change anything.

Replacing with pytest.raises(ZeroDivisionError): await f with await wait(f) makes the issue disappear.

Workaround

Calling f.release() does not solve the client-side leak, but it does remove the future (and all of its graph) from the scheduler and workers.

@sjperkins I believe this may be closely related with what you've been seeing on the scheduler side?

[crusaderky]

This is not a recent regression (I could reproduce it on 2022.1.1 as well) and is not related to the recent changes to the @log_errors decorator.