Raising `BaseException` unexpectedly results in `WorkerLostError` in `Pool`

[connorbrinton]

Quick Description

If a function executed in a Pool raises an exception that inherits from BaseException, that exception is silently swallowed by the Pool's worker machinery and a WorkerLostError is raised in the calling process.

Minimal Reproducible Example

To reproduce:

# /// script
# dependencies = [
#   "billiard",
# ]
# ///

from billiard.pool import Pool

def main() -> None:
    print("Starting pool...")
    with Pool(4) as pool:
        print("Calling _raise_exception()")
        try:
            pool.apply(_raise_exception)
        except Exception as e:
            print(f"Caught exception: {e}")
        
        print("Calling raise_base_exception()")
        try:
            pool.apply(raise_base_exception)
        except BaseException as e:
            print(f"Caught base exception: {e}")
        
        print("Shutting down pool...")
    
    print("Pool shutdown complete.")


def _raise_exception() -> None:
    raise Exception("This is a test exception.")


def raise_base_exception() -> None:
    raise BaseException("This is a test base exception.")


if __name__ == "__main__":
    main()

Output:

Starting pool...
Calling _raise_exception()
Caught exception: This is a test exception.
Calling raise_base_exception()
Caught base exception: 
"""
Traceback (most recent call last):
  File "/Users/connor/.cache/uv/environments-v2/billiard-example-1825566addfb05de/lib/python3.11/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
    raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0 Job: 1.
"""
Shutting down pool...
Pool shutdown complete.

I expected that the second pool.apply would raise BaseException("This is a test base exception."), not a WorkerLostError.

Discussion

Part of why this was so confusing to me when I first encountered it was that I wasn't actually dealing with a plain BaseException, I was dealing with a custom exception from an old version of a library that had incorrectly subclassed BaseException when it should have subclassed Exception. That exception in turn wrapped another exception and printed its traceback under certain conditions, which made it even more confusing 😅

On the other hand, I could see it being difficult or a problem for the Pool worker to capture all BaseExceptions. The parent process in a Pool sometimes even sends SIGTERM itself for various reasons, and we wouldn't want that to unexpectedly trigger a KeyboardInterrupt exception in the code using the pool.

Anyways, I'm not sure if there's a "right" solution to this problem. Maybe KeyboardInterrupts and other known "fatal" BaseExceptions should be handled internally by the worker and other BaseExceptions should be raised to the pool caller? Either way, I figured it would be useful to raise this issue so that others can understand more about this behavior if they ever encounter it themselves 🙂

Thanks for maintaining billiard! 😄

[auvipy]

would you mind coming up with a draft PR which might solve this?