Skip to content

Commit 0a85808

Browse files
fix: build SAQ workers inside child processes (Python 3.14 forkserver) (#106)
## Summary - Construct `Worker` (and its broker client) **inside** the spawned child process via a new top-level `run_worker_in_child` entry point, instead of pickling a live `redis.asyncio.Redis` across the `multiprocessing` boundary. - Pass a pickle-safe deep-copied `SAQConfig` (live `broker_instance` and `queue_instances` nulled out) into children; broker clients are rebuilt lazily from `dsn` in the child via the existing `QueueConfig.get_broker()` machinery. - Remove the macOS `multiprocessing.set_start_method("fork", force=True)` overrides in `cli.py` and `plugin.py` — they masked the same latent bug. Closes #104. ## Why Python 3.14 changed the default `multiprocessing` start method on Linux from `fork` to `forkserver`. `forkserver` pickles the target and its args. The previous code shipped a fully built `Worker` (containing a live `redis.asyncio.Redis` with lambda-based response callbacks) to each child, which fails with `_pickle.PicklingError`. On 3.13 and earlier this worked only because `fork` skipped pickling. The macOS `fork` override was hiding the same issue on Darwin. ## How the fix works 1. `prepare_config_for_spawn(config)` deep-copies the `SAQConfig` and nulls out `broker_instance` / `_broker_type` / `_queue_class` on each `QueueConfig`, plus `queue_instances` on the config itself. The copy holds only `dsn`s and plain config data — pickle-safe. 2. `run_worker_in_child(queue_name, config, logging_config)` is a top-level (picklable) function used as the `multiprocessing.Process` target. Inside the child it instantiates a fresh `SAQPlugin(config)`, fetches its `Worker` by name (which lazily rebuilds the broker client via `get_broker()`), and delegates to existing `run_saq_worker`. 3. Both spawn sites (CLI `workers run` in `cli.py` and `SAQPlugin.server_lifespan` in `plugin.py`) call `prepare_config_for_spawn` once and hand the prepared config to every child. If a user passes `QueueConfig(broker_instance=..., dsn=None)`, `prepare_config_for_spawn` raises `ImproperConfigurationError` at spawn time with a clear message — the broker can't be rebuilt in the child without a `dsn`. ## Follow-up updates - Added a pickle-safe Litestar logging snapshot for child workers so the default `LoggingConfig` / `StructLoggingConfig` path no longer ships Litestar's unpicklable default exception handler or queue listener handler through multiprocessing. - Switched the explicit serialization check to `multiprocessing.reduction.ForkingPickler`, matching the serializer multiprocessing uses internally and avoiding direct `pickle` imports in runtime code. - Preserved existing fork-based Linux behavior: fork child workers can still use a live `broker_instance` without `dsn`; the stricter `dsn` requirement only applies when the active start method needs pickle-safe args (`spawn` / `forkserver`). - Promoted the spawn helpers to public module-level helpers (`prepare_config_for_spawn`, `prepare_logging_config_for_spawn`, `run_worker_in_child`) and kept private helpers below the public functions. - Added regression coverage for default Litestar logging serialization, fork-vs-spawn argument requirements, parent-only broker-instance startup, and fork child-worker broker-instance startup. - Fixed the Python 3.9 CI failure by scrubbing live parent `queue_instances` and cached broker state before deep-copying the spawn config, then restoring the parent config in a `finally` block. ## Test plan - [x] New `tests/test_cli/test_multiprocess_spawn.py`: - `prepare_config_for_spawn` nulls live broker references and the result is `ForkingPickler`-able - `prepare_config_for_spawn` rejects `broker_instance` without `dsn` - `run_worker_in_child` is picklable - The exact `(target, args)` tuple `multiprocessing.Process` would pickle round-trips through `ForkingPickler` (the pickler `multiprocessing` uses internally) — proves pickle-safety end-to-end without spawning real worker loops - [x] Existing test suite via `make test` passes - [x] Lint: `uv run ruff check` clean; `uv run mypy litestar_saq/` clean - [x] Follow-up verification: `make test`, `make lint`, `make type-check`, and `uv run ruff check --preview --select S403,S301 ...` pass - [x] CI after `fc1d584`: validate, mypy, pyright, slotscheck, and Python 3.9-3.13 test matrix pass ## Notes - The first worker (`managed_workers[0]`) still runs in the parent process — no spawn, no behavior change for the single-queue / single-worker path. - Backwards compatible: no public API change to `SAQConfig`, `QueueConfig`, or `SAQPlugin`.
1 parent d3a5582 commit 0a85808

5 files changed

Lines changed: 677 additions & 196 deletions

File tree

0 commit comments

Comments
 (0)