Commit 0a85808
authored
fix: build SAQ workers inside child processes (Python 3.14 forkserver) (#106)
## Summary
- Construct `Worker` (and its broker client) **inside** the spawned child process via a new top-level `run_worker_in_child` entry point, instead of pickling a live `redis.asyncio.Redis` across the `multiprocessing` boundary.
- Pass a pickle-safe deep-copied `SAQConfig` (live `broker_instance` and `queue_instances` nulled out) into children; broker clients are rebuilt lazily from `dsn` in the child via the existing `QueueConfig.get_broker()` machinery.
- Remove the macOS `multiprocessing.set_start_method("fork", force=True)` overrides in `cli.py` and `plugin.py` — they masked the same latent bug.
Closes #104.
## Why
Python 3.14 changed the default `multiprocessing` start method on Linux from `fork` to `forkserver`. `forkserver` pickles the target and its args. The previous code shipped a fully built `Worker` (containing a live `redis.asyncio.Redis` with lambda-based response callbacks) to each child, which fails with `_pickle.PicklingError`. On 3.13 and earlier this worked only because `fork` skipped pickling. The macOS `fork` override was hiding the same issue on Darwin.
## How the fix works
1. `prepare_config_for_spawn(config)` deep-copies the `SAQConfig` and nulls out `broker_instance` / `_broker_type` / `_queue_class` on each `QueueConfig`, plus `queue_instances` on the config itself. The copy holds only `dsn`s and plain config data — pickle-safe.
2. `run_worker_in_child(queue_name, config, logging_config)` is a top-level (picklable) function used as the `multiprocessing.Process` target. Inside the child it instantiates a fresh `SAQPlugin(config)`, fetches its `Worker` by name (which lazily rebuilds the broker client via `get_broker()`), and delegates to existing `run_saq_worker`.
3. Both spawn sites (CLI `workers run` in `cli.py` and `SAQPlugin.server_lifespan` in `plugin.py`) call `prepare_config_for_spawn` once and hand the prepared config to every child.
If a user passes `QueueConfig(broker_instance=..., dsn=None)`, `prepare_config_for_spawn` raises `ImproperConfigurationError` at spawn time with a clear message — the broker can't be rebuilt in the child without a `dsn`.
## Follow-up updates
- Added a pickle-safe Litestar logging snapshot for child workers so the default `LoggingConfig` / `StructLoggingConfig` path no longer ships Litestar's unpicklable default exception handler or queue listener handler through multiprocessing.
- Switched the explicit serialization check to `multiprocessing.reduction.ForkingPickler`, matching the serializer multiprocessing uses internally and avoiding direct `pickle` imports in runtime code.
- Preserved existing fork-based Linux behavior: fork child workers can still use a live `broker_instance` without `dsn`; the stricter `dsn` requirement only applies when the active start method needs pickle-safe args (`spawn` / `forkserver`).
- Promoted the spawn helpers to public module-level helpers (`prepare_config_for_spawn`, `prepare_logging_config_for_spawn`, `run_worker_in_child`) and kept private helpers below the public functions.
- Added regression coverage for default Litestar logging serialization, fork-vs-spawn argument requirements, parent-only broker-instance startup, and fork child-worker broker-instance startup.
- Fixed the Python 3.9 CI failure by scrubbing live parent `queue_instances` and cached broker state before deep-copying the spawn config, then restoring the parent config in a `finally` block.
## Test plan
- [x] New `tests/test_cli/test_multiprocess_spawn.py`:
- `prepare_config_for_spawn` nulls live broker references and the result is `ForkingPickler`-able
- `prepare_config_for_spawn` rejects `broker_instance` without `dsn`
- `run_worker_in_child` is picklable
- The exact `(target, args)` tuple `multiprocessing.Process` would pickle round-trips through `ForkingPickler` (the pickler `multiprocessing` uses internally) — proves pickle-safety end-to-end without spawning real worker loops
- [x] Existing test suite via `make test` passes
- [x] Lint: `uv run ruff check` clean; `uv run mypy litestar_saq/` clean
- [x] Follow-up verification: `make test`, `make lint`, `make type-check`, and `uv run ruff check --preview --select S403,S301 ...` pass
- [x] CI after `fc1d584`: validate, mypy, pyright, slotscheck, and Python 3.9-3.13 test matrix pass
## Notes
- The first worker (`managed_workers[0]`) still runs in the parent process — no spawn, no behavior change for the single-queue / single-worker path.
- Backwards compatible: no public API change to `SAQConfig`, `QueueConfig`, or `SAQPlugin`.1 parent d3a5582 commit 0a85808
5 files changed
Lines changed: 677 additions & 196 deletions
File tree
- litestar_saq
- tests/test_cli
0 commit comments