Skip to content

Default to spawn multiprocessing context across platforms #291

Description

@FernandoCelmer

Labels: enhancement

Context

dotflow/core/workflow.py:20-31:

if sys.platform == "win32":
    _mp = get_context("spawn")
else:
    if sys.platform == "darwin":
        os.environ.setdefault("OBJC_DISABLE_INITIALIZE_FORK_SAFETY", "YES")
    _mp = get_context("fork")

Problems:

  • macOS fork with the Objective-C runtime is a known deadlock
    source. OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES only suppresses the warning; native libraries (numpy, cryptography, grpcio) can still crash silently.
  • Python 3.14 changes the macOS default to spawn for this reason.
    The library will inherit that behavior whether we like it or not; shipping ahead of the change avoids surprise breakage.
  • fork inherits all open file descriptors from the parent, which
    causes subtle bugs with database connection pools (the same socket is shared between processes).

Concept

Default mp_context to spawn on every platform. Allow override via env var or Config.

Proposed change

import os

DEFAULT_MP_CONTEXT = os.getenv("DOTFLOW_MP_CONTEXT", "spawn")
_mp = get_context(DEFAULT_MP_CONTEXT)

Optionally exposed on Config:

config = Config(mp_context="fork")  # opt back in if you accept the risk

Migration notes

spawn re-imports the worker module on each process start. Workflows that rely on importable steps will work; workflows that pass non-picklable closures will fail. Document the requirement.

Acceptance criteria

  • Default context is spawn on darwin and linux
  • DOTFLOW_MP_CONTEXT env var accepted (spawn | fork |
    forkserver)
  • CHANGELOG entry highlights the behavioral change
  • Documentation includes a "step picklability" note

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions