Skip to content

Commit cca926b

Browse files
authored
[pipeline] Skip process-group stats monitor in daemon processes (#1590)
`ProcessGroupStatsMonitor.run()` spawns a `multiprocessing` subprocess to read `/proc` for per-process-group CPU/mem/IO stats. When the pipeline is built inside a daemon process — e.g. a worker-pool process running a nested pipeline (a `.to()` region worker, or the old fused-subprocess pool worker) — `proc.start()` raises `AssertionError: daemonic processes are not allowed to have children`, which the background-task runner logs as an ERROR (once per worker, repeatedly noisy in MAST logs). A daemon genuinely cannot spawn the monitor subprocess, and a per-worker monitor would be redundant anyway: the main-process monitor already collects stats for the whole process group (it sums over the shared PGID). So guard the spawn — if the current process is a daemon, log a single WARNING and return cleanly instead of letting the assertion surface as an error.
1 parent 82f5050 commit cca926b

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

src/spdl/pipeline/_pgrp_stats.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -630,7 +630,20 @@ async def run(self) -> None:
630630
or this coroutine is cancelled. On cancellation the subprocess
631631
is terminated (then killed if it does not exit within 5 seconds)
632632
and ``CancelledError`` is re-raised.
633+
634+
A daemon process cannot spawn children, so the monitor subprocess cannot
635+
be started from one (e.g. a worker-pool process running a nested pipeline,
636+
which runs as a daemon). In that case this skips with a warning rather
637+
than raising: a main-process monitor already covers the whole process
638+
group, so a per-worker monitor here would be redundant anyway.
633639
"""
640+
if multiprocessing.current_process().daemon:
641+
_LG.warning(
642+
"Skipping process-group stats monitor: a daemon process cannot "
643+
"spawn the monitor subprocess (pid=%d).",
644+
os.getpid(),
645+
)
646+
return
634647
ctx = self._mp_context or multiprocessing.get_context()
635648
# pyrefly: ignore [missing-attribute]
636649
proc = ctx.Process(

0 commit comments

Comments
 (0)