Skip to content

Commit b127147

Browse files
committed
fix: create the RequestExecutingAgent ProcessPool in initialize
ProcessPool was built lazily on the first processPool() call, which happens inside execute(), after getBulkRequests() has already loaded a whole BulkRequest into memory. All MinProcess workers are forked at that instant, and because a worker never unwinds the frame it was forked from, every one of them pins that cycle's Requests for the lifetime of the agent. A core dump of one worker from a 60-process pool running BulkRequest=3000 still held the parent's requestsToExecute list and getRequests dict: 3003 Request, 3764 Operation and 40973 File objects, 54.8 MiB of live data in a process that only ever handles one request at a time. Creating the pool at the end of initialize() forks from the main thread before any request is fetched, and keeps the RequestDB engine out of the workers as well. Claude-Session: https://claude.ai/code/session_01Wj93RvpYkvmL2AGmUHneRN
1 parent aaa32ed commit b127147

1 file changed

Lines changed: 15 additions & 1 deletion

File tree

src/DIRAC/RequestManagementSystem/Agent/RequestExecutingAgent.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,12 @@ def __init__(self, *args, **kwargs):
115115
self.__rmsMonitoring = False
116116

117117
def processPool(self):
118-
"""facade for ProcessPool"""
118+
"""facade for ProcessPool
119+
120+
The pool is created by :meth:`initialize`. Creating it forks MaxProcess
121+
workers, which inherit everything reachable at that point, so the first
122+
call must stay outside of the execution cycle.
123+
"""
119124
if not self.__processPool:
120125
minProcess = max(1, self.__minProcess)
121126
maxProcess = max(self.__minProcess, self.__maxProcess)
@@ -273,6 +278,15 @@ def initialize(self):
273278
# # create request dict
274279
self.__requestCache = dict()
275280

281+
# # Spawn the ProcessPool here, while the agent is still small.
282+
# # ProcessPool forks its workers, so whatever is reachable at that moment is
283+
# # inherited by every one of them and, because the workers never unwind the
284+
# # frame they were forked from, is pinned for the lifetime of the agent.
285+
# # Creating the pool lazily from execute() meant forking in the middle of a
286+
# # cycle, freezing a full BulkRequest worth of Requests into all MaxProcess
287+
# # workers. Doing it here also keeps the RequestDB connection out of them.
288+
self.processPool()
289+
276290
self.__requestDB = RequestDB()
277291

278292
return S_OK()

0 commit comments

Comments
 (0)