Background
Ray Core currently records both the creator job and creator actor for a placement group, but the default automatic cleanup requires both the job and actor to be dead. This is not sufficient for components whose actual resource owner is a controller actor or another actor created within a long-lived driver job. When that actor exits unexpectedly while the job remains alive, its placement group can continue reserving resources until the driver job exits.
This limitation has led individual components to implement their own cleanup mechanisms. Ray Train V2 is one example: its current implementation needs a detached PlacementGroupCleaner to track controller-owned placement groups, persist registrations, recover after cleaner restarts, and clean up after abnormal controller exits.
Immediate Train problem
Train issue #65443 is specifically about the memory overhead of having one Python PlacementGroupCleaner process per concurrent Train run. In the reported workload, each cleaner used approximately 600 MB, so 24 concurrent runs could consume approximately 14.4 GB on the head node. PR #65447 changes this from O(number of runs) to one shared cleaner process.
This issue is distinct from, but related to, placement-group resource leaks: the immediate Train workaround reduces the number of cleaner processes, while a Core-native owner/lifetime model could eventually remove the need for the cleaner entirely.
Proposed Core direction
Consider adding an explicit placement-group lifetime/ownership mode that binds a non-detached placement group to its creator actor, without changing the existing default semantics. For example, an API equivalent to lifetime="creator_actor" could cause GCS to remove the placement group when its creator actor dies, while preserving the current job-and-actor behavior for existing callers.
The design should define behavior for:
- actor restart and
max_restarts;
- detached actors;
- driver/task-created placement groups;
- job death before or after actor death;
- GCS restart and placement-group state recovery;
- races between placement-group creation, actor death, and removal.
Decision and collaboration
PR #65447 is a short-term Train workaround, but it may not be merged if the Ray community prefers to implement the ownership/lifetime capability in Core first. A Core implementation would only address the Train memory overhead after Train migrates to it and removes the Python cleaner.
We would like community feedback on whether this Core capability is desirable and how its semantics should be defined. If the community decides to pursue it, we are happy to collaborate on the design, implementation, compatibility plan, tests, and the follow-up Train migration. If Core does not pursue this direction in the near term, the shared cleaner remains a practical Train-local solution.
Acceptance criteria
- Core Worker and Python APIs can request actor-owned lifetime explicitly.
- GCS persists and restores the ownership policy.
- Actor-death cleanup is reliable and idempotent.
- Existing placement-group lifetime behavior remains unchanged by default.
- Core tests cover actor/job death ordering, actor restart, detached actors, GCS restart, and creation/removal races.
- A follow-up Train migration can remove the component-specific cleaner once the Core capability is available.
Related issues and context
Background
Ray Core currently records both the creator job and creator actor for a placement group, but the default automatic cleanup requires both the job and actor to be dead. This is not sufficient for components whose actual resource owner is a controller actor or another actor created within a long-lived driver job. When that actor exits unexpectedly while the job remains alive, its placement group can continue reserving resources until the driver job exits.
This limitation has led individual components to implement their own cleanup mechanisms. Ray Train V2 is one example: its current implementation needs a detached
PlacementGroupCleanerto track controller-owned placement groups, persist registrations, recover after cleaner restarts, and clean up after abnormal controller exits.Immediate Train problem
Train issue #65443 is specifically about the memory overhead of having one Python
PlacementGroupCleanerprocess per concurrent Train run. In the reported workload, each cleaner used approximately 600 MB, so 24 concurrent runs could consume approximately 14.4 GB on the head node. PR #65447 changes this from O(number of runs) to one shared cleaner process.This issue is distinct from, but related to, placement-group resource leaks: the immediate Train workaround reduces the number of cleaner processes, while a Core-native owner/lifetime model could eventually remove the need for the cleaner entirely.
Proposed Core direction
Consider adding an explicit placement-group lifetime/ownership mode that binds a non-detached placement group to its creator actor, without changing the existing default semantics. For example, an API equivalent to
lifetime="creator_actor"could cause GCS to remove the placement group when its creator actor dies, while preserving the current job-and-actor behavior for existing callers.The design should define behavior for:
max_restarts;Decision and collaboration
PR #65447 is a short-term Train workaround, but it may not be merged if the Ray community prefers to implement the ownership/lifetime capability in Core first. A Core implementation would only address the Train memory overhead after Train migrates to it and removes the Python cleaner.
We would like community feedback on whether this Core capability is desirable and how its semantics should be defined. If the community decides to pursue it, we are happy to collaborate on the design, implementation, compatibility plan, tests, and the follow-up Train migration. If Core does not pursue this direction in the near term, the shared cleaner remains a practical Train-local solution.
Acceptance criteria
Related issues and context
vLLMEngineStagePGs aren't released on execution completion #64035