[Serve] limit num_workers in replica's ThreadPoolExecutor to num_cpus by myandpr · Pull Request #60271 · ray-project/ray

myandpr · 2026-01-18T16:17:36Z

Description

Limit the user-code event loop’s default ThreadPoolExecutor size to the deployment’s ray_actor_options["num_cpus"] (fractional values round up, <=0 leaves
defaults).
This ensures asyncio.to_thread in Serve replicas respects the CPU reservation and avoids oversubscription.
Added a Serve test that verifies the default executor’s max_workers matches num_cpus.

Related issues

Link related issues: "Fixes #59750 ", "Closes #59750 ", or "Related to #59750 ".

Additional information

Tests run:
- python -m pytest python/ray/serve/tests/unit/test_user_callable_wrapper.py
- python -m pytest python/ray/serve/tests/test_replica_sync_methods.py

gemini-code-assist

Code Review

This pull request introduces a useful optimization to limit the number of workers in the replica's ThreadPoolExecutor based on the num_cpus specified in ray_actor_options. This helps prevent CPU oversubscription when using asyncio.to_thread. The implementation is clean and the logic for calculating the number of workers is robust. I've added one suggestion to enhance the test coverage by including more edge cases, which will help ensure the feature is solid.

python/ray/serve/tests/test_replica_sync_methods.py

harshit-anyscale

lgtm
cc: @abrarsheikh for further review

python/ray/serve/_private/replica.py

abrarsheikh · 2026-01-22T05:34:31Z

doc/source/serve/doc_code/asyncio_best_practices.py

 # __ray_parallel_end__


 if __name__ == "__main__":


include your serve app here so that it runs as part of doc test

Ah, good catch — I hadn’t noticed the doc test flow, Thanks. Added the CustomThreadPool example to main so it runs there now. Please take another look.

abrarsheikh · 2026-01-22T20:31:55Z

CI failing, lease merge master into your branch

Signed-off-by: yaommen <myanstu@163.com>

myandpr · 2026-01-23T02:50:26Z

update: rebased on latest master

Signed-off-by: yaommen <myanstu@163.com>

myandpr · 2026-01-23T06:28:03Z

CI failure(https://buildkite.com/ray-project/premerge/builds/58429/steps/canvas?jid=019be912-9a16-4ce9-8684-eb64454ce290) reason: test_asyncio_default_executor_limited_by_num_cpus had too many parameters and timed out; teardown hit pytest-timeout SIGTERM.

To address this, I reduced the parametrized cases from 5 to 3 (0, 2.2, 30) to keep core coverage while lowering runtime.
Evidence log: -- Test timed out at 2026-01-23 04:50:57 UTC --

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

python/ray/serve/_private/replica.py

Signed-off-by: yaommen <myanstu@163.com>

myandpr · 2026-01-23T08:50:54Z

@abrarsheikh , I have

CI failing, lease merge master into your branch

Hi @abrarsheikh ,rebased on master and CI is green. When you have a moment, could you give it another look?

The test_replica_sync_methods_with_run_sync_in_threadpool test was configured with size="small" (60s timeout), but after PR ray-project#60271 added the test_asyncio_default_executor_limited_by_num_cpus test with a num_cpus=30 parameter, the test suite started timing out during teardown. Change the test size from "small" to "medium" (300s timeout) to give the test suite enough time to complete including cleanup. Fixes flaky test: test_asyncio_default_executor_limited_by_num_cpus[30-32] Signed-off-by: Seiji Eicher <seiji@anyscale.com>

…ray-project#60271) ## Description - Limit the user-code event loop’s default ThreadPoolExecutor size to the deployment’s ray_actor_options["num_cpus"] (fractional values round up, <=0 leaves defaults). - This ensures asyncio.to_thread in Serve replicas respects the CPU reservation and avoids oversubscription. - Added a Serve test that verifies the default executor’s max_workers matches num_cpus. ## Related issues > Link related issues: "Fixes ray-project#59750 ", "Closes ray-project#59750 ", or "Related to ray-project#59750 ". ## Additional information - Tests run: - python -m pytest python/ray/serve/tests/unit/test_user_callable_wrapper.py - python -m pytest python/ray/serve/tests/test_replica_sync_methods.py --------- Signed-off-by: yaommen <myanstu@163.com> Signed-off-by: jinbum-kim <jinbum9958@gmail.com>

…ray-project#60271) ## Description - Limit the user-code event loop’s default ThreadPoolExecutor size to the deployment’s ray_actor_options["num_cpus"] (fractional values round up, <=0 leaves defaults). - This ensures asyncio.to_thread in Serve replicas respects the CPU reservation and avoids oversubscription. - Added a Serve test that verifies the default executor’s max_workers matches num_cpus. ## Related issues > Link related issues: "Fixes ray-project#59750 ", "Closes ray-project#59750 ", or "Related to ray-project#59750 ". ## Additional information - Tests run: - python -m pytest python/ray/serve/tests/unit/test_user_callable_wrapper.py - python -m pytest python/ray/serve/tests/test_replica_sync_methods.py --------- Signed-off-by: yaommen <myanstu@163.com>

…ray-project#60271) ## Description - Limit the user-code event loop’s default ThreadPoolExecutor size to the deployment’s ray_actor_options["num_cpus"] (fractional values round up, <=0 leaves defaults). - This ensures asyncio.to_thread in Serve replicas respects the CPU reservation and avoids oversubscription. - Added a Serve test that verifies the default executor’s max_workers matches num_cpus. ## Related issues > Link related issues: "Fixes ray-project#59750 ", "Closes ray-project#59750 ", or "Related to ray-project#59750 ". ## Additional information - Tests run: - python -m pytest python/ray/serve/tests/unit/test_user_callable_wrapper.py - python -m pytest python/ray/serve/tests/test_replica_sync_methods.py --------- Signed-off-by: yaommen <myanstu@163.com> Signed-off-by: 400Ping <jiekaichang@apache.org>

myandpr requested a review from a team as a code owner January 18, 2026 16:17

gemini-code-assist bot reviewed Jan 18, 2026

View reviewed changes

python/ray/serve/tests/test_replica_sync_methods.py Outdated Show resolved Hide resolved

ray-gardener bot added serve Ray Serve Related Issue community-contribution Contributed by the community labels Jan 18, 2026

harshit-anyscale approved these changes Jan 20, 2026

View reviewed changes

abrarsheikh reviewed Jan 20, 2026

View reviewed changes

python/ray/serve/_private/replica.py Outdated Show resolved Hide resolved

myandpr requested a review from abrarsheikh January 21, 2026 19:05

abrarsheikh reviewed Jan 21, 2026

View reviewed changes

python/ray/serve/_private/replica.py Show resolved Hide resolved

myandpr requested a review from abrarsheikh January 21, 2026 19:29

abrarsheikh reviewed Jan 22, 2026

View reviewed changes

myandpr requested a review from abrarsheikh January 22, 2026 19:53

abrarsheikh added the go add ONLY when ready to merge, run all tests label Jan 22, 2026

myandpr added 5 commits January 23, 2026 10:47

[Serve] limit num_workers in replica's ThreadPoolExecutor to num_cpus

42d064c

Signed-off-by: yaommen <myanstu@163.com>

[Serve] limit num_workers in replica's ThreadPoolExecutor to num_cpus

f01d395

Signed-off-by: yaommen <myanstu@163.com>

fix review

418bde3

Signed-off-by: yaommen <myanstu@163.com>

add doc

e24321d

Signed-off-by: yaommen <myanstu@163.com>

add doc

ebafa28

Signed-off-by: yaommen <myanstu@163.com>

myandpr force-pushed the limit-num-workers branch from 00f56de to ebafa28 Compare January 23, 2026 02:48

This comment was marked as outdated.

Sign in to view

fix review

7c869a7

Signed-off-by: yaommen <myanstu@163.com>

myandpr self-assigned this Jan 23, 2026

fix ci flaky

33e79f8

Signed-off-by: yaommen <myanstu@163.com>

cursor bot reviewed Jan 23, 2026

View reviewed changes

python/ray/serve/_private/replica.py Outdated Show resolved Hide resolved

fix review

78ddc05

Signed-off-by: yaommen <myanstu@163.com>

abrarsheikh approved these changes Jan 27, 2026

View reviewed changes

abrarsheikh merged commit 8c4b2b6 into ray-project:master Jan 27, 2026
6 checks passed

eicherseiji mentioned this pull request Jan 28, 2026

[Serve] Fix test_replica_sync_methods timeout #60571

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Serve] limit num_workers in replica's ThreadPoolExecutor to num_cpus#60271

[Serve] limit num_workers in replica's ThreadPoolExecutor to num_cpus#60271
abrarsheikh merged 8 commits intoray-project:masterfrom
myandpr:limit-num-workers

myandpr commented Jan 18, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

harshit-anyscale left a comment

Uh oh!

Uh oh!

Uh oh!

abrarsheikh Jan 22, 2026

Uh oh!

myandpr Jan 22, 2026

Uh oh!

abrarsheikh commented Jan 22, 2026

Uh oh!

myandpr commented Jan 23, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

myandpr commented Jan 23, 2026

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

myandpr commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

myandpr commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Additional information

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

harshit-anyscale left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

abrarsheikh Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

myandpr Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

abrarsheikh commented Jan 22, 2026

Uh oh!

myandpr commented Jan 23, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

myandpr commented Jan 23, 2026

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

myandpr commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

myandpr commented Jan 18, 2026 •

edited

Loading