[core] Same actor/task has different number of threads in different environments

[kevin85421]

Description

/// The pool size for grpc server call.
RAY_CONFIG(int64_t,
           num_server_call_thread,
           std::max((int64_t)1, (int64_t)(std::thread::hardware_concurrency() / 4U)))

In core worker process, you can see multiple worker.io threads in the same process. It is equal to 1 + (num_cpus / 4). Is there any special reason for it?

If there is no special reason, it doesn't make sense to me to create (num_cpus / 4) threads for each core worker process. 'num_cpus' is a concept for the whole machine, but a machine can have an arbitrary number of core worker processes.

Ideally, we should benchmark the performance and find a reasonable max value of num_server_call_thread.

• Check the number of threads "worker.io" in a core worker process.

Use case

No response

[kevin85421]

@Yicheng-Lu-llll, are you interested in investigating this issue? You can check with @edoakes and @jjyao first to see if there's any context I missed.

[Yicheng-Lu-llll]

@kevin85421 Sure — I’ll review it first and reach out if I need more context.