[Performance] Performance Bottleneck due to intra_op_num_threads being set globally

[spgoswami1]

Describe the issue

Here is the thing.

1. if I keep intra_op_num_threads as 0 (default, uses all threads) then the convolution time comes down but rest other operators times increase which results in almost similar time or sometimes higher as compared to intra_op_num_threads = 1 option
2. intra_op_num_threads = 1, conv is taking 62 % of total time
3. intra_op_num_threads = 0, conv is 28 % of total time

Could not find any setting yet which enables intra_op_num_threads as 0 for conv only and keep it 1 for rest others.

Is there any setting in the code which lets me do this so that I can build the onnxruntime on my own locally?

To reproduce

I am facing this issue for my propritery model, so won't be able to share, though I can create a dummy model including Conv and Tanh and see this trend.

Urgency

No response

Platform

Windows

OS Version

10.0.26100 Build 26100

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.21.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

No

[xadupre]

It is possible to profile the model on your side to have a better understanding? (see https://onnxruntime.ai/docs/performance/tune-performance/profiling-tools.html)

[spgoswami1]

Hi @xadupre,Thanks for responding
Yes I have done the profilng and here are sample results for two operations [won't be able to attach complete json here]

setting -> session_options.intra_op_num_threads = 1
{"cat" : "Node","pid" :26108,"tid" :22496,"dur" :37,"ts" :364347,"ph" : "X","name" :"/0/ops.0/ops.0.1/encoder/resnet/activation/Tanh_kernel_time","args" : {"parameter_size" : "0","provider" : "CPUExecutionProvider","op_name" : "Tanh","input_type_shape" : [{"float":[1,416,5]}],"node_index" : "401","output_type_shape" : [{"float":[1,416,5]}],"activation_size" : "8320","output_size" : "8320","thread_scheduling_stats" : ""}}

{"cat" : "Node","pid" :26108,"tid" :22496,"dur" :780,"ts" :364399,"ph" : "X","name" :"/0/ops.0/ops.0.1/encoder/resnet/projections.1/conv/Conv_kernel_time","args" : {"parameter_size" : "2078336","provider" : "CPUExecutionProvider","op_name" : "Conv","input_type_shape" : [{"float":[1,416,5]},{"float":[416,416,3]},{"float":[416]}],"node_index" : "407","output_type_shape" : [{"float":[1,416,7]}],"activation_size" : "8320","output_size" : "11648","thread_scheduling_stats" : ""}},

setting -> session_options.intra_op_num_threads = 0
{"cat" : "Node","pid" :21980,"tid" :29156,"dur" :243,"ts" :516813,"ph" : "X","name" :"/0/ops.0/ops.0.1/encoder/resnet/activation/Tanh_kernel_time","args" : {"parameter_size" : "0","provider" : "CPUExecutionProvider","op_name" : "Tanh","input_type_shape" : [{"float":[1,416,5]}],"node_index" : "401","output_type_shape" : [{"float":[1,416,5]}],"activation_size" : "8320","output_size" : "8320","thread_scheduling_stats" : {"main_thread": {"thread_pool_name": "session-1-intra-op", "thread_id": "29156", "block_size": [], "core": 4, "Distribution": 0, "DistributionEnqueue": 0, "Run": 0, "Wait": 0, "WaitRevoke": 0}, "sub_threads": {"2664": {"num_run": 35, "core": 1},"9900": {"num_run": 24, "core": 0},"23464": {"num_run": 24, "core": 6},"14092": {"num_run": 24, "core": 9},"17116": {"num_run": 23, "core": 8},"10380": {"num_run": 23, "core": 5},"24104": {"num_run": 23, "core": 10},"20840": {"num_run": 23, "core": 11},"19380": {"num_run": 23, "core": 7}}}}},

{"cat" : "Node","pid" :21980,"tid" :29156,"dur" :389,"ts" :517109,"ph" : "X","name" :"/0/ops.0/ops.0.1/encoder/resnet/projections.1/conv/Conv_kernel_time","args" : {"parameter_size" : "2078336","provider" : "CPUExecutionProvider","op_name" : "Conv","input_type_shape" : [{"float":[1,416,5]},{"float":[416,416,3]},{"float":[416]}],"node_index" : "407","output_type_shape" : [{"float":[1,416,7]}],"activation_size" : "8320","output_size" : "11648","thread_scheduling_stats" : {"main_thread": {"thread_pool_name": "session-1-intra-op", "thread_id": "29156", "block_size": [1], "core": 3, "Distribution": 1, "DistributionEnqueue": 0, "Run": 248, "Wait": 99, "WaitRevoke": 0}, "sub_threads": {"2664": {"num_run": 36, "core": 1},"9900": {"num_run": 25, "core": 0},"23464": {"num_run": 25, "core": 6},"14092": {"num_run": 25, "core": 9},"17116": {"num_run": 24, "core": 8},"10380": {"num_run": 24, "core": 5},"24104": {"num_run": 24, "core": 10},"20840": {"num_run": 24, "core": 11},"19380": {"num_run": 24, "core": 7}}}}},

As you can see if we use all threads, then convolution time reduces from 780 to 389 but tanh time increase from 37 to 243. The similar trend is there for other convolutions and other ops like soft plus (increases from 27 to 165).

That certainly means, there is an extra overhead for launching managing threads for simple operations, which don't really require the workload to be split to multiple threads.

So, is there any option through which I can give intra_op_num_threads as 0 to certain Operators and keep it as 1 for rest other. Currently as I see, it is a global option, which does not make much sense in our case. Please suggest.

[xadupre]

Given the size of the inputs, the multithreaded version should be faster unless the other cores are busy. But maybe the parallelization settings maybe be wrong in that particular case. + @liqunfu The code is being run by an implementation similar to blas so I would expect this sequence to be as fast or faster than numpy. Without disclosing your model, it is possible to know more about the nodes around position 401 and 407.

[tianleiwu]

It is possible to have code that not uses multi-threading for short inputs, see example:
7fc7d5e

@spgoswami1, You may try modify operator code and benchmark the performance. Feel free to contribute if it helps.