[Test Utils] Limit tensor dimensions in Op Conformance to prevent OOM(Out of memory) - #34183
[Test Utils] Limit tensor dimensions in Op Conformance to prevent OOM(Out of memory)#34183Udayscode wants to merge 10 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the op conformance runner’s static-shape generation to cap per-dimension sizes based on tensor rank, reducing the likelihood of OOM during synthetic input generation for dynamic-shape models.
Changes:
- Made
dimMaxrank-dependent (127 for rank 1–2, 32 for rank 3–4, 20 for rank 5+). - Applied the adaptive cap consistently across generic shape generation, restricted-dim clipping, and pooling shape generation.
|
Proposed solution doesn't relate to Template Plugin code but Op Conformance test runner. |
|
@t-jankowski My apologies for the confusion in the PR title. I initially tagged it as [Template Plugin] because the OOM was specifically triggered during Template conformance runs, but the root cause is indeed in the test runner's shape generation. |
That's ok - no need to apologize. |
|
build_jenkins |
|
This PR will be closed in a week because of 2 weeks of no activity. |
|
build_jenkins |
|
build_jenkins |
|
build_jenkins |
|
build_jenkins |
|
build_jenkins |
|
This PR will be closed in a week because of 2 weeks of no activity. |
Fixes #23556
This PR limits the maximum dimension of generated static shapes based on tensor rank to prevent out-of-memory (OOM) crashes.
The previous implementation used a fixed limit of 127 for all dimensions. For high-rank tensors (5D+), this caused memory requests to exceed 270GB, leading to
std::bad_allocerrors. I have updated the logic to use adaptive limits: 127 for Rank 1-2, 32 for Rank 3-4, and 20 for Rank 5+.Local testing on a problematic 5D
Addoperation showed a reduction in memory usage from crash levels to 53MB. All 5,068 conformance tests now finish successfully.Changes
dimMaxcalculation ingenerate_static_shapes.cppto be rank-dependent.