Skip to content

[rocSOLVER] Increase test timeout#5087

Open
tfalders wants to merge 1 commit intoROCm:mainfrom
tfalders:increase_rocsolver_timeout
Open

[rocSOLVER] Increase test timeout#5087
tfalders wants to merge 1 commit intoROCm:mainfrom
tfalders:increase_rocsolver_timeout

Conversation

@tfalders
Copy link
Copy Markdown

@tfalders tfalders commented May 6, 2026

Motivation

Nightly rocSOLVER tests intermittently time out on theRock CI. The variance in test time will need to be investigated, but increasing the time out value was suggested as a short-term solution.

Technical Details

Increased test time out for rocSOLVER from 60 mins to 120 mins.

Submission Checklist

Copy link
Copy Markdown
Contributor

@TorreZuk TorreZuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New tests have been added, and can reduce after variability of runs is reduced.

@TorreZuk TorreZuk requested review from ScottTodd and marbre May 6, 2026 19:06
@ScottTodd ScottTodd requested review from geomin12 and jayhawk-commits and removed request for ScottTodd and marbre May 6, 2026 19:08
Copy link
Copy Markdown
Contributor

@tony-davis tony-davis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@jayhawk-commits
Copy link
Copy Markdown
Contributor

Please add some details on how many more tests were added and how much longer the execution time needs. Eventually, this can be captured automatically with prolense. CC @amd-shiraz

@tfalders
Copy link
Copy Markdown
Author

tfalders commented May 7, 2026

Please add some details on how many more tests were added and how much longer the execution time needs. Eventually, this can be captured automatically with prolense. CC @amd-shiraz

The newest tests are for SYTRS and SYTRS2; there are 3456 new test cases. I was able to run these tests in about 47 seconds on gfx942. But because of the test variance on TheRock CI, we also saw individual test cases taking multiple seconds in the nightly tests.

# 24 mins + 20% margin = 30 mins => ~40 mins (considering gpu delays and lags)
"timeout_minutes": 60,
# Extended tests on math-ci take approx 5 hrs (as of May 5, 2026)
"timeout_minutes": 120,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's work on test standardization for rocsolver soon! https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md

having a developer wait 360 minutes for tests to run in a PR is not good

perhaps, could we do just standard tests instead of extended? and if nightly, we run extended?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: TODO

Development

Successfully merging this pull request may close these issues.

5 participants