Description
Proposed Behaviour
After running regression tests we should automatically check the geomean of various run-time & QoR metrics to verify they are within the expected valid ranges.
Current Behaviour
Currently the regression test scripts check QoR ranges on a per-benchmark basis.
Since all of these metrics have some noise which can cause spurious failures the per-benchmark ranges are set relatively loose.
Possible Solution
We should extend parse_vtr_task.pl to produce geomean metric values (in addition to the per-benchmark results generated in parse_results.txt).
It should also be extended to then check that these geomean values are within the expected ranges and cause an error if not.
Currently parse_vtr_task.pl produces a file called qor_results.txt
its not clear what this does and how it differs from the parse_results.txt
or whether it could be used/changed to support this feature. If not clear why its there it could be removed.
Context
We should perform more stringent automated QoR checking to catch potential errors/bugs/degredations sooner.
A related issue is #371.
Loosely related #545.
Activity
kmurray commentedon May 31, 2019
@vaughnbetz @renrichard FYI
vaughnbetz commentedon Jun 20, 2019
Hi Richard,
When your current tasks are done, here's another useful upgrade.
renrichard commentedon Jun 20, 2019
Sounds good, I have already included this and #535 in my to-do list as well.
github-actions commentedon Apr 30, 2025
This issue has been inactive for a year and has been marked as stale. It will be closed in 15 days if it continues to be stale. If you believe this is still an issue, please add a comment.
github-actions commentedon May 15, 2025
This issue has been marked stale for 15 days and has been automatically closed.
vaughnbetz commentedon May 15, 2025
This is a good idea. @AmirhosseinPoolad : let's discuss in our next meeting and see if you can take on this upgrade.