Skip to content

Regression Tests should check average/geomean QoR metrics #630

Open
@kmurray

Description

@kmurray
Contributor

Proposed Behaviour

After running regression tests we should automatically check the geomean of various run-time & QoR metrics to verify they are within the expected valid ranges.

Current Behaviour

Currently the regression test scripts check QoR ranges on a per-benchmark basis.
Since all of these metrics have some noise which can cause spurious failures the per-benchmark ranges are set relatively loose.

Possible Solution

We should extend parse_vtr_task.pl to produce geomean metric values (in addition to the per-benchmark results generated in parse_results.txt).

It should also be extended to then check that these geomean values are within the expected ranges and cause an error if not.

Currently parse_vtr_task.pl produces a file called qor_results.txt its not clear what this does and how it differs from the parse_results.txt or whether it could be used/changed to support this feature. If not clear why its there it could be removed.

Context

We should perform more stringent automated QoR checking to catch potential errors/bugs/degredations sooner.

A related issue is #371.

Loosely related #545.

Activity

added
VTR FlowVTR Design Flow (scripts/benchmarks/architectures)
on May 31, 2019
kmurray

kmurray commented on May 31, 2019

@kmurray
ContributorAuthor
vaughnbetz

vaughnbetz commented on Jun 20, 2019

@vaughnbetz
Contributor

Hi Richard,

When your current tasks are done, here's another useful upgrade.

renrichard

renrichard commented on Jun 20, 2019

@renrichard
Contributor

Sounds good, I have already included this and #535 in my to-do list as well.

github-actions

github-actions commented on Apr 30, 2025

@github-actions

This issue has been inactive for a year and has been marked as stale. It will be closed in 15 days if it continues to be stale. If you believe this is still an issue, please add a comment.

github-actions

github-actions commented on May 15, 2025

@github-actions

This issue has been marked stale for 15 days and has been automatically closed.

vaughnbetz

vaughnbetz commented on May 15, 2025

@vaughnbetz
Contributor

This is a good idea. @AmirhosseinPoolad : let's discuss in our next meeting and see if you can take on this upgrade.

added
no-staleExclude issue from being automatically marked as stale
and removed on May 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

VTR FlowVTR Design Flow (scripts/benchmarks/architectures)no-staleExclude issue from being automatically marked as staletests

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @kmurray@vaughnbetz@renrichard@AmirhosseinPoolad

      Issue actions

        Regression Tests should check average/geomean QoR metrics · Issue #630 · verilog-to-routing/vtr-verilog-to-routing