Skip to content

Surprising results when scoring interop when a labelled test no longer exists #149

Open
@gsnedders

Description

@gsnedders

From web-platform-tests/interop#281, we hit a case where a number of scores moved away from 100%, which people weren't expecting.

This is because of:

// We always normalize against the number of tests we are looking for,
// rather than the total number of tests we found. The trade-off is all
// about new tests being added to the set.

and web-platform-tests/wpt#38413 splitting this test up into variants, thus making the old test URL no longer a test URL. It is, however, still labelled.

We should, at the very least, track what tests in allTestsSet we've seen, and log those that we didn't find a result for. This would put something in the log and make it much easier to understand what's happening. Though perhaps with older runs and labelling changes this would be too noisy?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions