WebCodecs scores inconsistent between test results page and dashboard

[nt1m]

https://wpt.fyi/results/webcodecs?label=experimental&label=master&product=chrome&product=firefox&product=safari&aligned&view=interop&q=label%3Ainterop-2023-webcodecs
https://wpt.fyi/interop-2023

First link has 72% pass rate, the dashboard shows 59.6%, seems like the scores are computed differently.

[gsnedders]

Could it be that we, once again, have tests labelled that no longer exist?

[nt1m]

@DanielRyanSmith @foolip Do you know what is going on here?

[DanielRyanSmith]

This is a logic error on wpt.fyi. It looks like there are a number of tests being marked as 100% passing, but have subtests with "PRECONDITION FAILED" statuses that are not being aggregated correctly. They're displaying with green backgrounds, indicating passing, but display "0 of X subtests" text in the cell. I'll have to investigate further to determine what is actually happening here.

For now, it is safe to assume that the numbers on the dashboard are correct, and the results page needs a change.

[DanielRyanSmith]

After looking further, this is very likely caused by "PRECONDITION_FAILED" not being explicitly defined as a valid status on the server side of wpt.fyi where search results are aggregated.

The aggregator method that populates search results (e.g. when searching for interop labels) ignores subtests with statuses that it does not recognize.

Note that this does not affect the normal view of wpt.fyi because only search results are aggregated at runtime (most view data is populated using pre-aggregated summary files).

Does anyone have a concrete definition of what "PRECONDITION_FAILED" means as a status? It would be nice to add a definition of what that status means when adding it to the valid status constants.

[nt1m]

@DanielRyanSmith PRECONDITION_FAILED is returned by assert_implements_optional when the thing that is optional is not there. So this should not be counted as a fail. In this case the test results page is correct (and the Interop dashboard is wrong).

See: https://web-platform-tests.org/writing-tests/testharness-api.html#optional-features

[nt1m]

In this case of WebCodecs, it's just hitting this status since the tested video codecs are not supported, and the test does not require them to be supported.

[nt1m]

web-platform-tests/results-analysis#178