Agenda 2024-06-18

[jgraham]

• Safari Update
• Android Regressions Spreadsheet

[jcscottiii]

Android Regressions Spreadsheet

• @jgraham: The spreadsheet is based on code I wrote. It may be out of date at this point. Did the Firefox data. @jcscottiii did the Chrome Android. Did a spot check for some and it looked to be right. Looking at Chrome, it looks like there are just missing things. Just isn't enabled in the configuration or is missing.
• @past: Chrome Android failures looks to be less in wpt.fyi but the spreadsheet has much more
• @jgraham: It is a comparison with Desktop to Mobile. So for Firefox, things may not be in either. Whereas Chrome may have a lot of things in Desktop but not Mobile. In Firefox, I count 118 for Interop labeled tests, for Chrome it's 1559. Likely ref tests.
• @past Filtering by Interop is about 1500 which is manageable.
• @jgraham: Looking like mostly ref tests and maybe one bug. Next steps: Figure out if it is just ref tests that are failing in Chrome Android. Then look at Interop tests failing in both (Firefox Mobile and Chrome Mobile). In the second case, we can see the tests that were probably not written for mobile.
• @jgraham: Some Firefox things are resource exhaustion. Lots of timeouts. Crashes. Maybe mobile is just slow and we need to account for that. Compare against the results that run internally. Will need to check with @WeizhongX if there's some setting that may be messing up the uploads.
• @jgraham We need to update the results in the spreadsheet since it is about a month out of date.
• @past Agreed. We need to get a sense of progress. To show that the differences are going down.

Action items:

• Chrome - Figure out what is going on with reftests
• Look at Interop tests failing in both (Firefox Mobile and Chrome Mobile)
• Track the progress of the numbers over time