From 60223febdfc7c9808e0aa851f29263ee1e8e9b49 Mon Sep 17 00:00:00 2001 From: Florian Rivoal Date: Thu, 25 Jan 2024 18:40:26 +0900 Subject: [PATCH 1/3] Skipping a reftest when preconditions are not met --- rfcs/reftest-precondition.md | 40 ++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 rfcs/reftest-precondition.md diff --git a/rfcs/reftest-precondition.md b/rfcs/reftest-precondition.md new file mode 100644 index 00000000..14692c99 --- /dev/null +++ b/rfcs/reftest-precondition.md @@ -0,0 +1,40 @@ +# RFC 178: Skipping a reftest when preconditions are not met + +## Summary + +This is a proposal to add a way to make a reftest neither fail nor pass, but instead get skipped. This is useful if some precondition of the normative statement we want to test is not satisfied. + +## Details + +Sometimes, a specification will make a normative statement about something that MUST be done if a certain precondition is fulfilled. If the precondition is not fulfilled, there's nothing to test. For testharness, there is [assert_implements_optional](http://web-platform-tests.org/writing-tests/testharness-api.html#assert_implements_optional), but for reftests, there is no equivalent. + +This is particularly annoying when the precondition is en environmental factor over which the test has no control. + +* "When printing, the UA must…", but you're running the test on screen, not printing +* "On devices with a keyboard, the UA must…", but you're running the test on a phone +* "If the UA uses overlay scrollbars, the UA must…", but you're running the test on a device with old-fashioned layout-affecting scrollbars + +Note: I'm not talking about the case where you cannot detect whether the precondition has been fulfilled or not, but rather of the case where the test can detect that it is not fulfilled. + +From a logical point of you, you could set up all these tests to pass, since the UA has violated no normative requirement. However, this would give a false sense of comfort: for example, UAs that have not even implemented the feature at all would pass the test when you run them in the right (i.e. wrong) environment. + +It'd be a lot more accurate to report that the test was skipped, rather than passed or failed. So far, there's no way to signal that. + +## Proposed solution: use a waiting reftest and add a class on the root + +If a reftest is setup with `class="reftest-wait"` on the root, the harness will wait before taking a screenshot until that class has been removed. We can take advantage of that by running tests in js during that time, and signaling through another class on the root when the precondition has failed. For instance `class="precondition-failed"` + +We could add a convenience method , similar to `takeScreenshot()` to `/common/reftest-wait.js`: + +```js +function skipTest() { + document.documentElement.classList.add("precondition-failed"); + document.documentElement.classList.remove("reftest-wait"); +} +``` + +This would be picked up by the test runner, which would then report a skipped test, rather than a failed or passed one. + +### Risks + +None identified. From e9c440018f58e30609ce2e6c204bd018ee79f091 Mon Sep 17 00:00:00 2001 From: Florian Rivoal Date: Thu, 25 Jan 2024 22:15:19 +0900 Subject: [PATCH 2/3] Adjust based on feedback from @jgraham --- rfcs/reftest-precondition.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rfcs/reftest-precondition.md b/rfcs/reftest-precondition.md index 14692c99..7f2365e3 100644 --- a/rfcs/reftest-precondition.md +++ b/rfcs/reftest-precondition.md @@ -1,8 +1,8 @@ -# RFC 178: Skipping a reftest when preconditions are not met +# RFC 178: Signaling when preconditions are not met on a reftest ## Summary -This is a proposal to add a way to make a reftest neither fail nor pass, but instead get skipped. This is useful if some precondition of the normative statement we want to test is not satisfied. +This is a proposal to add a way to make a reftest neither fail nor pass, but return `PRECONDITION_FAILED`. This is useful if some precondition of the normative statement we want to test is not satisfied. ## Details @@ -18,7 +18,7 @@ Note: I'm not talking about the case where you cannot detect whether the precond From a logical point of you, you could set up all these tests to pass, since the UA has violated no normative requirement. However, this would give a false sense of comfort: for example, UAs that have not even implemented the feature at all would pass the test when you run them in the right (i.e. wrong) environment. -It'd be a lot more accurate to report that the test was skipped, rather than passed or failed. So far, there's no way to signal that. +It'd be a lot more accurate to return `PRECONDITION_FAILED`, rather than passed or failed. So far, there's no way to signal that. ## Proposed solution: use a waiting reftest and add a class on the root @@ -33,8 +33,8 @@ function skipTest() { } ``` -This would be picked up by the test runner, which would then report a skipped test, rather than a failed or passed one. +This would be picked up by the test runner, which would then report the `PRECONDITION_FAILED` statust, rather than a failed or passed one. ### Risks -None identified. +Tests that return `PRECONDITION_FAILED` cannot be included in interop stats because it's not clear how they should be counted. From 2d4011a4f417803e0a01fa73fa36cecb9c6c6292 Mon Sep 17 00:00:00 2001 From: Florian Rivoal Date: Fri, 26 Jan 2024 13:15:33 +0900 Subject: [PATCH 3/3] Rename convenience function --- rfcs/reftest-precondition.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/reftest-precondition.md b/rfcs/reftest-precondition.md index 7f2365e3..8fcdd31f 100644 --- a/rfcs/reftest-precondition.md +++ b/rfcs/reftest-precondition.md @@ -27,7 +27,7 @@ If a reftest is setup with `class="reftest-wait"` on the root, the harness will We could add a convenience method , similar to `takeScreenshot()` to `/common/reftest-wait.js`: ```js -function skipTest() { +function preconditionFailed() { document.documentElement.classList.add("precondition-failed"); document.documentElement.classList.remove("reftest-wait"); }