origin-isolation tests need to be run in separate browsing context groups

[domenic]

Origin isolation has a browsing-context-group wide scope, for the reasons discussed here.

This means that if one of the origin isolation tests runs, and then the test runner navigates to a second origin isolation test, the second test will be impacted by what happens in the first test, invalidating the results.

I'm not sure how the test runner infrastructure works. Is there a guarantee that my tests will be run in separate browsing context groups? (E.g., separate top-level windows, with no opener relationships between them---including no common opener.)

[Hexcles]

separate top-level windows, with no opener relationships between them---including no common opener

I'd assume that's the case. @jgraham can you confirm?

[Hexcles]

(But for sure we don't restart the browser.)

[domenic]

It appears to not always be the case when using vpython third_party/blink/tools/run_web_tests.py in Chromium though, so perhaps this would become a crbug.

[Hexcles]

It appears to not always be the case when using vpython third_party/blink/tools/run_web_tests.py in Chromium though, so perhaps this would become a crbug.

That's highly possible. content_shell --run-web-tests manages lifecycles in a completely different and messy way...

[domenic]

I'll close this, since it seems like WPT is probably not at fault. On the blink side, I've resorted to using --restart-shell-between-tests=always when running the tests locally. On the test bots, they seem to not need that.

[stephenmcgruer]

From https://crbug.com/1097980 (and the associated PR #24259), it seems like maybe we are not putting tests in a separate browsing context, at least for stability checks? This needs to be investigated (unless @jgraham happens to know).

If its just stability checks (as opposed to between different tests), it may not be too bad.

[stephenmcgruer]

Right, so for stability checks we run something akin to the following, which reproduces the problem:

./wpt run --no-restart-on-unexpected --rerun=2 --channel=experimental chrome origin-isolation/removing-iframes.sub.https.html

The -no-restart-on-unexpected means we do the following:

1. Open https://web-platform.test:8443/testharness_runner.html in the browser
2. Call window.open('about:blank', '%s', 'noopener')" % self.window_id)
3. Load the test URL in the new window (well, tab)
4. After test, close the tab.
5. Again call window.open('about:blank', '%s', 'noopener')" % self.window_id)
6. Load the test URL in the new window (well, tab)

So the original https://web-platform.test:8443/testharness_runner.html tab becomes a common opener, I presume.

[Hexcles]

So the original https://web-platform.test:8443/testharness_runner.html tab becomes a common opener, I presume.

Noob question: doesn't noopener mean the new tab doesn't have an opener?

[stephenmcgruer]

Domenic and I chatted about this today. In general noopener 'should' work, but Domenic put forward the reasonable argument that artificially severing the opener relationship is a little bit dicey when one is trying to test things at the window.open() and noopener layer itself.

Refreshing my mind on how this works - we actually reuse the same testharness harness tab where possible between all tests I believe, not just when we are repeating tests. We only reload or change it when we need to go between http/https, or if something else in the setup changes between tests. So one could also reproduce the problem for Chromium by having two tests doing the same thing, without using --rerun=2.

One possibility would be to use webdriver to fully reload or replace the testharness harness tab between each test, but I suspect there is concern about the speed of that. Another possibility would be to introduce (yet another) file-name flag to force webdriver to fully reload before running such tests.

I think I lean towards a special flag, but I'm still not clear on the concrete cases where using noopener is truly problematic. For crbug.com/1099718 its technically the case (as far as I read it) that Chrome has a bug with noopener, not that noopener shouldn't work for the feature under test (origin isolation).

cc @jgraham @gsnedders @Hexcles, and @domenic