Safari Tech Preview test results are different on local machine

[beaufortfrancois]

I'm seeing some timeouts on Picture-in-Picture idl harness tests that I'm not seeing locally with Safari Tech Preview Release 67 (on macOS 10.13.6).

Note that it is the same with Safari stable locally.

Local machine

web-platform-tests dashboard

[foolip]

@jugglinmike do you have logs available for these runs, to see if they provide any clues?

[beaufortfrancois]

Note that I actually get timeouts when running from command line (see below)
Same for Safari Stable and Tech Preview.

./wpt run safari picture-in-picture/idlharness.window.html

Running 1 tests in web-platform-tests



  ▶ TIMEOUT [expected OK] /picture-in-picture/idlharness.window.html



  ▶ Unexpected subtest result in /picture-in-picture/idlharness.window.html:

  │ TIMEOUT [expected PASS] idl_test setup

  └   → Test timed out

[beaufortfrancois]

It turns out that if I attach <video> to document.body, test doesn't time out anymore.
Note that test times out because loadedmetadata event is NOT fired when <video> is not attached to document.body.
It seems to be specific to running from command line (using Automation mode).

 function loadVideo(activeDocument, sourceUrl) {
   return new Promise((resolve, reject) => {
     const document = activeDocument || window.document;
     const video = document.createElement('video');
+    document.body.appendChild('video'); // Needed only for Safari Automation mode.
     video.src = sourceUrl || getVideoURI('/media/movie_5');
     video.onloadedmetadata = () => { resolve(video); };
     video.onerror = error => { reject(error); };
   });
 }

https://github.com/web-platform-tests/wpt/blob/2ac8a23714d941c1a544f2b897a3c77f65425ae9/picture-in-picture/resources/picture-in-picture-helpers.js

[gsnedders]

@burg any idea what's up here with Safari behaving differently under automation?

[burg]

I'm not quite sure what "Automation mode" means. Does this run via WebDriver, or not?

[beaufortfrancois]

Yes @burg

[burg]

What do I run to test this locally from the wpt repository with a custom driver path?

[burg]

Two avenues come to mind:

• Is the window visible on the screen? If not, this might cause trouble. A video recording from a failing run would help triage this aspect.
• There may be a misconfiguration of the Automation window. safaridriver may need to set WKWebViewConfiguration._mediaDataLoadsAutomatically = YES to behave the same as Safari, but I can't check this easily.

[gsnedders]

@burg

./wpt run safari --webdriver-binary /Applications/Safari\ Technology\ Preview.app/Contents/MacOS/safaridriver picture-in-picture/idlharness.window.html

[foolip]

--channel=preview should also work after web-platform-tests/wpt#13460 too.

[burg]

Great, I'm tracking this internally as rdar://problem/45384220. I'll update this issue when I figure it out.

[beaufortfrancois]

@burg, here's a 28s screen recording: https://photos.app.goo.gl/qsxSgHijiWEfVW649

[foolip]

That's cool :)

Note that if you ever want to run a single test and don't want the window to stay open you can use --no-pause.

[beaufortfrancois]

FWIW @jernoble and I noticed that HTMLMediaElement logs were not present in the Console.app to help debugging this.

Moreover, on @padenot's machine, test was actually flaky, not always a timeout.

Update: I was able to get flakyness as well.

[beaufortfrancois]

@burg Let me know if I can help.