Skip to content

Safari WebDriver results dramatically incomplete / wdspec not restarting the browser #61616

Description

@gsnedders

Previously discussed on Matrix, with a variety of very vibecoded fixes put up in #61122.

See https://wpt.fyi/results/webdriver/tests/classic?run_id=5322123978735616&run_id=5776929038008320 (747f5ee...1903b4e).

The tl;dr is com.apple.WebDriver.HTTPService crashes during the tests added in b0dcd3d which causes safaridriver to then exit — however, we never recover from this.

The PytestExecutor never checks if the browser has crashed, merely setting the status to CRASH if OSError or socket.timeout is raised, which is extremely unlikely given pytest should have caught any exceptions itself, and because we keep getting OK harness statuses merely with every subtest returning ERROR: setup error: ConnectionRefusedError: [Errno 61] Connection refused (because, y'know, safaridriver isn't running) we don't fall into the TestRunnerManager restart for CRASH, EXTERNAL-TIMEOUT, INTERNAL-ERROR.

Digging into this shows quite a lot of things that could be improved:

  • Merging TimedRunner and PytestRun (there's a lot of duplicate logic there, though it wouldn't directly fix this!).
  • Make PytestExecutor check whether the browser is still alive regardless of result (because unlike every other executor, it can still get an OK harness result regardless of whether the browser is actually running).
  • Make TestRunnerManager check whether the browser is running itself, along with checking the various harness statuses (this will have a per cost, adding a bunch of redundant checks, but guarantees that an executor bug can't cause this cascading failure).
  • Make TestRunnerManager restart when any subtest has ERROR as a result, not the harness-level INTERNAL-ERROR (but then maybe we also should for harness-level ERROR too, which seems unnecessary).
  • Provide the Browser object to the Executor via a multiprocessing.Manager, avoiding the current ExecutorBrowser introduced way back in 9a9d62f, and would allow us to provide the ability to restart the browser from within the pytest fixtures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    infrawptrunnerThe automated test runner, commonly called through ./wpt run

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions