Auto-balancing Playwright test shards

[aslushnikov]

Hey folks,

I used to work on Playwright, and I noticed that you use Playwright too, including sharding in CI.

These days I’m building Flakiness.io: a GitHub-native test analytics dashboard that tracks flaky tests and detects regressions in pull requests.

We’re prototyping Playwright shard balancing, where Flakiness.io uses historical test timings to produce better-balanced shards. Since you have a real-world Playwright setup, I’d be very happy if you’re open to testing it on your workload.

OSS projects can use Flakiness.io for free. I’d be happy to help set it up and see if you find it useful.

[aslushnikov]

FYI: we’ve released shard balancing as part of the flakiness/playwright reporter.

To try it out:

1. Create a project on Flakiness.io to store your test history
2. Install flakiness/playwright
3. Replace npx playwright test with npx flakiness-playwright-shard in Github Actions:

-npx playwright test --shard 1/2 --project foo
+npx flakiness-playwright-shard --shard 1/2 --project foo

Details: https://docs.flakiness.io/shard-balancing/

Let me know if I can help with the integration - we’d love to see how it works for you.