Auto-balancing Playwright test shards

[aslushnikov]

Hey folks,

I used to work on Playwright, and I noticed that Langflow uses Playwright too, including sharding in CI.

These days I’m building Flakiness.io: a GitHub-native test analytics dashboard that tracks flaky tests and detects regressions in pull requests.

We’re prototyping Playwright shard balancing, where Flakiness.io uses historical test timings to produce better-balanced shards. Since Langflow has a real-world Playwright setup, I’d be very happy if you’re open to testing it on your workload.

OSS projects can use Flakiness.io for free. I’d be happy to help set it up and see if you find it useful.

[AntonioABLima]

Hey @aslushnikov , thanks for the message! Flakiness.io sounds like a great tool, especially for a real-world Playwright setup like ours.

I've pinged the folks on our team who handle our E2E tests (@daniellicnerski1 and @Victor-w-Madeira) to check this out. We'd love to see how it can help with our shard balancing. Thanks again for offering this to the OSS community!

[AntonioABLima]

Super beautiful website, btw!

[daniellicnerski1]

Hey, @aslushnikov

Thanks for reaching out!

I'm Daniel, QA engineer at OrionTech, I currently own the E2E regression test suite for Langflow. The suite actually lives in a separate repo from Langflow itself: it's an independent Playwright + TypeScript project that runs against any Langflow
instance via URL, covering core flows, components, MCP, auth, playground, and so on, with nightly and weekly CI schedules.

Happy to try Flakiness.io on it. It's a real-world Playwright workload with LLM-backed flows (so we do see real flakiness), and we've been collecting historical run data in JSONL logs already, should be a useful target. Let me know what setup steps
you'd like me to take and I'll get it wired up.

[aslushnikov]

Hey @daniellicnerski1!

The suite actually lives in a separate repo from Langflow itself: it's an independent Playwright + TypeScript project that runs against any Langflow instance via URL

With this poly-repo setup, Flakiness.io test analytics & regression analysis is pretty useless today: we attribute test results with the source code revision, thinking that commit id fully defines both the tests and the system-under-test. Maybe we can come up with some workarounds, depending on how exactly you set it up. Is the testing repository private? I can't find it, somehow.

On the plus side, shard auto-balancing should work just fine: it only uses historical timing data. For it to work, you just need to create an org + a project on Flakiness.io, and install the @flakiness/playwright reporter to upload the testing data. In a few weeks, once we release the experimental sharding, I'll come back to send a PR and enable the experiment for you. By this time, the project should already accumulate timing data, so we should be able to see how well it works.

[daniellicnerski1]

Hi @aslushnikov
The repository is currently private, but I'm looking into the possibility of making it public, and I'll send it your way once I do.

Either way, I'm going to test it out in the repo to use some of the features. Definitely keep me updated on new features; I found it really interesting! Also, I'll talk to the team in charge of some of Langflow's current automations and tell them about the tool.

Let's stay in touch, thanks!