[Feature] past tests results retention

[tzoor-eanygo]

In large test suites there are several challenges that emerge when you want to look at them over time:

1. which tests were added, changed and removed?
2. how did each test fare?
3. what is the overall state of our test suite?

seeing this information clearly in the report can help:

1. spot tests that tend to break more often than others, which indicates a fragile implementation
2. spot tests that are being changed frequently, which indicates a code hotspot that can benefit from refactoring
3. spot tests that did not change for a long time, implying that feature might be less used or was expanded without tests
4. see if our trend is positive, stagnant, or negative
5. possibly deduce other insights

bonus: graphs are pretty to look at 😃

to do this we need to:

1. save every test run results in some way that will enable us to parse it afterwards
2. aggregate test results per test, this will enable us to see a trend also for any grouped tests and the entire test suite
3. display a trend graph for the scope we are focusing on showing either results (pass, fail, etc..) over time or volume (added, changed, removed) over time

although, it is unclear how to account for trend breaking changes, for example, when a tests changes its name or its flow, how should we address this?

perhaps we can get inspired by how allure report implemented their history feature which divides the tests into several categories according to their results (passed, failed, broken, ignored, other)

[jared-christensen]

This feature would be super helpful! Right now, I’m stuck using both the Playwright HTML report and Allure (just for the history tracking). Honestly, it’s a bit of a hassle maintaining two systems when Playwright’s HTML report could handle this with the right feature.

For me, the big things I’d want are:

• Tracking flaky tests by seeing failures over time.
• Spotting trends for the whole suite (pass/fail trends, added/removed tests, etc.).
• Highlighting tests that keep changing or failing, so we know where to focus.

I noticed this issue in the Playwright Testing Service repo talks about something similar. Are features like this on the radar, or is the idea to lean on third-party tools for stuff like history?

I’d love to see it built into Playwright. It would make life a lot easier without having to pay for a dashboard or juggle multiple tools.