Describe the bug
When a Playwright test fails on the first attempt and passes on a retry within the same run, the allure-playwright reporter writes the final passing result with flaky: false. This is incorrect - a test that fails then passes in the same run is flaky by definition.
Root cause: onTestEnd in packages/allure-playwright/src/index.ts receives result: PlaywrightTestResult which includes result.retry (the attempt index - 0 for the first attempt, 1 for the first retry, etc.), but never reads it. flaky and retriesCount are therefore never written to the result file by the reporter.
Note on retriesCount specifically: I think the report generator does compensate for this by grouping result files via historyId and computing retriesCount when building data/test-cases/ in the generated report. However, flaky is never set, not by the reporter, and not by the report generator even after allure generate. The individual result file is not self-contained.
To Reproduce
// repro.spec.ts
import { test, expect } from "@playwright/test";
test("flaky test that retries and passes", async ({}, testInfo) => {
expect(testInfo.retry, "fails first time, passes on retry").toBeGreaterThan(0);
});
// playwright.config.ts
import { defineConfig } from "@playwright/test";
export default defineConfig({
retries: 1,
reporter: [["allure-playwright"]],
});
Run npx playwright test repro.spec.ts, then inspect the passing attempt's JSON in allure-results/:
{
"status": "passed",
"flaky": null,
"retriesCount": null
}
Expected behavior
flaky: true, retriesCount: 1 on the passing attempt file.
Screenshots
N/A
Desktop (please complete the following information):
- Runtime: Node.js (reporter runs server-side, no browser/OS dependency)
- allure-playwright version: 3.7.1 (latest)
- @playwright/test version: 1.x
Smartphone (please complete the following information):
N/A
Additional context
Confirmed fix (patched dist/cjs/index.js locally to verify):
testResult.status = statusToAllureStats(result.status, test.expectedStatus);
testResult.stage = Stage.FINISHED;
// add these:
if (result.retry > 0) {
testResult.retriesCount = result.retry;
}
if (result.retry > 0 && testResult.status === Status.PASSED) {
testResult.flaky = true;
}
After patching, the passing attempt correctly writes flaky: true and retriesCount: 1 without requiring allure generate.
Describe the bug
When a Playwright test fails on the first attempt and passes on a retry within the same run, the allure-playwright reporter writes the final passing result with flaky: false. This is incorrect - a test that fails then passes in the same run is flaky by definition.
Root cause: onTestEnd in packages/allure-playwright/src/index.ts receives result: PlaywrightTestResult which includes result.retry (the attempt index - 0 for the first attempt, 1 for the first retry, etc.), but never reads it. flaky and retriesCount are therefore never written to the result file by the reporter.
Note on retriesCount specifically: I think the report generator does compensate for this by grouping result files via historyId and computing retriesCount when building data/test-cases/ in the generated report. However, flaky is never set, not by the reporter, and not by the report generator even after allure generate. The individual result file is not self-contained.
To Reproduce
Run npx playwright test repro.spec.ts, then inspect the passing attempt's JSON in allure-results/:
Expected behavior
flaky: true, retriesCount: 1on the passing attempt file.Screenshots
N/A
Desktop (please complete the following information):
Smartphone (please complete the following information):
N/A
Additional context
Confirmed fix (patched dist/cjs/index.js locally to verify):
After patching, the passing attempt correctly writes
flaky: true and retriesCount: 1without requiring allure generate.