Skip to content

[allure-playwright] flaky is never set to true for tests that retry and pass in same run, result.retry is ignored in onTestEnd #1462

@westernesque

Description

@westernesque

Describe the bug
When a Playwright test fails on the first attempt and passes on a retry within the same run, the allure-playwright reporter writes the final passing result with flaky: false. This is incorrect - a test that fails then passes in the same run is flaky by definition.

Root cause: onTestEnd in packages/allure-playwright/src/index.ts receives result: PlaywrightTestResult which includes result.retry (the attempt index - 0 for the first attempt, 1 for the first retry, etc.), but never reads it. flaky and retriesCount are therefore never written to the result file by the reporter.

Note on retriesCount specifically: I think the report generator does compensate for this by grouping result files via historyId and computing retriesCount when building data/test-cases/ in the generated report. However, flaky is never set, not by the reporter, and not by the report generator even after allure generate. The individual result file is not self-contained.

To Reproduce

// repro.spec.ts
import { test, expect } from "@playwright/test";

test("flaky test that retries and passes", async ({}, testInfo) => {
  expect(testInfo.retry, "fails first time, passes on retry").toBeGreaterThan(0);
});                                                                                                                                                                                                  
// playwright.config.ts
import { defineConfig } from "@playwright/test";
export default defineConfig({
  retries: 1,
  reporter: [["allure-playwright"]],
});

Run npx playwright test repro.spec.ts, then inspect the passing attempt's JSON in allure-results/:

{
  "status": "passed",
  "flaky": null,
  "retriesCount": null
}

Expected behavior
flaky: true, retriesCount: 1 on the passing attempt file.

Screenshots
N/A

Desktop (please complete the following information):

  • Runtime: Node.js (reporter runs server-side, no browser/OS dependency)
  • allure-playwright version: 3.7.1 (latest)
  • @playwright/test version: 1.x

Smartphone (please complete the following information):
N/A

Additional context
Confirmed fix (patched dist/cjs/index.js locally to verify):

testResult.status = statusToAllureStats(result.status, test.expectedStatus);
testResult.stage = Stage.FINISHED;
// add these:
if (result.retry > 0) {
  testResult.retriesCount = result.retry;
}
if (result.retry > 0 && testResult.status === Status.PASSED) {
  testResult.flaky = true;                                                                                                                                                                           
}

After patching, the passing attempt correctly writes flaky: true and retriesCount: 1 without requiring allure generate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions