[Abandoned] Add selective test retry for E2E/integration tests by Copilot · Pull Request #5355 · microsoft/fhir-server

Copilot · 2026-01-28T18:27:07Z

Description

Implements a selective test retry mechanism that only retries failed tests instead of rerunning the entire test suite. This replaces the retryCountOnTaskFailure approach which caused pipeline timeouts and poor test result visibility.

AB#182243

Pipelines

Example of failing pipeline: https://microsofthealthoss.visualstudio.com/FhirServer/_build/results?buildId=47394&view=results

Example of working pipeline attached to this PR.

Changes

New PowerShell test retry script (build/scripts/Invoke-TestWithRetry.ps1):

Parses TRX files to extract failed test names
Retries only failed tests using --filter expressions
Extracts method names from complex xUnit fixture test names for reliable filtering
Generates separate TRX files per attempt for Azure DevOps retry-aware publishing
Supports code coverage collection via AdditionalArgs parameter

Azure DevOps pipeline updates:

Added AllowPtrToDetectTestRunRetryFiles: true variable to enable retry-aware test result publishing
Tests now show as "Passed with Retry" instead of "Failed" when they succeed after initial failure
Updated E2E tests (e2e-tests.yml), SQL integration tests (run-sql-tests.yml), and Cosmos integration tests (run-cosmos-tests.yml) to use the new retry script
Added explicit PublishTestResults@2 tasks for proper TRX file publishing
Fixed coverage file path patterns from */coverage.cobertura.xml to **/coverage.cobertura.xml

Removed retryCountOnTaskFailure:

Removed from E2E test tasks
Removed from SQL and Cosmos integration test tasks
Removed from export test tasks (Cosmos and SQL)

Benefits

Faster retries: Only failed tests are re-run instead of the entire suite
Better visibility: Azure DevOps shows which tests passed after retry vs. which truly failed
No timeouts: Retrying a subset of tests completes much faster than full suite re-runs
Complete audit trail: All TRX files from all attempts are preserved and published

Related issues

Addresses pipeline timeout and test result visibility issues with E2E and integration tests.

Testing

PowerShell syntax validated
Argument parsing tested with quoted strings (single and double quotes)
Method name extraction tested with xUnit fixture test name patterns
Applied to E2E tests (Cosmos, SQL, all categories) and Integration tests (Cosmos, SQL)

FHIR Team Checklist

Update the title of the PR to be succinct and less than 65 characters
Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
Tag the PR with the type of update: Build, Enhancement
Tag the PR with Open source
Tag the PR with Schema Version unchanged
CI is green before merge
Review squash-merge requirements

Semver Change (docs)

Skip (build infrastructure only)

- Add Invoke-TestWithRetry.ps1 script that retries only failed tests - Update e2e-tests.yml to use retry script instead of retryCountOnTaskFailure - Remove retryCountOnTaskFailure from integration and export test tasks - Keep infrastructure task retries unchanged Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

… script - Add comprehensive README.md for test retry scripts - Fix Invoke-TestWithRetry.ps1 to properly inherit environment variables - Use & operator instead of Start-Process for better env var handling Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

- Add escaping for pipe characters in test names - Add comment about special character handling - Improve robustness of filter building Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

- Create VALIDATION.md with detailed validation steps - Include expected outcomes for success, retry, and failure scenarios - Add troubleshooting guide for common issues - Provide clear sign-off checklist Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

mikaelweave · 2026-01-28T18:40:23Z

/azp run

azure-pipelines · 2026-01-28T18:40:41Z

Azure Pipelines successfully started running 1 pipeline(s).

- Update run-cosmos-tests.yml to use Invoke-TestWithRetry.ps1 - Update run-sql-tests.yml to use Invoke-TestWithRetry.ps1 - Replace DotNetCoreCLI@2 with PowerShell@2 + PublishTestResults@2 - Maintain code coverage collection via AdditionalArgs - Update README.md with integration test usage example - Ensure environment variables are properly passed Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

- Add comprehensive validation steps for integration tests - Include code coverage validation checks - Add integration test retry examples to expected outcomes - Update checklist to cover both E2E and integration scenarios Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

- Delete README.md (moved content to script comments) - Delete VALIDATION.md (validation complete) - Enhanced Invoke-TestWithRetry.ps1 with comprehensive inline documentation: - Added WHERE IT'S USED section - Added BENEFITS OVER retryCountOnTaskFailure section - Added HOW IT WORKS section - Added CODE COVERAGE SUPPORT section - Added E2E and Integration test examples - Added notes about PublishTestResults@2 usage - All necessary documentation now in PowerShell script for maintainability Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

- Add AllowPtrToDetectTestRunRetryFiles: true to build-variables.yml - This enables Azure DevOps to detect retry attempts from multiple TRX files - Tests that fail initially but pass on retry now show as "Passed with Retry" - Update PowerShell script comments to document this feature - Reference: https://devblogs.microsoft.com/dotnet/microsoft-testing-platform-azure-retry/ Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

mikaelweave · 2026-01-28T20:49:47Z

/azp run

azure-pipelines · 2026-01-28T20:50:08Z

Azure Pipelines successfully started running 1 pipeline(s).

mikaelweave · 2026-01-30T00:41:18Z

/azp run

azure-pipelines · 2026-01-30T00:41:36Z

Azure Pipelines successfully started running 1 pipeline(s).

… refine test filtering

mikaelweave · 2026-01-30T04:40:11Z

/azp run

azure-pipelines · 2026-01-30T04:40:27Z

Azure Pipelines successfully started running 1 pipeline(s).

…or testing purposes

mikaelweave · 2026-01-30T14:10:55Z

/azp run

azure-pipelines · 2026-01-30T14:11:13Z

Azure Pipelines successfully started running 1 pipeline(s).

…ed file resolution

mikaelweave · 2026-01-30T15:23:51Z

/azp run

azure-pipelines · 2026-01-30T15:24:11Z

Azure Pipelines successfully started running 1 pipeline(s).

…rt wildcard patterns with recursive search

mikaelweave · 2026-01-30T17:56:50Z

/azp run

azure-pipelines · 2026-01-30T17:57:13Z

Azure Pipelines successfully started running 1 pipeline(s).

…olution

mikaelweave · 2026-01-30T18:36:07Z

/azp run

azure-pipelines · 2026-01-30T18:36:18Z

Azure Pipelines successfully started running 1 pipeline(s).

mikaelweave · 2026-01-30T19:02:00Z

/azp run

azure-pipelines · 2026-01-30T19:02:22Z

Azure Pipelines successfully started running 1 pipeline(s).

mikaelweave · 2026-01-30T19:04:25Z

/azp run

azure-pipelines · 2026-01-30T19:04:49Z

Azure Pipelines successfully started running 1 pipeline(s).

Co-Authored-By: Warp <agent@warp.dev>

mikaelweave · 2026-01-30T20:32:01Z

/azp run

azure-pipelines · 2026-01-30T20:32:24Z

Azure Pipelines successfully started running 1 pipeline(s).

jestradaMS · 2026-02-03T14:35:00Z

build/jobs/run-export-tests.yml


  - task: DotNetCoreCLI@2
    displayName: 'Export E2E ${{ parameters.version }} CosmosDB'
-    retryCountOnTaskFailure: 1


Was there a reason we removed retry logic here, but did not update to use the Invoke-TestWithRetry.ps1 vs. continuing to u se DotNetCoreCli?

mikaelweave · 2026-02-06T16:39:32Z

/azp run

azure-pipelines · 2026-02-06T16:39:40Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

Initial plan

dd4322b

Copilot AI assigned Copilot and mikaelweave Jan 28, 2026

Copilot started work on behalf of mikaelweave January 28, 2026 18:27 View session

Copilot AI and others added 4 commits January 28, 2026 18:32

Improve test name escaping in retry filter

0adadf2

- Add escaping for pipe characters in test names - Add comment about special character handling - Improve robustness of filter building Co-authored-by: mikaelweave <753437+mikaelweave@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Investigate adding job-level retries for E2E tests~~ Replace task-level test retry with selective failed-test retry Jan 28, 2026

Copilot AI requested a review from mikaelweave January 28, 2026 18:37

Copilot finished work on behalf of mikaelweave January 28, 2026 18:37

Copilot started work on behalf of mikaelweave January 28, 2026 20:23 View session

Copilot AI and others added 2 commits January 28, 2026 20:26

Copilot AI changed the title ~~Replace task-level test retry with selective failed-test retry~~ Add selective test retry to E2E and integration tests Jan 28, 2026

Copilot finished work on behalf of mikaelweave January 28, 2026 20:28

Copilot started work on behalf of mikaelweave January 28, 2026 20:29 View session

Copilot AI changed the title ~~Add selective test retry to E2E and integration tests~~ Consolidate test retry documentation into PowerShell script Jan 28, 2026

Copilot finished work on behalf of mikaelweave January 28, 2026 20:32

Copilot started work on behalf of mikaelweave January 28, 2026 20:39 View session

Copilot stopped work on behalf of mikaelweave due to an error January 28, 2026 20:41
The session was cancelled by the user.

Copilot started work on behalf of mikaelweave January 28, 2026 20:45 View session

Copilot AI changed the title ~~Consolidate test retry documentation into PowerShell script~~ Enable retry-aware test publishing in Azure DevOps Jan 28, 2026

Copilot finished work on behalf of mikaelweave January 28, 2026 20:48

Improve retry logic in Invoke-TestWithRetry.ps1 to handle crashes and…

52acedd

… refine test filtering

Temporarily change expected status code in FhirPathPatchTests to OK f…

703e899

…or testing purposes

Refactor glob pattern handling in Invoke-TestWithRetry.ps1 for improv…

2e41d80

…ed file resolution

Enhance assembly path resolution in Invoke-TestWithRetry.ps1 to suppo…

4ee7515

…rt wildcard patterns with recursive search

Add function to compute relative paths for improved assembly path res…

9af99f0

…olution

Add pwsh support to Invoke-TestWithRetry.ps1 calls in build YAML files

17528dc

Undo the 'failed test' changes

48a0fbe

Co-Authored-By: Warp <agent@warp.dev>

jestradaMS reviewed Feb 3, 2026

View reviewed changes

mikaelweave closed this Feb 6, 2026

mikaelweave changed the title ~~Add selective test retry for E2E/integration tests~~ [Abandoned] Add selective test retry for E2E/integration tests Feb 6, 2026

Conversation

Copilot AI commented Jan 28, 2026 • edited by mikaelweave Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pipelines

Changes

Benefits

Related issues

Testing

FHIR Team Checklist

Semver Change (docs)

Uh oh!

mikaelweave commented Jan 28, 2026

Uh oh!

azure-pipelines bot commented Jan 28, 2026

Uh oh!

mikaelweave commented Jan 28, 2026

Uh oh!

azure-pipelines bot commented Jan 28, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

mikaelweave commented Jan 30, 2026

Uh oh!

azure-pipelines bot commented Jan 30, 2026

Uh oh!

jestradaMS Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

mikaelweave commented Feb 6, 2026

Uh oh!

azure-pipelines bot commented Feb 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jan 28, 2026 •

edited by mikaelweave

Loading