Skip to content

[CI] Use the local test report generator for Outerloop workflow #9439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented May 21, 2025

This PR updates the Outerloop workflow to use the tools/GenerateTestSummary tool for generating test reports, aligning it with how test reporting is done in other workflows.

Changes made

  • Added GenerateTestSummary tool call to individual test jobs in the Outerloop workflow, providing consistent per-test reporting
  • Set up environment variables for running the dotnet script based on OS platform
  • Replaced custom PowerShell script in the final results job with the GenerateTestSummary tool
  • Added logic to organize test results by OS so that the summary generator can properly categorize them

Benefits

  • Consistent test reporting format across all workflows
  • Simplified maintenance by leveraging the same reporting tool everywhere
  • Better alignment with the existing test summary generation pattern
  • Reduced code duplication by removing custom parsing and report generation scripts

Fixes #9438.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] [CI] Use the local test report generator for Outerloop workflow [CI] Use the local test report generator for Outerloop workflow May 21, 2025
@Copilot Copilot AI requested a review from radical May 21, 2025 03:02
@radical
Copy link
Member

radical commented May 21, 2025

Can you trigger the Outerloop tests workflow to validate the changes here?

@radical
Copy link
Member

radical commented May 21, 2025

We need to validate the case where one of the test job has a test failure. For that can you do the following:

  1. Modify one of the quarantined tests to add a bug so the test fails
  2. Then trigger the outerloop workflow so the changes here can be validated.

@radical
Copy link
Member

radical commented May 21, 2025

Try that again

@radical
Copy link
Member

radical commented May 21, 2025

@copilot We need to validate the case where one of the test job has a test failure. For that can you do the following:

Modify one of the quarantined tests to add a bug so the test fails
Then trigger the outerloop workflow so the changes here can be validated.

Copy link
Contributor Author

Copilot AI commented May 21, 2025

@copilot We need to validate the case where one of the test job has a test failure. For that can you do the following:

Modify one of the quarantined tests to add a bug so the test fails
...

I've modified the YarpFunctionalTests.cs file to introduce a deliberate test failure. The test now expects a BadRequest status code instead of OK for the HTTP response, which will cause it to fail. The test is already quarantined and marked with the [QuarantinedTest] attribute, so it won't cause the main workflow to fail. This change will help validate that the test summary generator correctly identifies and reports failures in quarantined tests. The change is in commit 827cc3a.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/aspire/aspire/.dotnet/dotnet build tests/Aspire.Hosting.Yarp.Tests/Aspire.Hosting.Yarp.Tests.csproj (dns block)
  • pkgs.dev.azure.com
    • Triggering command: /home/REDACTED/work/aspire/aspire/.dotnet/dotnet build tests/Aspire.Hosting.Yarp.Tests/Aspire.Hosting.Yarp.Tests.csproj (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@radical
Copy link
Member

radical commented May 21, 2025

This is working well. You can drop the artificial test failure now, and we can take this PR out of draft.

Copy link
Contributor Author

Copilot AI commented May 21, 2025

This is working well. You can drop the artificial test failure now, and we can take this PR out of draft.

I've removed the artificial test failure from YarpFunctionalTests.cs that was added for validation purposes. The test now correctly expects an OK status code again. This change is in commit 430266a.

@radical radical marked this pull request as ready for review May 21, 2025 21:21
@radical radical added the area-engineering-systems infrastructure helix infra engineering repo stuff label May 21, 2025
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the Outerloop workflow to leverage the GenerateTestSummary tool for consistent test reporting. Key changes include adding OS-specific environment variable setup for the dotnet script, replacing a custom PowerShell processing block with a direct tool invocation, and reorganizing test result files by OS.

@radical
Copy link
Member

radical commented May 23, 2025

@joperezr @danmoseley this is ready for merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-engineering-systems infrastructure helix infra engineering repo stuff
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] Use the local test report generator for Outerloop workflow
2 participants