Skip to content

ci: Add comparison and visualization workflow for agent performance testing in CI#3496

Merged
nr-ahemsath merged 9 commits intomainfrom
ci/performance-test-visualization
Mar 20, 2026
Merged

ci: Add comparison and visualization workflow for agent performance testing in CI#3496
nr-ahemsath merged 9 commits intomainfrom
ci/performance-test-visualization

Conversation

@nr-ahemsath
Copy link
Copy Markdown
Member

Description

Implements #3308, adding the ability to run up to four performance tests with different agent versions (plus a no-agent baseline) and produce a summary table and charts for easy comparison of results.

Here is an example of the workflow, comparing the latest release (10.50.0) with the latest overnight all_solutions build (which should actually be identical, but this is just a demo): https://github.com/newrelic/newrelic-dotnet-agent/actions/runs/23308138260

Details:

  1. The performance_tests.yml workflow (which ran a single performance test) was refactored into an action at .github/actions/run-perf-test/action.yml
  2. compare_performance.yml is added to run up to four test scenarios, with the option to include a no-agent baseline (defaulting to true). Each run takes a label and a string which can be either an agent version number or a GitHub actions workflow run ID, which should be the run ID of an all_solutions.yml run from which to obtain the agent bits. If the input string is blank it defaults to the latest released agent version.
  3. The charts are generated in a containerized ReportGenerator process that uses ScottPlot.
  4. The output from each individual perf test has been made less verbose by default, and more human-readable.

There will be a follow-up PR or PRs which will do (at least) the following:

  1. Make the nightly comparison compare current main with current release, instead of current release to no agent.
  2. Enable convenient local performance test comparison runs.
  3. Add a README.md that explains this whole system and how to use it.
  4. Add the ability to configure the agent for each run with env vars, to enable more interesting performance comparisons.

1. Change the agent health check to just look for "agent fully connected" so we don't have to have the agent log level set to DEBUG
2. Don't dump traffic driver and test app logs by default
3. Parameterize the python executable used to clean up the test output (part of follow-on work to make running comparisons locally convenient)
@nr-ahemsath nr-ahemsath requested a review from a team as a code owner March 19, 2026 17:59
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.77%. Comparing base (1487c66) to head (853bfd3).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3496      +/-   ##
==========================================
- Coverage   81.79%   81.77%   -0.02%     
==========================================
  Files         508      508              
  Lines       34220    34220              
  Branches     4040     4040              
==========================================
- Hits        27990    27984       -6     
- Misses       5265     5269       +4     
- Partials      965      967       +2     
Flag Coverage Δ
Agent 82.76% <ø> (-0.02%) ⬇️
Profiler 71.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nr-ahemsath nr-ahemsath merged commit 425cdd1 into main Mar 20, 2026
212 of 215 checks passed
@nr-ahemsath nr-ahemsath deleted the ci/performance-test-visualization branch March 20, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants