Skip to content

Track test counts per execution. #115707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

WangyangZhou90
Copy link
Contributor

@WangyangZhou90 WangyangZhou90 commented May 19, 2025

Previous code Tests ran counter by VincentBu · Pull Request #111145 · dotnet/runtime added test run counter which records how many times each test run and write them at the end of logging file. However it doesn't work when tests is killed by assert failures or other errors. This pr aims to record each time a test is completed.
the log before looks like:

[5/14/2025 8:34:28 PM 1] RF.StartTest, RTs(7) - Instances of this test: 2 - New Test:GCPerfSim - LOH A Lot Of Live Data., 31 threads
[5/14/2025 8:34:28 PM 1] RUN 5/14/2025 8:34:28 PM Test Started: GCPerfSim - POH Scenario.
 3
[5/14/2025 8:34:28 PM 1] RF.StartTest, RTs(8) - Instances of this test: 2 - New Test:GCPerfSim - POH Scenario., 32 threads
[5/14/2025 8:34:28 PM 1] RUN 5/14/2025 8:34:28 PM Test Started: GCPerfSim - LOH No Live Data.
 0
[5/14/2025 8:34:28 PM 1] RF.StartTest, RTs(9) - Instances of this test: 3 - New Test:GCPerfSim - LOH No Live Data., 33 threads
[5/14/2025 8:34:28 PM 1] RUN 5/14/2025 8:34:28 PM Test Started: GCPerfSim - LOH Some Live Data.
 1
[5/14/2025 8:34:28 PM 1] RF.StartTest, RTs(10) - Instances of this test: 3 - New Test:GCPerfSim - LOH Some Live Data., 34 threads
[5/14/2025 8:34:28 PM 1] RUN 5/14/2025 8:34:28 PM Test Started: GCPerfSim - LOH A Lot Of Live Data.
 2
[5/14/2025 8:34:28 PM 1] RF.StartTest, RTs(11) - Instances of this test: 3 - New Test:GCPerfSim - LOH A Lot Of Live Data., 35 threads
[5/14/2025 8:34:28 PM 1] RUN 5/14/2025 8:34:28 PM Test Started: GCPerfSim - POH Scenario.

after:

[5/20/2025 2:20:32 PM 1] RF.StartTest, RTs(7) - Instances of this test: 2 - New Test:GCPerfSim - LOH A Lot Of Live Data., 31 threads
[5/20/2025 2:20:32 PM 1] Run tests "GCPerfSim - LOH A Lot Of Live Data." 2 times.
[5/20/2025 2:20:32 PM 1] RUN 5/20/2025 2:20:32 PM Test Started: GCPerfSim - POH Scenario.
 3
[5/20/2025 2:20:32 PM 1] RF.StartTest, RTs(8) - Instances of this test: 2 - New Test:GCPerfSim - POH Scenario., 32 threads
[5/20/2025 2:20:32 PM 1] Run tests "GCPerfSim - POH Scenario." 2 times.
[5/20/2025 2:20:32 PM 1] RUN 5/20/2025 2:20:32 PM Test Started: GCPerfSim - LOH No Live Data.
 0
[5/20/2025 2:20:32 PM 1] RF.StartTest, RTs(9) - Instances of this test: 3 - New Test:GCPerfSim - LOH No Live Data., 33 threads
[5/20/2025 2:20:32 PM 1] Run tests "GCPerfSim - LOH No Live Data." 3 times.
[5/20/2025 2:20:32 PM 1] RUN 5/20/2025 2:20:32 PM Test Started: GCPerfSim - LOH Some Live Data.
 1
[5/20/2025 2:20:32 PM 1] RF.StartTest, RTs(10) - Instances of this test: 3 - New Test:GCPerfSim - LOH Some Live Data., 34 threads
[5/20/2025 2:20:32 PM 1] Run tests "GCPerfSim - LOH Some Live Data." 3 times.
[5/20/2025 2:20:32 PM 1] RUN 5/20/2025 2:20:32 PM Test Started: GCPerfSim - LOH A Lot Of Live Data.
 2
[5/20/2025 2:20:32 PM 1] RF.StartTest, RTs(11) - Instances of this test: 3 - New Test:GCPerfSim - LOH A Lot Of Live Data., 35 threads
[5/20/2025 2:20:32 PM 1] Run tests "GCPerfSim - LOH A Lot Of Live Data." 3 times.
[5/20/2025 2:20:32 PM 1] RUN 5/20/2025 2:20:32 PM Test Started: GCPerfSim - POH Scenario.
 3
[5/20/2025 2:20:32 PM 1] RF.StartTest, RTs(12) - Instances of this test: 3 - New Test:GCPerfSim - POH Scenario., 36 threads
[5/20/2025 2:20:32 PM 1] Run tests "GCPerfSim - POH Scenario." 3 times.

@Copilot Copilot AI review requested due to automatic review settings May 19, 2025 02:25
@dotnet-policy-service dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label May 19, 2025
@github-actions github-actions bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 19, 2025
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces functionality to track test counts per execution by recording and logging the number of test runs identified by a reference or ID. Key changes include:

  • Adding the RecordTestRunCountSingle method to log individual test counts.
  • Updating the StartTest method to increment test run counts and call the new logging method.
  • Removing redundant test count increment logic from the StartTestWorker method.

Comment on lines 342 to 348
foreach(var item in _testRunCounter)
{
if(item.Key == refOrID)
{
_logger.WriteToInstrumentationLog(_curTestSet, LoggingLevels.StartupShutdown, $"Run tests \"{item.Key}\" {item.Value} times.");
}
}
Copy link
Preview

Copilot AI May 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of iterating over the entire _testRunCounter dictionary to find a matching key, consider using a direct dictionary lookup (e.g., TryGetValue) to improve performance.

Suggested change
foreach(var item in _testRunCounter)
{
if(item.Key == refOrID)
{
_logger.WriteToInstrumentationLog(_curTestSet, LoggingLevels.StartupShutdown, $"Run tests \"{item.Key}\" {item.Value} times.");
}
}
if (_testRunCounter.TryGetValue(refOrID, out var value))
{
_logger.WriteToInstrumentationLog(_curTestSet, LoggingLevels.StartupShutdown, $"Run tests \"{refOrID}\" {value} times.");
}

Copilot uses AI. Check for mistakes.

Comment on lines 339 to 350
StringBuilder sb = new();
lock (_testRunCounterLock)
{
foreach(var item in _testRunCounter)
{
if(item.Key == refOrID)
{
_logger.WriteToInstrumentationLog(_curTestSet, LoggingLevels.StartupShutdown, $"Run tests \"{item.Key}\" {item.Value} times.");
}
}
}

Copy link
Preview

Copilot AI May 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor the loop in RecordTestRunCountSingle to use direct access of _testRunCounter by key, which will simplify the code and reduce the risk of missing the log when the key is absent.

Suggested change
StringBuilder sb = new();
lock (_testRunCounterLock)
{
foreach(var item in _testRunCounter)
{
if(item.Key == refOrID)
{
_logger.WriteToInstrumentationLog(_curTestSet, LoggingLevels.StartupShutdown, $"Run tests \"{item.Key}\" {item.Value} times.");
}
}
}
lock (_testRunCounterLock)
{
if (_testRunCounter.TryGetValue(refOrID, out var value))
{
_logger.WriteToInstrumentationLog(_curTestSet, LoggingLevels.StartupShutdown, $"Run tests \"{refOrID}\" {value} times.");
}
}

Copilot uses AI. Check for mistakes.

@am11 am11 added area-GC-coreclr and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 19, 2025
Copy link
Contributor

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

@Maoni0
Copy link
Member

Maoni0 commented May 19, 2025

Copilot had good suggestions.

also please add some description to the PR - is there already per test logging? if so what value does this add (was the current logging insufficient)? what does the log look like before and after your change? and etc.

@WangyangZhou90
Copy link
Contributor Author

Copilot had good suggestions.

also please add some description to the PR - is there already per test logging? if so what value does this add (was the current logging insufficient)? what does the log look like before and after your change? and etc.

thanks for the feedback, I've updated the code and added some description to the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-GC-coreclr community-contribution Indicates that the PR has been added by a community member
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants