Detect and correct worker node over-provisioning

[Copilot]

Work item (Internal use):

Summary

When multiple MSBuild instances run concurrently, each spawns worker nodes up to /m limit (defaults to NUM_PROCS), causing over-provisioning. 25+ nodes can linger for ~15 minutes consuming resources.

Modified NodeProviderOutOfProcBase.ShutdownConnectedNodes to:

• Enumerate system-wide MSBuild processes when marking nodes idle
• Apply threshold (NUM_PROCS/2 for worker nodes)
• Send NodeBuildComplete(PrepareForReuse=false) to excess nodes, keeping only those needed to meet threshold

Implementation:

• DetermineNodesForReuse() - Core logic for selective reuse based on system-wide count
• GetNodeReuseThreshold() - Virtual method returning threshold (default: max(1, cores/2))
• CountSystemWideActiveNodes() - Virtual method using improved node detection from PR Improve cross-platform node discovery for reuse with NodeMode filtering #13256
  • Filters by NodeMode.OutOfProcNode for accurate worker node counting
  • Automatically handles dotnet processes and MSBuild.dll filtering
  • Leverages cross-platform process command line retrieval (Windows/macOS/Linux)

Integration with PR #13256:
This PR is rebased on top of PR #13256 ("Improve cross-platform node discovery for reuse with NodeMode filtering") and leverages its improved node detection logic for more accurate and consistent worker node identification across platforms.

Server and RAR nodes use separate lifecycle management and are not affected by this change.

Customer Impact

Reduces resource consumption in scenarios with concurrent builds (DevKit, LLM agents, CI/CD pipelines). System will maintain reasonable node count instead of accumulating 25+ lingering nodes.

Improved accuracy in node counting through NodeMode-based filtering ensures only actual worker nodes are counted, preventing false positives from main MSBuild processes or task hosts.

Regression?

• Yes
• No

Behavior change only when enableReuse=true (default). Conservative implementation with try/catch fallback if process enumeration fails.

Testing

8 unit tests added covering:

• Reuse disabled, threshold=0, under/at/over threshold scenarios
• Edge cases (single node, partial termination)
• Default threshold calculation

All existing tests pass. Tests remain valid with improved node detection as they use mocked implementations.

Risk

Low. Changes isolated to node shutdown path when reuse is enabled. Failure to enumerate processes defaults to keeping all nodes (current behavior). Diagnostic traces added for troubleshooting.

The integration with PR #13256's improved node detection reduces risk by providing more accurate node identification across platforms and better handling of dotnet processes.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Worker Nodes should attempt to detect and correct over-provisioning</issue_title>
<issue_description>### Summary

When multiple instances of MSBuild are run concurrently from the command line, each instance tries to create and claim a number of worker nodes up to the /m limit (which is NUMPROCS for dotnet by default).

This can lead to entirely too many nodes existing on the host machine, lingering for the configured timeout, which is ~15m by default.

These nodes suck up machine resources and look bad - the engine should work to minimize the number of active nodes past an expected threshold to preserve system resources.

Background and Motivation

As more tools, like DevKit, LLM Agents, and human users, delegate their build and inner-loop experiences to the dotnet CLI, it becomes more and more likely that collisions like the above will occur. I was looking at node trace logs this morning where more than 25 worker nodes were lingering on the machine. Managing these nodes is a pain for users of all kinds - you either have to carefully sequence your build operations, apply /nodereuse:false which slows down subsequent builds, or dangerously and likely incompletely terminate processes.

Proposed Feature

Instead, the nodes themselves should clean themselves up.

When a build completes and an out of proc node marks itself idle, it should attempt to handshake with its siblings. If there are a number of active nodes of the same type equal to a threshold, the idling node should terminate.

Proposed thresholds:

• Worker nodes: NUM_PROCS/2
• Server Nodes: > 0
• RAR-as-a-service-node: >0

Alternative Designs

No response</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Worker Nodes should attempt to detect and correct over-provisioning #13218

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[dotnet-policy-service]

Hello @@copilot, I noticed that you’re changing an .swr file or any file under src/Package/MSBuild.VSSetup.. Please make sure to validate this change by an experimental VS insertion. This is accomplished by pushing to an exp/* branch, which requires write permissions to this repo.

[baronfel]

@copilot rebase all of this work on #13256, and use the improved node-detection logic from that PR to handle the reaping more consistently.

[Copilot]

@copilot rebase all of this work on #13256, and use the improved node-detection logic from that PR to handle the reaping more consistently.

Rebased on PR #13256 and integrated the improved node detection logic.

Key improvements in commit f6e3056:

• CountSystemWideActiveNodes now filters by NodeMode.OutOfProcNode for accurate worker node counting
• Automatically handles dotnet processes and MSBuild.dll filtering
• Leverages cross-platform process command line retrieval from PR Improve cross-platform node discovery for reuse with NodeMode filtering #13256

The over-provisioning detection now uses the more robust node discovery mechanism, providing better consistency across platforms and more accurate identification of worker nodes vs. main MSBuild processes.