[fix] Fix dotnet muxer detection under proot/sandboxed environments (Environment.ProcessPath)

[github-actions]

🤖 This is an automated fix generated in response to issue #16446.

Root cause

ProcessHelper.GetCurrentProcessFileName() returned Process.MainModule.FileName. On Unix, .NET derives MainModule.FileName from the first mapping in /proc/self/maps. Under sandboxing/emulation layers such as proot (used e.g. by Termux on Android), an unrelated loader binary is injected at the lowest address of every process, so MainModule.FileName reports that loader path instead of the real executable.

This breaks DotnetHostHelper.TryGetDotnetPathByArchitecture's fast path: when the current process architecture matches the target, it checks whether Path.GetFileName(GetCurrentProcessFileName()) equals dotnet (the muxer name) to short-circuit and use the current process as the muxer. Under proot, the reported filename is the injected loader (e.g. .../proot/loader), not dotnet, so this check fails even though the current process genuinely is the dotnet muxer. Execution then falls through to the search branch (DOTNET_ROOT* env vars, global registration, default install path), where IsValidArchitectureMuxer can never validate a muxer on Linux (it only inspects PE/Mach-O headers, not ELF), so every candidate — even a valid, explicitly configured one — is rejected, and dotnet test fails with "Could not find a 'dotnet' host for the '...' architecture."

Fix

Environment.ProcessPath (backed by /proc/self/exe on Unix, since .NET Core 3.0+) is not affected by the /proc/self/maps ordering issue and reliably returns the real process executable path. GetCurrentProcessFileName() now prefers Environment.ProcessPath when available (guarded by #if NET, i.e. the netcoreapp target), falling back to the previous Process.MainModule.FileName behavior otherwise (net462 target is unaffected by this bug, since it only ever runs on Windows).

Testing

Added ProcessHelperTests.GetCurrentProcessFileNameShouldReturnEnvironmentProcessPathWhenAvailable, a regression test asserting GetCurrentProcessFileName() returns Environment.ProcessPath when it is available. Ran the full vstest.console.UnitTests project filtered to ProcessHelperTests — all 5 tests (4 existing + 1 new) pass.

This is a narrow, low-risk change scoped to the exact root cause identified in the issue (suggested fix #1). The other suggestions in the issue (ELF header inspection for IsValidArchitectureMuxer, honoring explicit DOTNET_ROOT_ARM64) are larger, more architecturally involved changes better suited for a follow-up if this fix alone doesn't fully resolve the scenario — they aren't required once the fast path correctly recognizes the current process as the muxer.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• southcentralus0.in.applicationinsights.azure.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "southcentralus0.in.applicationinsights.azure.com"

See Network Configuration for more information.

🔍 Triaged by Issue Repro Triage & Auto-Fix 🔍

---

Note

This was originally intended as a pull request, but GitHub Actions is not permitted to create or approve pull requests in this repository.
The changes have been pushed to branch fix/issue-16446-96ee5fee03509c06.

Click here to create the pull request

To fix the permissions issue, go to Settings → Actions → General and enable Allow GitHub Actions to create and approve pull requests. See also: gh-aw FAQ

Show patch preview (88 of 88 lines)

From bf12df2ecd81548c61d5de30d035c54ae71c2abd Mon Sep 17 00:00:00 2001
X-GH-AW-Base-Commit: 1f0a9175d5dc8c95232e80ab538149c79f13597c
From: "github-actions[bot]" <github-actions[bot]@users.noreply.github.com>
Date: Mon, 7 Sep 2026 12:52:13 +0000
Subject: [PATCH] Fix dotnet muxer detection when MainModule.FileName is
 inaccurate (proot)

GetCurrentProcessFileName() used Process.MainModule.FileName, which on Unix
is derived from the first entry in /proc/self/maps. Under sandboxing/emulation
layers like proot (Android/Termux), an unrelated loader binary is mapped at
the lowest address, causing MainModule.FileName to report that loader instead
of the real dotnet executable. This broke DotnetHostHelper.TryGetDotnetPathByArchitecture's
fast path, which relies on detecting that the current process is the dotnet muxer.

Prefer Environment.ProcessPath (backed by /proc/self/exe on Unix) when
available, since it is not affected by this issue.

Fixes #16446

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
 .../common/System/ProcessHelper.cs            | 13 +++++++++++++
 .../ProcessHelperTests.cs                     | 19 +++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs b/src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs
index cb40d81..499da22 100644
--- a/src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs
+++ b/src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs
@@ -245,6 +245,19 @@ public partial class ProcessHelper : IProcessHelper
     /// <inheritdoc/>
     public string? GetCurrentProcessFileName()
     {
+#if NET
+        // On Unix, Process.MainModule.FileName is derived from the first entry in
+        // '/proc/self/maps'. Under some sandboxing/emulation layers (e.g. proot on
+        // Android/Termux) an unrelated loader binary is mapped at the lowest address,
+  
... (truncated)

[github-actions]

Thanks for the analysis! This is a duplicate of #16446, which already has an open fix PR: #16462. Closing the loop here to avoid duplicate parallel PRs — please follow #16462 for review/merge status.

🔍 Triaged by Issue Repro Triage & Auto-Fix 🔍