Skip to content

TEMPORARY: Overlay custom MSBuild for dotnet/msbuild#12927 investigation#66003

Draft
ilonatommy wants to merge 31 commits intodotnet:mainfrom
ilonatommy:msbuild-bootstrap-investigation
Draft

TEMPORARY: Overlay custom MSBuild for dotnet/msbuild#12927 investigation#66003
ilonatommy wants to merge 31 commits intodotnet:mainfrom
ilonatommy:msbuild-bootstrap-investigation

Conversation

@ilonatommy
Copy link
Copy Markdown
Member

@ilonatommy ilonatommy commented Mar 27, 2026

DO NOT MERGE - Investigation branch only

Purpose

Overlay custom Microsoft.Build.Tasks.Core.dll from the MSBuild team to diagnose intermittent MSB3030: Could not copy file errors during parallel CI builds.

Tracking issue: dotnet/msbuild#12927
Known Build Error: #62807

hits from the known build issue happened on:

Job Step Platform
Test: Ubuntu x64 Run build.sh Linux
Build: Windows x64/x86/arm64 Run build.cmd Windows
Build: Windows x64/x86/arm64 Build Installers Windows
(internal) dotnet-aspnetcore Run build.sh Linux
(internal) dotnet-aspnetcore Build Installers Windows

So it hits both Linux and Windows build legs, specifically during the large parallel compilation steps (Run build.sh, Run build.cmd, Build Installers). These are the jobs where /m parallelism triggers the race condition. Our overlay covers both build.sh and build.ps1, so we're
covered on all the failing legs.

❯ are these legs running on our playground PR?

What this does

  1. Commits a custom Microsoft.Build.Tasks.Core.dll (with extra logging) to eng/msbuild-bootstrap
  2. After SDK initialization in eng/build.sh and eng/build.ps1, overlays the custom DLL onto .dotnet/sdk<version>/Microsoft.Build.Tasks.Core.dll
  3. Prints SHA256 hashes to the build log so we can verify the overlay worked

Expected outcome

  • Build logs will show === MSBuild Bootstrap Overlay ===section confirming the DLL was replaced
  • If the MSB3030 error occurs, the binlog will contain additional diagnostic information from the custom MSBuild
  • The binlog artifact should be shared with the MSBuild team (@OvesN) for analysis

How to revert

Close this PR. No repo-wide changes are needed.

This PR should be updated automatically when my devbox's task scheduler detects that CI finished without error that we're waiting for.

Add custom Microsoft.Build.Tasks.Core.dll from the MSBuild team with
additional logging to diagnose intermittent MSB3030 'Could not copy file'
errors in parallel builds.

The DLL is overlaid onto the SDK's copy in eng/build.sh and eng/build.ps1
after SDK initialization, with SHA256 verification output in the build log.

DO NOT MERGE - investigation branch only.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions bot added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Mar 27, 2026
ilonatommy and others added 2 commits March 27, 2026 14:59
Windows PowerShell 5.1 (used on CI agents) only supports 2 arguments
to Join-Path. Nest the calls instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The env var may not be set in all CI build configurations. Fall back
to RepoRoot/.dotnet which configure-toolset.ps1 always forces.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ilonatommy ilonatommy added the * NO MERGE * Do not merge this PR as long as this label is present. label Mar 27, 2026
ilonatommy and others added 6 commits March 27, 2026 15:56
Scheduled task script that:
- Checks Build Analysis check run to see if CI is done
- Searches AzDO logs for MSB3030/MSB3026 errors
- If not found, merges upstream/main and retriggers CI
- If found, logs alert and stops

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ilonatommy
Copy link
Copy Markdown
Member Author

/azp run aspnetcore-ci

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@ilonatommy
Copy link
Copy Markdown
Member Author

/azp run aspnetcore-ci

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@ilonatommy
Copy link
Copy Markdown
Member Author

🚨 MSB3030 "File Not Found" Error Detected (Automated)

Build: #1365043

Errors found:

  • [Run build.sh] 2026-04-03T03:09:32.6419596Z ##[error].dotnet/sdk/11.0.100-preview.1.26104.118/Microsoft.Common.CurrentVersion.targets(5094,5): error MSB3030: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy the file "/mnt/vss/_work/1/s/artifacts/bin/Microsoft.Net.Http.Headers/Release/net11.0/Microsoft.Net.Http.Headers.pdb" because it was not found.
  • [Run build.sh] 2026-04-03T03:09:32.6428055Z /mnt/vss/_work/1/s/.dotnet/sdk/11.0.100-preview.1.26104.118/Microsoft.Common.CurrentVersion.targets(5094,5): error MSB3030: Could not copy the file "/mnt/vss/_work/1/s/artifacts/bin/Microsoft.Net.Http.Headers/Release/net11.0/Microsoft.Net.Http.Headers.pdb" because it was not found. [/mnt/vss/_work/1/s/src/Middleware/Localization/test/UnitTests/Microsoft.AspNetCore.Localization.Tests.csproj]
  • [Run build.sh] 2026-04-03T03:16:00.7069704Z /mnt/vss/_work/1/s/.dotnet/sdk/11.0.100-preview.1.26104.118/Microsoft.Common.CurrentVersion.targets(5094,5): error MSB3030: Could not copy the file "/mnt/vss/_work/1/s/artifacts/bin/Microsoft.Net.Http.Headers/Release/net11.0/Microsoft.Net.Http.Headers.pdb" because it was not found. [/mnt/vss/_work/1/s/src/Middleware/Localization/test/UnitTests/Microsoft.AspNetCore.Localization.Tests.csproj]
  • [Test: Ubuntu x64] 2026-04-03T03:09:32.6419631Z ##[error].dotnet/sdk/11.0.100-preview.1.26104.118/Microsoft.Common.CurrentVersion.targets(5094,5): error MSB3030: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy the file "/mnt/vss/_work/1/s/artifacts/bin/Microsoft.Net.Http.Headers/Release/net11.0/Microsoft.Net.Http.Headers.pdb" because it was not found.
  • [Test: Ubuntu x64] 2026-04-03T03:09:32.6428075Z /mnt/vss/_work/1/s/.dotnet/sdk/11.0.100-preview.1.26104.118/Microsoft.Common.CurrentVersion.targets(5094,5): error MSB3030: Could not copy the file "/mnt/vss/_work/1/s/artifacts/bin/Microsoft.Net.Http.Headers/Release/net11.0/Microsoft.Net.Http.Headers.pdb" because it was not found. [/mnt/vss/_work/1/s/src/Middleware/Localization/test/UnitTests/Microsoft.AspNetCore.Localization.Tests.csproj]
  • [Test: Ubuntu x64] 2026-04-03T03:16:00.7069715Z /mnt/vss/_work/1/s/.dotnet/sdk/11.0.100-preview.1.26104.118/Microsoft.Common.CurrentVersion.targets(5094,5): error MSB3030: Could not copy the file "/mnt/vss/_work/1/s/artifacts/bin/Microsoft.Net.Http.Headers/Release/net11.0/Microsoft.Net.Http.Headers.pdb" because it was not found. [/mnt/vss/_work/1/s/src/Middleware/Localization/test/UnitTests/Microsoft.AspNetCore.Localization.Tests.csproj]

Action required: Collect binlog from this build and share with MSBuild team (@OvesN on dotnet/msbuild#12927).

This comment was posted automatically by the CI monitor script.

ilonatommy and others added 6 commits April 10, 2026 08:54
New DLL: 18.4.0+a18aac5435e (build 20901)
Old DLL: 18.4.0+6e61e96acff (build 17701)
SHA256: 95684FA027FCA4A5212925F1E429D9C34C458715D1591872F28BE57C74E88810

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…estigation

# Conflicts:
#	eng/Version.Details.props
#	eng/Version.Details.xml
#	global.json
#	src/Components/WebAssembly/WebAssembly/src/Properties/ILLink.Substitutions.xml
#	src/Components/test/E2ETest/Tests/WebAssemblyTrimmingTest.cs
#	src/Components/test/testassets/BasicTestApp/Index.razor
#	src/Components/test/testassets/Components.TestServer/Components.TestServer.csproj
Preview.3 removed FrameworkFileUtilities.RemoveRelativeSegments which
the custom Tasks DLL depends on. Preview.1 has it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
System.IO.FileHandleType was added in preview.3. With preview.1 pinned,
this is not needed, but the merge brought in code that uses unqualified
FileHandleType. Fully qualify to Microsoft.AspNetCore.Connections.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Revert SDK pin to preview.3 since V3 DLL is now compatible.
SHA256: 5F3AE9B75488BBCBAB51A9954414BA5BC70A1D487B5EC789876697F8A1092E99

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The custom Tasks DLL calls RemoveRelativeSegments which was removed from
the SDK's Framework DLL in preview.3. Overlay the matching Framework DLL
from the same MSBuild build (18.4.0+a18aac54) to ensure compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework * NO MERGE * Do not merge this PR as long as this label is present.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant