[BUG]: DatadogCoverage writes a damaged assembly when a referenced assembly is only in a shared framework

[DmytroKuian]

Tracer Version(s)

3.50.0

Operating system and platform

Linux (x64), Ubuntu 24.04, self-hosted GitHub Actions runner

Instrumentation Mode

Manual (dd-trace ci run)

TFM

net10.0

Bug Report

Datadog.Trace.Coverage.collector changes each assembly in the test output directory with Mono.Cecil. The change fails when Mono.Cecil must find a type from an assembly that is only in the installed ASP.NET Core shared framework. CoverageAssemblyResolver looks in the test output directory, but it does not look in dotnet/shared/Microsoft.AspNetCore.App/<version>/. Thus the search fails and Mono.Cecil throws an exception.

There are two different problems:

1. The resolver does not find shared-framework assemblies. These assemblies are on the machine. They are in the same directory from which the runtime loads them at test time.
2. The failure mode is not safe. The collector catches the exception and continues, but it already opened the target assembly for write. A damaged DLL stays on the disk. The test host then fails with FileNotFoundException for the assembly of the project. A coverage collector must not damage the assemblies under test. This second problem does more damage than the first one. You can correct it independently.

Version details

• .NET SDK: 10.0.302; runtime: 10.0.10; shared framework: Microsoft.AspNetCore.App 10.0.10
• Source code references below are from master at the time of this report

We saw the failure when we use dd-trace ci run -- dotnet test .... In this mode CiUtils adds --collect DatadogCoverage to the command line. The source code shows a second path to the same collector: TestCommandctorIntegration adds -property:VSTestCollect="DatadogCoverage" to the MSBuild arguments when only the CLR profiler is active. We did not test that second path, but it uses the same collector code.

Expected behaviour

One of these two results:

• CoverageAssemblyResolver finds Microsoft.Extensions.Logging.Abstractions in the installed shared framework, and the change of the assembly is successful.
• The collector does not change that assembly, and it keeps the initial assembly on the disk. The initial assembly stays correct, and the tests run.

Actual behaviour

Data collector 'DatadogCoverage' message: CoverageAssemblyResolver failed to resolve dependency
'Microsoft.Extensions.Logging.Abstractions, Version=10.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
while processing target assembly '.../bin/Release/net10.0/Repro.Library.dll'.

Mono.Cecil.AssemblyResolutionException: Failed to resolve assembly:
'Microsoft.Extensions.Logging.Abstractions, Version=10.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Mono.Cecil.BaseAssemblyResolver.Resolve(AssemblyNameReference name, ReaderParameters parameters)
   at Datadog.Trace.Coverage.Collector.CoverageAssemblyResolver.ResolveWithoutDirectoryFallback(AssemblyNameReference name)
   at Datadog.Trace.Coverage.Collector.CoverageAssemblyResolver.ResolveAndCache(AssemblyNameReference name)
   at Datadog.Trace.Coverage.Collector.CoverageAssemblyResolver.Resolve(AssemblyNameReference name, ReaderParameters parameters)

Then the tests start, and each test that uses the assembly fails:

SetUp : System.IO.FileNotFoundException : Could not load file or assembly
'Repro.Library, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.
The system cannot find the file specified.

Why the assembly is not in the output directory

This is correct SDK behaviour. It is not a configuration error. A test project that references an ASP.NET Core project gets a Microsoft.AspNetCore.App framework reference:

"frameworks": [
  { "name": "Microsoft.NETCore.App",    "version": "10.0.0" },
  { "name": "Microsoft.AspNetCore.App", "version": "10.0.0" }
]

Microsoft.Extensions.Logging.Abstractions is in that shared framework. Thus conflict resolution removes the NuGet copy from bin/ and from deps.json. At test time the host loads the assembly from the shared framework directory. Only the Mono.Cecil rewrite fails, because it looks in the output directory.

Why a simple reference does not cause the failure

This is important, because it makes the bug look irregular. An assembly reference to an absent assembly is usually safe. Mono.Cecil does not resolve each entry in the assembly reference table. A class with an ILogger<T> field that calls LogInformation is safe, also in an async method.

The cause is metadata that makes Mono.Cecil resolve the reference. In our code it is an optional parameter with an enum type from the absent assembly:

public static async Task<T?> ProcessAsync<T>(
    Task<T> operation,
    ILogger logger,
    string failureMessage,
    LogLevel logLevel = LogLevel.Error,   // <-- external enum as a parameter constant
    params object[] messageArgs)
    where T : class

The collector writes the changed assembly back to the disk. At that moment Mono.Cecil.AssemblyWriter.GetConstantType finds the external enum. It calls CheckedResolve() to get the integral type of the enum. That call goes to CoverageAssemblyResolver, and the resolver throws. Other metadata that needs resolution at write time can cause the same failure.

Remove only the LogLevel logLevel = LogLevel.Error parameter, and the run is successful. This shows the cause.

Related problem: the code coverage flag has no effect

If Test Impact Analysis is on for the service, the failure also happens when you do not set DD_CIVISIBILITY_CODE_COVERAGE_ENABLED. It continues to happen when you set that variable to false. Test Impact Analysis requires per-test coverage and re-enables the collector. The workaround is to set both DD_CIVISIBILITY_CODE_COVERAGE_ENABLED=false and DD_CIVISIBILITY_ITR_ENABLED=false.

Where the code does this

For the first problem, CoverageAssemblyResolver.GetSearchDirectoryCandidates() gives these directories:

1. _preferredSearchDirectory, which is the directory of the assembly that the collector changes.
2. The search directories of the Mono.Cecil BaseAssemblyResolver.

If these directories do not contain the assembly, ResolveWithoutDirectoryFallback removes all search directories and uses only the platform or TPA path. No step looks in the installed shared frameworks.

For the second problem, AssemblyProcessor.WriteTargetAssembly writes the new assembly directly to the initial file path:

using var assemblyLock = CoverageAssemblyPathLock.EnterWrite(assemblyFilePath);
assemblyDefinition.Write(assemblyFilePath, new WriterParameters { ... });

There is no temporary file, and there is no copy of the initial assembly. If Write fails in the middle, the initial assembly on the disk is already damaged. The nearest catch blocks accept only SymbolsNotFoundException and SymbolsNotMatchingException, so an AssemblyResolutionException goes to the caller and the damaged file stays.

Suggested correction

For the first problem, use the same solution as Coverlet. Coverlet had the same failure with the same assembly (issue 1231, issue 1631). It was corrected in PR 1449, which adds a resolver that looks in the installed shared frameworks. This is why Coverlet changes these assemblies correctly in the same test run in which DatadogCoverage fails.

For the second problem, make sure that a failed rewrite cannot leave a damaged assembly on the disk. Two possible methods:

• Write to a temporary file. Move that file to the initial path only after the write is successful.
• Make a copy of the initial assembly before the write. Put the copy back if the write fails.

Then a future gap in the resolver causes only one result: the collector does not change one assembly. It does not cause a test run that stops with an incorrect FileNotFoundException.

Workarounds that we tested

• DD_CIVISIBILITY_ITR_ENABLED=false and DD_CIVISIBILITY_CODE_COVERAGE_ENABLED=false
• A copy of the assembly into the test output with an MSBuild target. This works, but each project must contain the assembly name, the package version, and the path layout. It is difficult to keep correct.

Related issue

#8592 is a different bug in the same collector (file locking during parallel processing). It is closed.

Reproduction Code

Three projects, all with the target framework net10.0:

• Repro.Library — a class library. It has a PackageReference to Microsoft.Extensions.Logging.Abstractions 10.0.5, and it contains the method above.
• Repro.Web — a Microsoft.NET.Sdk.Web project with the default minimal API. It gives the test project a transitive Microsoft.AspNetCore.App framework reference. It has no other function.
• Repro.Tests — a Microsoft.NET.Sdk project with Microsoft.NET.Test.Sdk and NUnit. It has a ProjectReference to the two projects above, and one test that calls ProcessAsync.

Repro.Library/Repro.Library.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.5" />
  </ItemGroup>
</Project>

Repro.Library/LoggingHelper.cs

using Microsoft.Extensions.Logging;

namespace Repro.Library;

public static class LoggingHelper
{
    public static async Task<T?> ProcessAsync<T>(
        Task<T> operation,
        ILogger logger,
        string failureMessage,
        LogLevel logLevel = LogLevel.Error,
        params object[] messageArgs)
        where T : class
    {
        try
        {
            var result = await operation;
            if (result is not null)
                return result;

            logger.Log(logLevel, "Operation failed: {FailureMessage}", failureMessage);
            return default;
        }
        catch (Exception exception)
        {
            logger.LogError(exception, "Unexpected failure: {FailureMessage}", failureMessage);
            return default;
        }
    }
}

Repro.Web/Repro.Web.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
  </PropertyGroup>
</Project>

Repro.Web/Program.cs

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/", () => "ok");
app.Run();

Repro.Tests/Repro.Tests.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.3.0" />
    <PackageReference Include="NUnit" Version="4.5.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="6.2.0" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="../Repro.Library/Repro.Library.csproj" />
    <ProjectReference Include="../Repro.Web/Repro.Web.csproj" />
  </ItemGroup>
</Project>

Repro.Tests/GreeterTests.cs

using Microsoft.Extensions.Logging.Abstractions;
using NUnit.Framework;
using Repro.Library;

public class GreeterTests
{
    [Test]
    public async Task ProcessesAsyncOperation()
    {
        var result = await LoggingHelper.ProcessAsync(
            Task.FromResult("Hello, world"),
            NullLogger.Instance,
            "Greeting failed");

        Assert.That(result, Is.EqualTo("Hello, world"));
    }
}

dotnet new sln -n Repro
dotnet sln add Repro.Library/Repro.Library.csproj Repro.Web/Repro.Web.csproj Repro.Tests/Repro.Tests.csproj
dotnet build Repro.sln -c Release

# Precondition 1: the assembly is not in the test output.
ls Repro.Tests/bin/Release/net10.0/Microsoft.Extensions.Logging.Abstractions.dll   # no such file

# Precondition 2: the framework reference is present.
grep Microsoft.AspNetCore.App Repro.Tests/bin/Release/net10.0/Repro.Tests.runtimeconfig.json

export DD_API_KEY=<key>
export DD_SITE=datadoghq.com
export DD_CIVISIBILITY_AGENTLESS_ENABLED=true
export DD_CIVISIBILITY_CODE_COVERAGE_ENABLED=true

dd-trace ci run -- dotnet test Repro.sln -c Release --no-build