Wrong xUnit runner "xunit.execution.dotnet" chosen for .NET Framework property tests

[ronnieholm]

If you create a new .NET 4.8 test project and reference

• FsCheck.3.0.0-rc3
• FsCheck.Xunit.3.0.0-rc3
• xunit.2.7.0
• Other xUnit dependencies in 2.7.0 version

then the FsCheck [<Property>] test fails while the regular [<Fact>] test succeeds.

module FsCheckBug

open Xunit
open FsCheck.Xunit

[<Fact>]
let succeeds() =
    Assert.True(true)

[<Property>]
let fails() =
    Assert.True(true)

The failure is:

System.InvalidOperationException : Exception during discovery:
System.IO.FileNotFoundException: Could not load file or assembly 'xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
File name: 'xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
   at FsCheck.Xunit.PropertyDiscoverer.Xunit.Sdk.IXunitTestCaseDiscoverer.Discover(ITestFrameworkDiscoveryOptions discoveryOptions, ITestMethod testMethod, IAttributeInfo _arg1)
   at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForMethod(ITestMethod testMethod, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 119
   at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForType(ITestClass testClass, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 135

Two things stand out about the failure:

1. The test should've executed with xunit.execution.desktop.dll, not xunit.execution.dotnet.dll as the test project is a .NET Framework 4.8 project, not a .NET Core project.
2. The 2.4.1.0 version is wrong and should've been 2.7.0.0 as that's the version of the xunit.extensibility.execution NuGet package referenced.

Any tips on how to narrow down the cause?

[kurtschelfthout]

Maybe it’s the strong naming that causes trouble here.  I believe on newer .NET versions it’s handled automatically, but on framework you need an assembly redirect in a config file somewhere. This is compounded by figuring out how to make Xunit see that assembly redirect, as it does its one loading (ie there is no exe). Xunit docs may be some help there. On 2 Apr 2024, at 14:17, Ronnie Holm ***@***.***> wrote: Repro is available at https://github.com/ronnieholm/FsCheckWrongRunner If you create a new .NET 4.8 test project and reference FsCheck.3.0.0-rc3 FsCheck.Xunit.3.0.0-rc3 xunit.2.7.0 Other xUnit dependencies in 2.7.0 version then in the same test class the FsCheck [property] test fails while the regular [Fact] test succeeds. The failure is: System.InvalidOperationException : Exception during discovery: System.IO.FileNotFoundException: Could not load file or assembly 'xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified. File name: 'xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' at FsCheck.Xunit.PropertyDiscoverer.Xunit.Sdk.IXunitTestCaseDiscoverer.Discover(ITestFrameworkDiscoveryOptions discoveryOptions, ITestMethod testMethod, IAttributeInfo _arg1) at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForMethod(ITestMethod testMethod, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 119 at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForType(ITestClass testClass, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 135 Two things stand out about the failure: The test should've executed with xunit.execution.desktop.dll, not xunit.execution.dotnet as the test project is a .NET Framework 4.8 project, not a .NET Core project. The 2.4.1.0 version is wrong and should've been 2.7.0.0 as that's the version of the xunit.extensibility.execution NuGet package referenced. Any tips on how to narrow down the cause? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ronnieholm]

on framework you need an assembly redirect in a config file somewhere.

Indeed. I have other .NET Framework 4.8 test projects with xUnit and FsCheck where I need the following binding redirects in the test project's App.config.

<dependentAssembly>
    <assemblyIdentity name="xunit.core" publicKeyToken="8d05b1bb7a6fdb6c" culture="neutral" />
    <bindingRedirect oldVersion="2.2.0.0-2.4.2.0" newVersion="2.4.2.0" />
</dependentAssembly>
<dependentAssembly>
    <assemblyIdentity name="xunit.execution.desktop" publicKeyToken="8d05b1bb7a6fdb6c" culture="neutral" />
    <bindingRedirect oldVersion="2.2.0.0-2.4.2.0" newVersion="2.4.2.0" />
</dependentAssembly>

The above is from a test project running xUnit 2.4.2 and FsCheck 2.16.5. No problem with the [Property] attribute and discovery there.

For the Repro example, I've been using PerfView to trace assembly load events during test discovery and execution in VS. The following assemblies are getting loaded:

Notice the highlighted one: xunit.execution.desktop, version 2.7.0.0 as expected for a .NET Framework project.

What makes xUnit attempt to load xunit.execution.dotnet into the process is a bit of a mystery. It should be either xunit.execution.desktop or xunit.execution.dotnet, I believe.

[ronnieholm]

I was experimenting with keeping the xUnit version fixed at 2.7.0 and varying the FsCheck version, going back to 2.16.6.

FsCheck starts failing between release 3.0.0-alpha4 and 3.0.0-alpha5, i.e., alpha5 is the first release with the failure.

[ronnieholm]

From the results below, I'm not convinced the missing xunit.execution.dotnet.dll error is an issue with assembly redirects. Rather, it seems FsCheck.Xunit.dll is accidentally referencing xunit.execution.dotnet.dll.

I enabled Fusion logging and re-ran the test from the PowerShell terminal:

$ & "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" C:\Users\rh\Desktop\FsCheckWrongRunner\bin\Debug\Repro.dll

It yielded the following output:

  Failed Repro.Class1.ShouldNotFail [1 ms]
  Error Message:
   System.InvalidOperationException : Exception during discovery:
System.IO.FileNotFoundException: Could not load file or assembly 'xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
File name: 'xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
   at FsCheck.Xunit.PropertyDiscoverer.Xunit.Sdk.IXunitTestCaseDiscoverer.Discover(ITestFrameworkDiscoveryOptions discoveryOptions, ITestMethod testMethod, IAttributeInfo _arg1)
   at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForMethod(ITestMethod testMethod, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 119
   at Xunit.Sdk.XunitTestFrameworkDiscoverer.FindTestsForType(ITestClass testClass, Boolean includeSourceInformation, IMessageBus messageBus, ITestFrameworkDiscoveryOptions discoveryOptions) in /_/src/xunit.execution/Sdk/Frameworks/XunitTestFrameworkDiscoverer.cs:line 135

=== Pre-bind state information ===
LOG: DisplayName = xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
 (Fully-specified)
LOG: Appbase = file:///C:/Users/rh/Desktop/FsCheckWrongRunner/bin/Debug
LOG: Initial PrivatePath = NULL
Calling assembly : FsCheck.Xunit, Version=3.0.0.0, Culture=neutral, PublicKeyToken=44dc6321e9b07168.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Users\rh\Desktop\FsCheckWrongRunner\bin\Debug\Repro.dll.config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: xunit.execution.dotnet, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
LOG: Attempting download of new URL file:///C:/Users/rh/Desktop/FsCheckWrongRunner/bin/Debug/xunit.execution.dotnet.DLL.
LOG: Attempting download of new URL file:///C:/Users/rh/Desktop/FsCheckWrongRunner/bin/Debug/xunit.execution.dotnet/xunit.execution.dotnet.DLL.
LOG: Attempting download of new URL file:///C:/Users/rh/Desktop/FsCheckWrongRunner/bin/Debug/xunit.execution.dotnet.EXE.
LOG: Attempting download of new URL file:///C:/Users/rh/Desktop/FsCheckWrongRunner/bin/Debug/xunit.execution.dotnet/xunit.execution.dotnet.EXE.

Observe how the loader is attempting to locate xunit.execution.dotnet.dll in the FsCheckWrongRunner/bin/Debug folder. The file isn't there because the test project isn't a .NET project. The xunit.execution.desktop.dll on the other hand is present, copied there by MSBuild.

I then build Repro.csproj with binary logging enabled:

$ msbuild /bl .\Repro.csproj

Inspecting the log with the MSBuild Structured Log Viewer, it contains unexpected entries like this one:

Notice at the bottom it says Required by "FsCheck.Xunit" ... (the calling assembly according to the Fusion log). That's unexpected because according to the MSBuild log xunit.execution.desktop.dll is not required by FsCheck.Xunit.dll.

To verify, I loaded FsCheck.Xunit.dll into dnSpy:

Sure enough, xunit.execution.dotnet.dll (Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c) is a dependency of FsCheck.Xunit.dll.

I think this shouldn't be the case.

[kurtschelfthout]

That's indeed strange. Alpha 5 was the first version that used strong signing, perhaps that is related.

I have no idea how that dependency gets there either:

[kurtschelfthout]

Oh wait. Needed to click one more open:

Though I suppose "Compile time assemblies" means they should only be used at compile time?

[kurtschelfthout]

Ok, from the xunit site https://xunit.net/docs/nuget-packages

xunit.extensibility.execution | This package contains the execution libraries for all Supports         (xunit.execution.*.dll). It is intended to be used by developers who wish         to reference this DLL for extensibility purposes, such as creating custom test discovery         and execution.                       It differs from the xunit.core package in that it adds a reference         to the platform specific execution DLL, rather than simply copying to the project's output folder.

So that seems to be the right thing for us to reference.

Perhaps these are not forward compatible with xunit 2.7 though. If you downgrade xunit to 2.4 does it work?

[hglee]

I have same problem.

How about change FsCheck.Xunit.fsproj to multi target to net452;netstandard2.0?

FsCheck/src/FsCheck.Xunit/FsCheck.Xunit.fsproj

Line 5 in 0b2e216

<TargetFramework>netstandard2.0</TargetFramework>

[bash]

I am running into this same issue just now while trying to update Funcky to FsCheck 3.0.

@hglee's suggestion seems to work for me: I have tested the change to a multi-target locally.

[kurtschelfthout]

See xunit/xunit#2576

[kurtschelfthout]

released in 3.0.1