`AssemblyLoadContext` load multiple assemblies with same name

[linkem]

Description

Application loaded by AssemblyLoadContext (implementation from documentation) loads multiple .dll files with same AssemblyName which leads to exceptions like System.MissingMethodException Method not found: ...

Reproduction Steps

Small repo with reproduction code here

Repro with exception

• Build PluginApp1 application
• Start PluginHost
• Exception is thrown:

  System.MissingMethodException
      HResult=0x80131513
      Message=Method not found: 'System.Threading.Tasks.Task`1<!!0> System.Net.Http.Json.HttpContentJsonExtensions.ReadFromJsonAsync(System.Net.Http.HttpContent, System.Text.Json.JsonSerializerOptions, System.Threading.CancellationToken)'.
      Source=PluginApp1
      StackTrace:
          at PluginApp1.PluginApp1.<Execute>d__0.MoveNext() in C:\projects\GitHub\Github.AssemblyLoadContextError\PluginApp1\PluginApp1.cs:line 32
          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilder.cs:line 33
          at PluginApp1.PluginApp1.Execute(CancellationToken cancellationToken) in C:\projects\GitHub\Github.AssemblyLoadContextError\PluginApp1\PluginApp1.cs:line 9
          at Program.<<Main>$>d__0.MoveNext() in C:\projects\GitHub\Github.AssemblyLoadContextError\HostApp\Program.cs:line 12
          at Program.<Main>(String[] args)
  

Repro with listed duplicated assemblies

• Comment out PluginApp1.PluginApp1.cs line 30
• Build PluginApp1 application
• Start PluginHost
• Application should print output similar to this one:

  For Assembly 'System.Text.Json' found multiple instances:
  
  System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51; Location: <PROJECT_PATH>\PluginApp1\bin\Debug\net6.0\System.Text.Json.dll
  
  System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51; Location: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.11\System.Text.Json.dll
  

• Output indicates that there were loaded multiple assemblies with same name 'System.Text.Json'

Expected behavior

Plugin application should be started without duplicated assemblies, it should use assembly that is in local folder or in shared\Microsoft.NETCore.App but not both.

Actual behavior

In some cases duplicated assemblies are loaded.

Regression?

No response

Known Workarounds

For case presented in repro, in application PluginApp1 package reference of System.Text.Json can be changed from 6.0.7 to 6.0.0, but this is not applicable in more complex scenarios

Configuration

.NET SDK:
Version:   7.0.100
Commit:    e12b7af219

Runtime Environment:
OS Name:     Windows
OS Version:  10.0.19042
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\7.0.100\

Host:
Version:      7.0.0
Architecture: x64
Commit:       d099f075e4

But it also happens on dotnet 6 runtime

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Application loaded by AssemblyLoadContext (implementation from documentation) loads multiple .dll files with same AssemblyName which leads to exceptions like System.MissingMethodException Method not found: ...

Reproduction Steps

Small repo with reproduction code here

Repro with exception

• Build PluginApp1 application
• Start PluginHost
• Exception is thrown:

  System.MissingMethodException
      HResult=0x80131513
      Message=Method not found: 'System.Threading.Tasks.Task`1<!!0> System.Net.Http.Json.HttpContentJsonExtensions.ReadFromJsonAsync(System.Net.Http.HttpContent, System.Text.Json.JsonSerializerOptions, System.Threading.CancellationToken)'.
      Source=PluginApp1
      StackTrace:
          at PluginApp1.PluginApp1.<Execute>d__0.MoveNext() in C:\projects\GitHub\Github.AssemblyLoadContextError\PluginApp1\PluginApp1.cs:line 32
          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilder.cs:line 33
          at PluginApp1.PluginApp1.Execute(CancellationToken cancellationToken) in C:\projects\GitHub\Github.AssemblyLoadContextError\PluginApp1\PluginApp1.cs:line 9
          at Program.<<Main>$>d__0.MoveNext() in C:\projects\GitHub\Github.AssemblyLoadContextError\HostApp\Program.cs:line 12
          at Program.<Main>(String[] args)
  

Repro with listed duplicated assemblies

• Comment out PluginApp1.PluginApp1.cs line 30
• Build PluginApp1 application
• Start PluginHost
• Application should print output similar to this one:

  For Assembly 'System.Text.Json' found multiple instances:
  
  System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51; Location: <PROJECT_PATH>\PluginApp1\bin\Debug\net6.0\System.Text.Json.dll
  
  System.Text.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51; Location: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\6.0.11\System.Text.Json.dll
  

• Output indicates that there were loaded multiple assemblies with same name 'System.Text.Json'

Expected behavior

Plugin application should be started without duplicated assemblies, it should use assembly that is in local folder or in shared\Microsoft.NETCore.App but not both.

Actual behavior

In some cases duplicated assemblies are loaded.

Regression?

No response

Known Workarounds

For case presented in repro, in application PluginApp1 package reference of System.Text.Json can be changed from 6.0.7 to 6.0.0, but this is not applicable in more complex scenarios

Configuration

.NET SDK:
Version:   7.0.100
Commit:    e12b7af219

Runtime Environment:
OS Name:     Windows
OS Version:  10.0.19042
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\7.0.100\

Host:
Version:      7.0.0
Architecture: x64
Commit:       d099f075e4

But it also happens on dotnet 6 runtime

Other information

No response

Author:	linkem
Assignees:	-
Labels:	area-AssemblyLoader-coreclr, untriaged
Milestone:	-

[vitek-karas]

The problem is that your plugin brings its own version of System.Text.Json while using parts of the shared framework which also reference it. The framework which is loaded into the Default context uses S.T.Json from the Default context. The plugin code references the S.T.Json in the plugin load context. But then it calls code from the framework (HttpClient and so on). The problem is not very visible, it's default parameters. ReadFromJsonAsync has several default parameters one of which is JsonSerializerOptions which is a type from S.T.Json. The way default parameters work in .NET is that the compiler hardcodes the default at the callsite - so even though the source code looks like:

ReadFromJsonAsync<TodoModel>()

In reality the runtime sees the call as

ReadFromJsonAsync<TodoModel>((JsonSerializerOptions)null, default(CancellationToken));

The problem is that the JsonSerializerOptions type comes from the plugin - so the version of S.T.Json from the plugin load context. And thus it doesn't match the method declared in the framework which takes that type from the S.T.Json in the defautl load context.

In short - any assembly which is part of the shared framework should not be also carried by the plugins - it will break anytime where the plugin uses APIs which refer to types from such assembly and interoperates with the default framework APIs which do the same.

So in this case, remove the PackageReference to System.Text.Json.

The reason why it works if the package version is 6.0.0: If the version is 6.0.0 the SDK realizes that the version is available in the framework and won't actually put the file into the plugin. If the version is 6.0.7 that doesn't happen and the plugin carries its own copy.

There should be no need to have a package reference to System.Text.Json - it's part of the framework, why would you bring your own copy of it.

I know this is confusing to debug, because the assemblies/types look identical while the runtime doesn't see them as the same.

[linkem]

@vitek-karas Thank you for quick response and very detailed summary of issue.

I know that referencing S.T.Json in net6 applications is not required and can be easily avoided. My example is intentionally over simplified to reproduce problem. In real life, our net6 applications reference netstandard libraries that reference S.T.Json. In that case net6 application will have S.T.Json .dll in output without referencing it directly(transitive reference). It's basically impossible to avoid transitive references, especially in case of third party libs and nuget packages.

I have created repro with netstandard lib here. PluginApp don't reference S.T.Json directly but its present in output folder.

Just for record, this issue is not exclusive for S.T.Json, same thing happend with other libs.

Few more questions:

1. Is it intended behavior that AssemblyLoadContext loads assemblies with duplicated names from framework and local folder at the same time? I thought that the assembly present in the local folder will only be used if the Default context doesn't have this assembly or has it in older version, but will never load assemblies with same names.

2. How could we avoid issues like this? What would be best solution if we can't avoid referencing netstandard packages that have transitive references to packages from framework?

[vitek-karas]

The problem is versioning - the version 6.0.7 of System.Text.Json package contains the assembly with higher version than the one in the framework. This leads the SDK to keep it, instead of relying on the framework.
Note that this has nothing to do with netstandard targetting - if you change your project to reference version 6.0.0 the System.Text.Json will not be part of the plugin output even with the netstandard library used.

I don't know what are the version resolution rules in the SDK, they seem a bit harsh for this scenario.

@dsplaisted : If an app targeting net6.0 has a package reference to System.Text.Json version 6.0.7 (latest) it will get its own private copy of the assembly, even though that version is the same as the version in the latest runtime 6.0.11. Is there some way to tell the SDK to always rely on framework provided assembly even if the versions are a bit off? (I assume the behavior is because the ref package for 6.0.0 has a lower version of the assembly).

Is it intended behavior that AssemblyLoadContext loads assemblies with duplicated names from framework and local folder at the same time? I thought that the assembly present in the local folder will only be used if the Default context doesn't have this assembly or has it in older version, but will never load assemblies with same names.

Each instance of AssemblyLoadContext (ALC) will only load one version of a given assembly. But the important thing to realize is that ALCs are not isolation boundaries. They're only used when resolving assembly references (or explicit loads). Once assemblies are loaded ALCs are basically ignored. So you can have code which in theory has instances of the same type from two different versions of the assembly (from different ALCs) in one array (for example).

The assembly resolution for the Default ALC is governed by the app.deps.json which is produced by the SDK. Typically this means that the Default ALC will resolve only assemblies in the main app (and only those which were part of the build of the app).

The assembly resolution for the secondary ALCs is driven by the implementation of the AssemblyLoadContext.Load method. In your case (and typically for plugins), this uses AssemblyDependencyResolver. That class also relies on plugin.deps.json which is produced by the SDK. And it behaves similarly - in that it will only resolve assemblies which are part of the build of the plugin.

The "Sharing" of assemblies (in your case the plugin interface assembly) is done by not including that assembly in the output of the plugin, in which case AssemblyDependencyResolver will not resolve it, the AssemblyLoadContext.Load will return null and it will fall back to the Default ALC, which will resolve it from the main app.

How could we avoid issues like this? What would be best solution if we can't avoid referencing netstandard packages that have transitive references to packages from framework?

You could hardcode that you don't allow your plugins to carry their own version of assemblies provided by the host. In the AssemblyLoadContext.Load override call AssemblyLoadContext.Default.LoadFromAssemblyName(assemblyName) first and if it doesn't fail, use the retuned assembly over anything returned by the AssemblyDependencyResolver. The downside of this approach is that it requires try/catch as LoadFromAssemblyName will throw if it can't resolve the assembly. But if you're not loading that many plugins it should not be a big issue.

/cc @elinor-fung

[linkem]

You could hardcode that you don't allow your plugins to carry their own version of assemblies provided by the host. In the AssemblyLoadContext.Load override call AssemblyLoadContext.Default.LoadFromAssemblyName(assemblyName) first and if it doesn't fail, use the retuned assembly over anything returned by the AssemblyDependencyResolver. The downside of this approach is that it requires try/catch as LoadFromAssemblyName will throw if it can't resolve the assembly. But if you're not loading that many plugins it should not be a big issue.

This approach is something that I consider as last resort, only if we won't find any better solution.
Implementing it this way, we will lose possibility of loading Assemblies with different version than host already have. For example, if host depends on MediatR package in version 10.0.0 and my plugin app depends on version 11.0.0, then all plugins will be forced to use 10.0.0, this may break everything.
In real life scenario we have host application that have a lot of references and whole idea of using ALC was to enable our plugins to use their own versions of assemblies and override ones that host already have. Our approximate number of plugins will be around 10-15.

Is there any safe way to support scenario where host application has some assembly and plugin brings other version of same assembly?

[vitek-karas]

Is there any safe way to support scenario where host application has some assembly and plugin brings other version of same assembly?

It should work just fine, the problem only occurs if the code tries to use types from such assembly to communicate across the host boundary. That typically happens for assemblies which are part of the framework (as above) or those used in the plugin "interface" - all those should not ship with the plugin, but instead should be provided by the host.

[linkem]

It should work just fine, the problem only occurs if the code tries to use types from such assembly to communicate across the host boundary. That typically happens for assemblies which are part of the framework (as above) or those used in the plugin "interface" - all those should not ship with the plugin, but instead should be provided by the host.

For plugin "interface" situation is quite easy, we can use construct in csproj file like this

<ProjectReference Include="..\Plugin.Abstraction\Plugin.Abstraction.csproj">
    <Private>false</Private>
    <ExcludeAssets>runtime</ExcludeAssets>
</ProjectReference>

but what about all framework assemblies?

We have to create hardcoded list of all assemblies that are in dotnet/shared/6.0.*/Microsoft.NETCore.App and dotnet/shared/6.0.*/Microsoft.AspNetCore.App, then during build/publish plugin app should remove them from output folder? (it would be nice if there was some csproj parameter that could do it).

Another way would be to use hardcoded list of assemblies in Loader and use Default context to resolve them.

Anyway, both solution looks a bit weird, maybe there are some other way to achieve it?

[vitek-karas]

I agree it's not ideal - I honestly would like to see some SDK solution, because typically one would expect the SDK to resolve the framework dependencies to the framework.

There's no good way to tell if assembly belongs to the framework or not (you could use path, but that only works for framework dependent apps, and is still a bit weird) - which is kind of by design, because for self-contained app there's no "framework" really, everything is the "app".

[linkem]

Thank you for many tips.
For now we do not consider to use self-contained apps, at least not in terms of plugins. I have created small PoC of the solution that we came up here. It looks like in this simple example its working fine, I need some more time to check it on bigger "real life" project.
Could you check if my PoC is something that you had in mind (PluginLoadContext and hardcoded list of assemblies)?

[vitek-karas]

That should work, it's obviously fragile - I don't think new assemblies will be added a patch release, but I wouldn't be surprised if the list looked different for .NET 7 and so on. That said it's probably the best/simplest one can do right now.

[dsplaisted]

The problem is versioning - the version 6.0.7 of System.Text.Json package contains the assembly with higher version than the one in the framework. This leads the SDK to keep it, instead of relying on the framework. Note that this has nothing to do with netstandard targetting - if you change your project to reference version 6.0.0 the System.Text.Json will not be part of the plugin output even with the netstandard library used.

I don't know what are the version resolution rules in the SDK, they seem a bit harsh for this scenario.

@dsplaisted : If an app targeting net6.0 has a package reference to System.Text.Json version 6.0.7 (latest) it will get its own private copy of the assembly, even though that version is the same as the version in the latest runtime 6.0.11. Is there some way to tell the SDK to always rely on framework provided assembly even if the versions are a bit off? (I assume the behavior is because the ref package for 6.0.0 has a lower version of the assembly).

No, there's not currently a way to do this. I've filed dotnet/sdk#29280 to track adding a way to do so.