Addressing Assembly Conflict Challenges in OpenTelemetry.AutoInstrumentation for .NET

[alexeypukhov]

Current State and Challenges

OpenTelemetry.AutoInstrumentation for .NET (Core) often requires setting the DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE environment variables. This helps resolve assembly version conflicts between the instrumented application and the instrumentation tool, allowing both to load their required dependencies and ensuring that auto-instrumentation works correctly.
However, this is not a foolproof solution. The following issues remain:

• Ongoing assembly conflicts:
  See [Feature]: StartupHook dependency resolving dotnet/runtime#66138 and Consider removing Microsoft.Extensions.* from additional deps #4155

• Limited support for self-contained applications:
  Mentioned in the original ticket (needs re-verification): Diagnostic Source assembly versioning Issue #260

• Frequent maintenance:
  Updates to the .NET runtime, application dependencies, or instrumentation tools may require repeated updates to the shared store and deps files.

Profiler-based Proposed Solution

Based on my initial experiments with the assembly reference redirection approach (already used in profiler-based instrumentation for .NET Framework), it seems promising that it can be effectively adapted for .NET (Core) by leveraging an already in-use custom AssemblyLoadContext which can be further enhanced (right now it is not used in its full potential).

StartupHook-only Proposed Solution

For StartupHook-only deployments (without a profiler), I experimented with a custom AssemblyLoadContext AND redirecting the instrumented application into this custom context (the secret sauce for this solution). I tried it in a sandbox project (not with the actual OpenTelemetry project yet) but loading the instrumented application into a custom context reduces the influence of the Trusted Platform Assemblies (TPA) list on the automatic resolution of assemblies by the .NET runtime. This makes it much easier to intervene and customize the assembly loading process, giving us more flexibility to resolve conflicts as needed in the use case where profiler is not loaded.

---

Both solutions could potentially work side by side, giving customers the option to choose their preferred approach (with assembly reference redirection as the default for profiler-based scenarios and the extended AssemblyLoadContext for StartupHook-only scenarios). Together, these approaches might address most dependency issues without requiring to set additional dependencies nor shared store environment variables. I would even go further and remove the store approach altogether; we may make it work with the store approach, but it will probably add unnecessary complexity. In addition, these two new approaches would build on what already works for .NET Framework and make a fuller use of already existing AssemblyLoadContext for assembly isolation in .NET (Core).

Potential Considerations

• The StartupHook-only solution would result in the customer assembly being loaded twice (including once in the additional context) which will require additional handling to be eliminated.
• Some applications may explicitly use AssemblyLoadContext.Default to load assemblies, which could require additional handling, especially for StartupHook-only deployments, where the application is loaded into a separate context.
• If a customer subscribes to an assembly resolve event to load a newer version of an assembly than the one we've already loaded, this may not be supported, though this scenario should be relatively rare.

Overall, while these considerations exist, I believe these approaches should significantly reduce dependency conflicts and simplify deployment for most use cases.

[nrcventura]

A similar idea was investigated in #3896. If I'm remembering correctly we ran into some problems with a couple of our features when managing the different ALCs. It would be interesting to know if there is a way to resolve those problems, or if the proposal here will run into similar challenges.

[alexeypukhov]

A similar idea was investigated in #3896. If I'm remembering correctly we ran into some problems with a couple of our features when managing the different ALCs. It would be interesting to know if there is a way to resolve those problems, or if the proposal here will run into similar challenges.

@nrcventura Thank you for the insight. I missed that issue when I was looking through existing tickets. I will explore that further and will get back here!

[iskiselev]

#3896 tries to solve the same problem in a different way.

We have problem when OTEL and customer application has different versions of sama dependencies. That problem can be solved in two different way:

• (1) use different version of the same assemblies at the same time though separate ALC. In that case customer app works with own set of assemblies, OTEL works with different set of depdendencies. That appoach was tried in [NET] Resolve dependency conflicts by using separate ALC #3896. It is possible to solve, but would require a lot of work and may require some changes in a way OTEL is builded.
• (2) assembly unification - select highes version between OTEL dependency and customer application dependency. That is a mode used in .NET Framework (through assembly redirection) and a way of NuGet-based work (with unification in compile time by NuGet magic - unless it is broken by direct refence of lower version - which results in compile-time warning and runtime failure).

In this issue we suggest use (2) option. On .NET it can't be used directly, as TPA created at the load time - and it is not possible to load in default ALC assembly with higher version than used in TPA. To solve it, currently DOTNET_ADDITIONAL_DEPS used (and finaly by DOTNET_SHARED_STORE as a cherry on top). We suggest instead of using (or in addition) DOTNET_ADDITIONAL_DEPS / DOTNET_SHARED_STORE same can be solved by custom assembly resolver. The only trick, if assembly conflict with the assembly from TPA - we must load it in separate ALC. We would stll try to prevent application to load two versions of the same assembly - so we still use assembly unification. It is our trial to workaround inability to change TPA list.
So to solve it we can use either of mechanics:

• rewrite assembly reference by profiler, so assembly required by app would be newer than presents in TPA - TPA version will be never loaded (if older), our version would be loaded by assembly resolver in separate ALC to prevent conflict with TPA version
• if profiler is not available, entire application starting from entry point would be forced to load into new ALC, where our load method would not allow anything to be loaded in default ALC except of System.Private.CorLib, hooks and loader.

[zacharycmontoya]

SIG: Next step is to continue prototype for the profiler solution correctly loading a different version of System.Diagnostics.DiagnosticSource and then extending to all libraries. From there we can continue to outline the other scenarios we'd like to cover with such a solution.