[mono][loader] Calling assembly resolve while holding the loader lock can lead to deadlocks

[lambdageek]

The original issue is mono/mono#21032 Adding a copy here to make sure we do this work for .NET 6, too. The assembly loading algorithm is different, but the same situation is possible

---

Sample stack trace from VS Mac https://gist.github.com/Therzok/df71c056b0dabc8a2f3ffb6f1c264488

Thread 8 calls mono_class_create_from_typedef which then needs to get the parent type and calls mono_class_get_checked... etc... until we need to load an assembly, and when we can't find it, we call the managed assembly resolver event from mono_domain_assembly_postload_search.

The managed code tries to lock some managed data structure, but the monitor is already locked, so it waits.

Meanwhile other threads that want to do any type creation are deadlocked because they can't take the loader lock.

Tagging subscribers to this area: @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The original issue is mono/mono#21032 Adding a copy here to make sure we do this work for .NET 6, too. The assembly loading algorithm is different, but the same situation is possible

---

Sample stack trace from VS Mac https://gist.github.com/Therzok/df71c056b0dabc8a2f3ffb6f1c264488

Thread 8 calls mono_class_create_from_typedef which then needs to get the parent type and calls mono_class_get_checked... etc... until we need to load an assembly, and when we can't find it, we call the managed assembly resolver event from mono_domain_assembly_postload_search.

The managed code tries to lock some managed data structure, but the monitor is already locked, so it waits.

Meanwhile other threads that want to do any type creation are deadlocked because they can't take the loader lock.

Author:	lambdageek
Assignees:	lambdageek
Labels:	area-VM-meta-mono
Milestone:	6.0.0

[lambdageek]

Repro https://github.com/lambdageek/repro-gh-21032

(to run with mono on dotnet/runtime, need to cd X && dotnet publish --self-contained -r osx-x64 and then replace libcoreclr and System.Private.CoreLib by ones from a build of mono from dotnet/runtime)

[lambdageek]

This is hard mono/mono#21032 (comment)

It's essentially equivalent to having class loader levels - in particular being able to talk, outside of the loader lock, about a class without its parent being set.

[lambdageek]

One other, more targeted approach that is worth considering is to have some way for class creating and class initialization to return a "I need to load an assembly, please load it and retry" value. That means that instead of always just calling the loader and failing the class (setting MonoClass:has_failure) if it doesn't succeed, we have a flag on a class that says "I depend on something", and checking for that in many places in class loading.