[mono] Initial support for unloadable ALCs (#77399)

* [mono] Add LoaderAllocator type, whose instances are used to
detect whenever a collectible ALC has managed references.

* [mono] Add an implicit GC reference between objects allocated from a collectible ALC and its LoaderAllocator object.

* [mono] Add a new hash table which is similar to MonoGHashTable, but it doesn't keep the key/value objects alive by itself.

* [mono] Add a keepalive field to some reflection objects to keep the alc alive if user code
holds a reference to them. Change the reflection hashes in MonoMemoryManager to weak hashes
so the hashes themselves don't keep the alc alive.

* Fix reflection hashes.

* [mono] Optimize the case when mono_method_get_signature_checked () is called with a non-zero context and a non-generic signature.

* Free memory manager caches.

* [mono] Store static variables with GC references in collectible alcs
on the GC heap.

Normally, static variables are stored in an array inside MonoVTable
which is registered as a GC root. For collectible alcs, this would
not work, since GC references in these arrays would keep the alc alive.
Instead, store them in arrays referenced by the LoaderAllocator object.
This assumes the static variables will no longer be accessed after
the LoaderAllocator object dies.

* Add basic unload functionality.

* Fix weak hashes.

* Free MonoJitInfos belonging to unloaded memory managers.

* Avoid returning collectible types from mono_metadata_get_shared_type ().

* Add docs.

* Fix the build.

* Fix the build.

* Disable unloading for now.

Author: vargaz

docs/design/mono/unloadability.md

@@ -0,0 +1,80 @@
+
+Design
+------
+
+The design is based on docs/design/features/unloadability.md.
+
+Memory managers
+---------------
+
+Most runtime memory is owned by MonoMemoryManager structures, which are owned by one or more ALCs. For example,
+if type T1 is from ALC1, and T2 is from ALC2, then the memory related to the generic instance T1<T2> will
+be owned by the memory manager for [ALC1, ALC2]. A memory manager is collectible if one of its ALCs is collectible.
+
+LoaderAllocator objects
+-----------------------
+
+Each memory manager has a corresponding LoaderAllocator object. This object is referenced explicitly or
+implicitly by every managed object which is related to the memory manager. Until unloading starts,
+the memory manager keeps a strong reference to this object.
+
+For objects whose type is from a collectible ALC, the GC maintains an implicit reference by marking
+the LoaderAllocator object when the object is marked.
+
+Reflection objects referencing the ALC have an explicit 'keepalive' field which points to the
+corresponding LoaderAllocator object.
+
+When the LoaderAllocator object is collected by the GC, it means there are no more managed
+references to its memory allocator, so it can be unloaded.
+
+Reflection caches and static variables
+--------------------------------------
+
+Reflection objects are kept in caches inside the runtime. These caches keep strong references to the
+reflection objects. This doesn't work for collectible ALCs, since the objects keep the LoaderAllocator
+object alive. So for collectible ALCs, we use a different kind of hash table whose keys/values are
+stored in object[] arrays inside the LoaderAllocator object, and the hash table holds a weak reference
+to these arrays. This means that the reflection objects are only kept alive by the LoaderAllocator and
+by application code.
+
+Similarly, normal static variables are treated as GC roots so any static variable pointing to an object
+inside the ALC would keep the ALC alive. Instead, we store static variables inside arrays in the
+LoaderAllocator object. Normal reference type variable are allocated an entry in a pinned object[],
+and their address becomes an interior pointer to the pinned array element. For valuetypes, we
+allocate a pinned boxed object and the static variable address becomes the address of the unboxed
+object.
+
+Unloading process
+-----------------
+
+- The app calls AssemblyLoadContext.Unload ()
+- The runtime iterates over the memory managers of the ALC. Each ALC holds a strong reference to its
+LoaderAllocator object. We change the strong reference to a weak one, allowing the LoaderAllocator
+to the collected.
+- The app eventually drops all implicit and explicit references to the LoaderAllocator object.
+- The finalizer of the LoaderAllocator object calls into the runtime to free the corresponding
+memory manager.
+- When all memory managers referencing the ALC are freed, the ALC itself is freed.
+
+See the document in the Design section about the LoaderAllocatorScout object.
+
+Remaining work
+---------------
+
+- Managed frames
+Frames belonging to code in the ALC should keep the ALC alive. This can be implemented by
+having all methods allocate a volatile local variable and store a reference to their LoaderAllocator object
+into it.
+- Thread abort
+Although its not part of current .NET APIs, it might be useful to have a way to abort threads executing code
+in an ALC, similarly to how domain unloads worked previously.
+- Reflection pointers
+Icalls which take a assembly/type etc. handle as parameter need to keep the ALC alive, otherwise there will
+be subtle races.
+- Boehm GC support
+- TLS variables
+- Enable/disable compiler flag
+- Profiling, perf counters, etc. support
+- Diagnostics support, i.e. what keeps an ALC alive
+- Testing
+- Leak detection

src/mono/System.Private.CoreLib/System.Private.CoreLib.csproj

@@ -208,6 +208,7 @@
       <Compile Include="$(BclSourcesRoot)\System\Reflection\ConstructorInvoker.Mono.cs" />
       <Compile Include="$(BclSourcesRoot)\System\Reflection\CustomAttribute.cs" />
       <Compile Include="$(BclSourcesRoot)\System\Reflection\FieldInfo.Mono.cs" />
+      <Compile Include="$(BclSourcesRoot)\System\Reflection\LoaderAllocator.cs" />
       <Compile Include="$(BclSourcesRoot)\System\Reflection\MemberInfo.Mono.cs" />
       <Compile Include="$(BclSourcesRoot)\System\Reflection\MethodBase.Mono.cs" />
       <Compile Include="$(BclSourcesRoot)\System\Reflection\MethodInvoker.Mono.cs" />

src/mono/System.Private.CoreLib/src/System/Reflection/Emit/AssemblyBuilder.Mono.cs

@@ -179,6 +179,7 @@ public sealed partial class AssemblyBuilder : Assembly
         //
 #region Sync with RuntimeAssembly.cs and ReflectionAssembly in object-internals.h
         internal IntPtr _mono_assembly;
+        private LoaderAllocator? m_keepalive;
 
         private UIntPtr dynamic_assembly; /* GC-tracked */
         private ModuleBuilder[] modules;

src/mono/System.Private.CoreLib/src/System/Reflection/LoaderAllocator.cs

@@ -0,0 +1,56 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System.Runtime.InteropServices;
+using System.Runtime.CompilerServices;
+
+namespace System.Reflection
+{
+    // See src/coreclr/System.Private.CoreLib/src/System/Reflection/LoaderAllocator.cs for
+    // more comments
+    internal sealed class LoaderAllocatorScout
+    {
+        internal IntPtr m_native;
+
+        [MethodImplAttribute(MethodImplOptions.InternalCall)]
+        private static extern bool Destroy(IntPtr native);
+
+        internal LoaderAllocatorScout(IntPtr native)
+        {
+            m_native = native;
+        }
+
+        ~LoaderAllocatorScout()
+        {
+            if (!Destroy(m_native))
+            {
+                GC.ReRegisterForFinalize(this);
+            }
+        }
+    }
+
+    //
+    // This object is allocated by the runtime, every object
+    // in a collectible alc has an implicit reference to it, maintained by
+    // the GC, or an explicit reference through a field.
+    //
+    [StructLayout(LayoutKind.Sequential)]
+    internal sealed class LoaderAllocator
+    {
+#region Sync with MonoManagedLoaderAllocator in object-internals.h
+#pragma warning disable CA1823, 414, 169
+        private LoaderAllocatorScout m_scout;
+        // These point to objects created by the runtime which are kept
+        // alive by this LoaderAllocator
+        private object[]? m_slots;
+        private object[]? m_hashes;
+        private int m_nslots;
+#pragma warning restore CA1823, 414, 169
+#endregion
+
+        private LoaderAllocator(IntPtr native)
+        {
+            m_scout = new LoaderAllocatorScout(native);
+        }
+    }
+}

src/mono/System.Private.CoreLib/src/System/Reflection/RuntimeAssembly.cs

@@ -64,9 +64,11 @@ public unsafe UnmanagedMemoryStreamForModule(byte* pointer, long length, Module
 
         //
         // KEEP IN SYNC WITH _MonoReflectionAssembly in /mono/mono/metadata/object-internals.h
+        // and AssemblyBuilder.cs.
         //
         #region VM dependency
         private IntPtr _mono_assembly;
+        private LoaderAllocator? m_keepalive;
         #endregion
 
         internal IntPtr GetUnderlyingNativeHandle() { return _mono_assembly; }

src/mono/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.Mono.cs

@@ -24,12 +24,20 @@ internal IntPtr NativeALC
         [DynamicDependency(nameof(_nativeAssemblyLoadContext))]
         private IntPtr InitializeAssemblyLoadContext(IntPtr thisHandlePtr, bool representsTPALoadContext, bool isCollectible)
         {
+            if (isCollectible)
+                KeepLoaderAllocator();
             using (SafeStringMarshal handle = RuntimeMarshal.MarshalString(Name))
             {
                 return InternalInitializeNativeALC(thisHandlePtr, handle.Value, representsTPALoadContext, isCollectible);
             }
         }
 
+        // Keep the type alive since instances are created by the runtime
+        [DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(LoaderAllocator))]
+        private static void KeepLoaderAllocator()
+        {
+        }
+
         [MethodImplAttribute (MethodImplOptions.InternalCall)]
         private static extern void PrepareForAssemblyLoadContextRelease (IntPtr nativeAssemblyLoadContext, IntPtr assemblyLoadContextStrong);
 

src/mono/System.Private.CoreLib/src/System/Type.Mono.cs

@@ -4,15 +4,18 @@
 using System.Reflection;
 using System.Diagnostics.CodeAnalysis;
 using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
 using System.Runtime.Versioning;
 using System.Threading;
 
 namespace System
 {
+    [StructLayout(LayoutKind.Sequential)]
     public partial class Type
     {
         #region keep in sync with object-internals.h
         internal RuntimeTypeHandle _impl;
+        internal LoaderAllocator? m_keepalive;
         #endregion
 
         internal IntPtr GetUnderlyingNativeHandle()

src/mono/mono/metadata/CMakeLists.txt

@@ -107,6 +107,8 @@ set(metadata_common_sources
     mono-hash.h
     mono-conc-hash.c
     mono-conc-hash.h
+    weak-hash.c
+    weak-hash.h
     mono-ptr-array.h
     monitor.h
     object.c

src/mono/mono/metadata/assembly-load-context.c

@@ -38,17 +38,26 @@ static void
 mono_alc_init (MonoAssemblyLoadContext *alc, gboolean collectible)
 {
 	MonoLoadedImages *li = g_new0 (MonoLoadedImages, 1);
+
+	// FIXME:
+	collectible = FALSE;
+
 	mono_loaded_images_init (li, alc);
 	alc->loaded_images = li;
 	alc->loaded_assemblies = NULL;
-	alc->memory_manager = mono_mem_manager_new (&alc, 1, collectible);
 	alc->generic_memory_managers = g_ptr_array_new ();
 	mono_coop_mutex_init (&alc->memory_managers_lock);
 	alc->unloading = FALSE;
 	alc->collectible = collectible;
 	alc->pinvoke_scopes = g_hash_table_new_full (g_str_hash, g_str_equal, g_free, NULL);
 	mono_coop_mutex_init (&alc->assemblies_lock);
 	mono_coop_mutex_init (&alc->pinvoke_lock);
+
+	alc->memory_manager = mono_mem_manager_new (&alc, 1, collectible);
+
+	if (collectible)
+		/* Eagerly create the loader alloc object for the main memory manager */
+		mono_mem_manager_get_loader_alloc (alc->memory_manager);
 }
 
 static MonoAssemblyLoadContext *
@@ -98,6 +107,7 @@ mono_alc_cleanup_assemblies (MonoAssemblyLoadContext *alc)
 	// The minimum refcount on assemblies is 2: one for the domain and one for the ALC.
 	// The domain refcount might be less than optimal on netcore, but its removal is too likely to cause issues for now.
 	GSList *tmp;
+	const char *name = alc->name ? alc->name : "<default>";
 
 	// Remove the assemblies from loaded_assemblies
 	for (tmp = alc->loaded_assemblies; tmp; tmp = tmp->next) {
@@ -107,8 +117,8 @@ mono_alc_cleanup_assemblies (MonoAssemblyLoadContext *alc)
 		loaded_assemblies = g_slist_remove (loaded_assemblies, assembly);
 		alcs_unlock ();
 
-		mono_assembly_decref (assembly);
-		mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Unloading ALC [%p], removing assembly %s[%p] from loaded_assemblies, ref_count=%d\n", alc, assembly->aname.name, assembly, assembly->ref_count);
+		//mono_assembly_decref (assembly);
+		mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Unloading ALC '%s'[%p], removing assembly %s[%p] from loaded_assemblies, ref_count=%d\n", name, alc, assembly->aname.name, assembly, assembly->ref_count);
 	}
 
 	// Release the GC roots
@@ -122,7 +132,7 @@ mono_alc_cleanup_assemblies (MonoAssemblyLoadContext *alc)
 		MonoAssembly *assembly = (MonoAssembly *)tmp->data;
 		if (!assembly->image || !image_is_dynamic (assembly->image))
 			continue;
-		mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Unloading ALC [%p], dynamic assembly %s[%p], ref_count=%d", alc, assembly->aname.name, assembly, assembly->ref_count);
+		mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Unloading ALC '%s'[%p], dynamic assembly %s[%p], ref_count=%d", name, alc, assembly->aname.name, assembly, assembly->ref_count);
 		if (!mono_assembly_close_except_image_pools (assembly))
 			tmp->data = NULL;
 	}
@@ -134,7 +144,7 @@ mono_alc_cleanup_assemblies (MonoAssemblyLoadContext *alc)
 			continue;
 		if (!assembly->image || image_is_dynamic (assembly->image))
 			continue;
-		mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Unloading ALC [%p], non-dynamic assembly %s[%p], ref_count=%d", alc, assembly->aname.name, assembly, assembly->ref_count);
+		mono_trace (G_LOG_LEVEL_DEBUG, MONO_TRACE_ASSEMBLY, "Unloading ALC '%s'[%p], non-dynamic assembly %s[%p], ref_count=%d", name, alc, assembly->aname.name, assembly, assembly->ref_count);
 		if (!mono_assembly_close_except_image_pools (assembly))
 			tmp->data = NULL;
 	}
@@ -173,15 +183,21 @@ mono_alc_cleanup (MonoAssemblyLoadContext *alc)
 
 	mono_alc_cleanup_assemblies (alc);
 
-	mono_mem_manager_free (alc->memory_manager, FALSE);
+	// FIXME: Do it for every memory manager
+	mono_gchandle_free_internal (alc->memory_manager->loader_allocator_handle);
+	alc->memory_manager->loader_allocator_handle = NULL;
+
+	// FIXME: Change to FALSE
+	mono_mem_manager_free (alc->memory_manager, TRUE);
 	alc->memory_manager = NULL;
 
 	/*for (int i = 0; i < alc->generic_memory_managers->len; i++) {
 		MonoGenericMemoryManager *memory_manager = (MonoGenericMemoryManager *)alc->generic_memory_managers->pdata [i];
 		mono_mem_manager_free_generic (memory_manager, FALSE);
 	}*/
-	g_ptr_array_free (alc->generic_memory_managers, TRUE);
-	mono_coop_mutex_destroy (&alc->memory_managers_lock);
+	// FIXME:
+	//g_ptr_array_free (alc->generic_memory_managers, TRUE);
+	//mono_coop_mutex_destroy (&alc->memory_managers_lock);
 
 	mono_gchandle_free_internal (alc->gchandle);
 	alc->gchandle = NULL;
@@ -260,6 +276,10 @@ ves_icall_System_Runtime_Loader_AssemblyLoadContext_PrepareForAssemblyLoadContex
 	MonoGCHandle strong_gchandle = (MonoGCHandle)strong_gchandle_ptr;
 	MonoAssemblyLoadContext *alc = (MonoAssemblyLoadContext *)alc_pointer;
 
+	// FIXME:
+	if (!alc->collectible)
+		return;
+
 	g_assert (alc->collectible);
 	g_assert (!alc->unloading);
 	g_assert (alc->gchandle);
@@ -270,6 +290,12 @@ ves_icall_System_Runtime_Loader_AssemblyLoadContext_PrepareForAssemblyLoadContex
 	MonoGCHandle weak_gchandle = alc->gchandle;
 	alc->gchandle = strong_gchandle;
 	mono_gchandle_free_internal (weak_gchandle);
+
+	mono_mem_manager_start_unload (alc->memory_manager);
+	mono_alc_memory_managers_lock (alc);
+	for (guint i = 0; i < alc->generic_memory_managers->len; i++)
+		mono_mem_manager_start_unload ((MonoMemoryManager *)g_ptr_array_index (alc->generic_memory_managers, i));
+	mono_alc_memory_managers_unlock (alc);
 }
 
 gpointer
@@ -655,3 +681,44 @@ mono_alc_get_all_loaded_assemblies (void)
 	alcs_unlock ();
 	return assemblies;
 }
+
+
+MonoBoolean
+ves_icall_System_Reflection_LoaderAllocatorScout_Destroy (gpointer native)
+{
+	/* Not enabled yet */
+	return FALSE;
+
+#if 0
+	MonoMemoryManager *mem_manager = (MonoMemoryManager *)native;
+	MonoAssemblyLoadContext *alc;
+
+	MonoGCHandle loader_handle = mem_manager->loader_allocator_weak_handle;
+	if (mono_gchandle_get_target_internal (loader_handle))
+		return FALSE;
+
+	/*
+	 * The weak handle is NULL, meaning the managed LoaderAllocator object is dead, we can
+	 * free the native side.
+	 */
+	for (int i = 0; i < mem_manager->n_alcs; ++i) {
+		alc = mem_manager->alcs [i];
+		mono_alc_memory_managers_lock (alc);
+		g_ptr_array_remove (alc->generic_memory_managers, mem_manager);
+		mono_alc_memory_managers_unlock (alc);
+	}
+
+	alc = mem_manager->alcs [0];
+
+	// FIXME: Generic memory managers might need to be destroyed in a specific order/together,
+	// hold ref counts on alcs etc.
+	if (mem_manager == alc->memory_manager) {
+		mono_alc_cleanup (alc);
+	} else {
+		// FIXME: Use debug_unload=FALSE
+		mono_mem_manager_free (mem_manager, TRUE);
+	}
+
+	return TRUE;
+#endif
+}

src/mono/mono/metadata/class-internals.h

@@ -155,7 +155,8 @@ struct _MonoClassField {
 	 * field, it's the offset from the start of the object, if
 	 * it's static, it's from the start of the memory chunk
 	 * allocated for statics for the class.
-	 * For special static fields, this is set to -1 during vtable construction.
+	 * -1 means its a special static field.
+	 * -2 means its a collectible static field.
 	 */
 	int              offset;
 };
@@ -360,6 +361,7 @@ struct MonoVTable {
 	MonoDomain *domain;  /* each object/vtable belongs to exactly one domain */
 	gpointer    type; /* System.Type type for klass */
 	guint8     *interface_bitmap;
+	MonoGCHandle loader_alloc; /* LoaderAllocator object for objects in collectible alcs */
 	guint32     max_interface_id;
 	guint8      rank;
 	/* Keep this a guint8, the jit depends on it */

src/mono/mono/metadata/icall-decl.h

@@ -208,4 +208,6 @@ ICALL_EXPORT MonoBoolean ves_icall_System_Array_IsValueOfElementTypeInternal (Mo
 
 ICALL_EXPORT MonoBoolean ves_icall_System_Array_FastCopy (MonoObjectHandleOnStack source_handle, int source_idx, MonoObjectHandleOnStack dest_handle, int dest_idx, int length);
 
+ICALL_EXPORT MonoBoolean ves_icall_System_Reflection_LoaderAllocatorScout_Destroy (gpointer native);
+
 #endif // __MONO_METADATA_ICALL_DECL_H__