Could Assembly.Load cause deadlocks?

[kuskmen]

Hi,

We have a high throughput application that generates thousands of tasks per second doing some processing on aerospike records in parallel. It's all good when the load is relatively small, but when we want to activate 250k records in parallel (we have Semaphore which limits each batch at 1000) we start experiencing deadlock scenarios. I've created a dump file and I observed that 85.27% (according to Visual Studio analyzer) of the current tasks are all stuck with this call stack:

 	System.Private.CoreLib.dll!System.Reflection.Assembly.Load(System.Reflection.AssemblyName assemblyRef) Line 53	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.ObjectReader.ResolveSimpleAssemblyName(System.Reflection.AssemblyName assemblyName) Line 871	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.ObjectReader.FastBindToType(string assemblyName, string typeName) Line 830	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, string name) Line 924	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.ObjectMap.ObjectMap(string objectName, string[] memberNames, System.Runtime.Serialization.Formatters.Binary.BinaryTypeEnum[] binaryTypeEnumA, object[] typeInformationA, int[] memberAssemIds, System.Runtime.Serialization.Formatters.Binary.ObjectReader objectReader, int objectId, System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.Runtime.Serialization.Formatters.Binary.SizedArray assemIdToAssemblyTable) Line 64	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.BinaryParser.ReadObjectWithMapTyped(System.Runtime.Serialization.Formatters.Binary.BinaryObjectWithMapTyped record) Line 558	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.BinaryParser.Run() Line 117	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(System.Runtime.Serialization.Formatters.Binary.BinaryParser serParser) Line 129	C#
 	System.Runtime.Serialization.Formatters.dll!System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(System.IO.Stream serializationStream) Line 149	C#
>	AerospikeClient.dll!Aerospike.Client.ByteUtil.BytesToObject(byte[] buf, int offset, int len) Line 147	C#
 	AerospikeClient.dll!Aerospike.Client.Unpacker.UnpackBlob(int count) Line 191	C#
 	AerospikeClient.dll!Aerospike.Client.Unpacker.UnpackObject() Line 282	C#
 	AerospikeClient.dll!Aerospike.Client.Unpacker.UnpackList(int count) Line 64	C#
 	AerospikeClient.dll!Aerospike.Client.ByteUtil.BytesToParticle(int type, byte[] buf, int offset, int len) Line 58	C#
 	AerospikeClient.dll!Aerospike.Client.AsyncRead.ParseRecord(int opCount, int fieldCount, int generation, int expiration) Line 158	C#
 	AerospikeClient.dll!Aerospike.Client.AsyncRead.ParseResult() Line 86	C#
 	AerospikeClient.dll!Aerospike.Client.AsyncSingleCommand.ParseCommand() Line 18	C#
 	AerospikeClient.dll!Aerospike.Client.AsyncCommand.ReceiveEvent() Line 491	C#
 	AerospikeClient.dll!Aerospike.Client.AsyncCommand.ReceiveEvent() Line 464	C#
 	AerospikeClient.dll!Aerospike.Client.AsyncCommand.EventHandlers.HandleSocketEvent(object sender, System.Net.Sockets.SocketAsyncEventArgs args) Line 40	C#
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 183	C#
 	System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() Line 261	C#
 	System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 729	C#
 	System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 63	C#

I know it sounds bizarre but I think because the load that we generate is not gradual but rather 1k at once, it might relate to Assembly.Load deadlocking so I wanted to contact you guys if you know of such a possibility?

[vitek-karas]

Anything is possible, but if so, this would be the first time I heard about it.
Adding @elinor-fung @jkotas @janvorli if they remember something like this anywhere else.

To diagnose this, it would be interesting to look at the native call stacks and see where exactly are the threads blocked. Or if it's possible to share the dump? If you can share the dump, it's probably better to do so via https://developercommunity.visualstudio.com/dotnet/report?ftype=problem.

[Xyncgas]

Yes, Assembly.Load can cause deadlocks in certain scenarios.

Assembly.Load is a method in the .NET Framework that loads an assembly into the current application domain. It can be called with a string that specifies the name of the assembly to load or with a byte array that contains the binary representation of the assembly.

Deadlocks occur when two or more threads are blocked, waiting for each other to release resources. Deadlocks can occur when calling Assembly.Load if there are multiple threads loading assemblies simultaneously, and the assemblies have dependencies on each other.

For example, suppose thread A is loading assembly X, which has a dependency on assembly Y. At the same time, thread B is loading assembly Y, which has a dependency on assembly X. If both threads are blocked waiting for the other to release the resources, a deadlock occurs, and the program becomes unresponsive.

To avoid deadlocks when loading assemblies, you can use the AppDomain.AssemblyResolve event to load assemblies on demand and avoid loading them simultaneously. You can also use locking mechanisms to prevent multiple threads from loading assemblies simultaneously.

It is important to note that deadlocks can occur in any multithreaded application, not just when using Assembly.Load. It is essential to design and test the application carefully to avoid deadlocks and ensure that it is thread-safe.

[KeterSCP]

Is this ChatGPT generated answer?