Crash on reentry of tiered compilation

[angelowang]

Description

When our product (Autodesk Civil 3D) turns on tiered compilation and open some specific file, it will crash when the JIT is triggered in a reentry way. Let me try to describe it clearly.

• There are two managed DLLs involved, AeccDbMgd.dll and AecBaseMgd.dll. AeccDbMgd.dll uses AecBaseMgd.dll.
• The process will not load AecBaseMgd on start up, but on demand.
• During start up, some native 'observer' code will try to construct a .NET object wrapper in AeccDbMgd.dll, which triggers the first JIT.
• Due to the dependency between the two dlls, AecBaseMgd.dll will be loaded.
• AutoCAD has some code that fires some events during loading of AecBaseMgd.dll, eventually triggers similar 'observer' code as above, and then the second time JIT.

Callstack is something like below:

Exception thrown at 0x00007FF88B0BB699 in acad.exe: Microsoft C++ exception: EETypeLoadException at memory location 0x00000070FDFEA860.
Some lines are omitted/obfuscated for confidentiality.

 	KernelBase.dll!00007ff88b0bb699()	Unknown
 	clrjit.dll!Compiler::impImportBlockCode(BasicBlock * block=0x000001d48ac788a0) Line 8397	C++
 	clrjit.dll!Compiler::impImportBlock(BasicBlock * block=0x000001d48ac788a0) Line 11237	C++
 	clrjit.dll!Compiler::impImport() Line 12187	C++
 	clrjit.dll!Compiler::fgImport() Line 580	C++
 	[Inline Frame] clrjit.dll!Phase::Run() Line 61	C++
 	[Inline Frame] clrjit.dll!DoPhase(Compiler *) Line 136	C++
 	clrjit.dll!Compiler::compCompile(void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0) Line 4542	C++
 	clrjit.dll!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd=0x00000070fdfef1d0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdfef0c0, void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0) Line 7150	C++
 	clrjit.dll!Compiler::compCompile(CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0) Line 6292	C++
 	clrjit.dll!jitNativeCode(CORINFO_METHOD_STRUCT_ * methodHnd=0x00007fff45131a08, CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, ICorJitInfo * compHnd=0x00000070fdfef1d0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdfef0c0, void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0, void * inlineInfoPtr) Line 7783	C++
 	clrjit.dll!CILJit::compileMethod(ICorJitInfo * compHnd=0x00000070fdfef1d0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdfef0c0, unsigned int flags=2147483652, unsigned char * * entryAddress=0x00000070fdfeee68, unsigned int * nativeSizeOfCode=0x00000070fdfeee58) Line 299	C++
 	[Managed to Native Transition]	
 	[Native to Managed Transition]	
 	KernelBase.dll!00007ff88b0bb699()	Unknown
 	clrjit.dll!Compiler::impImportBlockCode(BasicBlock * block=0x000001d48ac788a0) Line 8397	C++
 	clrjit.dll!Compiler::impImportBlock(BasicBlock * block=0x000001d48ac788a0) Line 11237	C++
 	clrjit.dll!Compiler::impImport() Line 12187	C++
 	clrjit.dll!Compiler::fgImport() Line 580	C++
 	[Inline Frame] clrjit.dll!Phase::Run() Line 61	C++
 	[Inline Frame] clrjit.dll!DoPhase(Compiler *) Line 136	C++
 	clrjit.dll!Compiler::compCompile(void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0) Line 4542	C++
 	clrjit.dll!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd=0x00000070fdfef1d0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdfef0c0, void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0) Line 7150	C++
 	clrjit.dll!Compiler::compCompile(CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0) Line 6292	C++
 	clrjit.dll!jitNativeCode(CORINFO_METHOD_STRUCT_ * methodHnd=0x00007fff45131a08, CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, ICorJitInfo * compHnd=0x00000070fdfef1d0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdfef0c0, void * * methodCodePtr=0x00000070fdfeeda0, unsigned int * methodCodeSize=0x00000070fdfeee58, JitFlags * compileFlags=0x00000070fdfeedc0, void * inlineInfoPtr) Line 7783	C++
 	clrjit.dll!CILJit::compileMethod(ICorJitInfo * compHnd=0x00000070fdfef1d0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdfef0c0, unsigned int flags=2147483652, unsigned char * * entryAddress=0x00000070fdfeee68, unsigned int * nativeSizeOfCode=0x00000070fdfeee58) Line 299	C++
 	[Managed to Native Transition]	
 	acdbmgd.dll!<<<<<< Create .NET wrapper object 2 via C++/CLI >>>>>>
 	[Native to Managed Transition]	
 	acdb25.dll!<<<<<< Reactors fired on object 2 modification >>>>>>
        acdb25.dll!<<<<<< Object 2 is modified >>>>>>
 	acdb25.dll!<<<<<< Load AecBaseMgd.dll during JIT 1st time >>>>>>
 	[Managed to Native Transition]	
 	System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(string assemblyPath) Line 347	C#
 	System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(System.Reflection.AssemblyName assemblyName = {System.Reflection.AssemblyName}) Line 632	C#
 	System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName assemblyName) Line 698	C#
 	System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(nint gchManagedAssemblyLoadContext, System.Reflection.AssemblyName assemblyName) Line 134	C#
 	[Native to Managed Transition]	
 	clrjit.dll!Compiler::impImportBlockCode(BasicBlock * block=0x000001d495cf2a90) Line 8397	C++
 	clrjit.dll!Compiler::impImportBlock(BasicBlock * block=0x000001d495cf2a90) Line 11237	C++
 	clrjit.dll!Compiler::impImport() Line 12187	C++
 	clrjit.dll!Compiler::fgImport() Line 580	C++
 	[Inline Frame] clrjit.dll!Phase::Run() Line 61	C++
 	[Inline Frame] clrjit.dll!DoPhase(Compiler *) Line 136	C++
 	clrjit.dll!Compiler::compCompile(void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0) Line 4542	C++
 	clrjit.dll!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd=0x00000070fdffc7e0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdffc6d0, void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0) Line 7150	C++
 	clrjit.dll!Compiler::compCompile(CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0) Line 6292	C++
 	clrjit.dll!jitNativeCode(CORINFO_METHOD_STRUCT_ * methodHnd=0x00007fff451354a8, CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, ICorJitInfo * compHnd=0x00000070fdffc7e0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdffc6d0, void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0, void * inlineInfoPtr) Line 7783	C++
 	clrjit.dll!CILJit::compileMethod(ICorJitInfo * compHnd=0x00000070fdffc7e0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdffc6d0, unsigned int flags=2147483652, unsigned char * * entryAddress=0x00000070fdffc478, unsigned int * nativeSizeOfCode=0x00000070fdffc468) Line 299	C++
 	[Managed to Native Transition]	
 	[Native to Managed Transition]	
 	clrjit.dll!Compiler::impImportBlockCode(BasicBlock * block=0x000001d495cf2a90) Line 8397	C++
 	clrjit.dll!Compiler::impImportBlock(BasicBlock * block=0x000001d495cf2a90) Line 11237	C++
 	clrjit.dll!Compiler::impImport() Line 12187	C++
 	clrjit.dll!Compiler::fgImport() Line 580	C++
 	[Inline Frame] clrjit.dll!Phase::Run() Line 61	C++
 	[Inline Frame] clrjit.dll!DoPhase(Compiler *) Line 136	C++
 	clrjit.dll!Compiler::compCompile(void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0) Line 4542	C++
 	clrjit.dll!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd=0x00000070fdffc7e0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdffc6d0, void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0) Line 7150	C++
 	clrjit.dll!Compiler::compCompile(CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0) Line 6292	C++
 	clrjit.dll!jitNativeCode(CORINFO_METHOD_STRUCT_ * methodHnd=0x00007fff451354a8, CORINFO_MODULE_STRUCT_ * classPtr=0x00007fff45054c98, ICorJitInfo * compHnd=0x00000070fdffc7e0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdffc6d0, void * * methodCodePtr=0x00000070fdffc3b0, unsigned int * methodCodeSize=0x00000070fdffc468, JitFlags * compileFlags=0x00000070fdffc3d0, void * inlineInfoPtr) Line 7783	C++
 	clrjit.dll!CILJit::compileMethod(ICorJitInfo * compHnd=0x00000070fdffc7e0, CORINFO_METHOD_INFO * methodInfo=0x00000070fdffc6d0, unsigned int flags=2147483652, unsigned char * * entryAddress=0x00000070fdffc478, unsigned int * nativeSizeOfCode=0x00000070fdffc468) Line 299	C++
 	[Managed to Native Transition]	
 	acdbmgd.dll!<<<<<< Create .NET wrapper object 1 via C++/CLI >>>>>>
 	[Native to Managed Transition]	
 	acdb25.dll!<<<<<< Reactors fired on object 1 modification >>>>>>
        acdb25.dll!<<<<<< Object 1 is modified during file open >>>>>>
 	acad.exe!<<<<<<< Open drawing >>>>>>

Reproduction Steps

It's quite hard for me to have such a sample.

Expected behavior

No crash.

Actual behavior

Crash.

Regression?

No idea.

Known Workarounds

Pre-load the mgd dll.

Configuration

.NET 8.0.100
Windows
x64

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

Can you share the dump with us?

[angelowang]

errorreport_778684275.zip
This is what our crash system captures. It contains a minidump file.

[AndyAyersMS]

A few other questions:

• Is this a regression from an earlier .NET version?
• Are you able to reproduce the problem?
• Does the problem go away if tiered compilation is disabled?

[angelowang]

Is this a regression from an earlier .NET version?

We don't know. Civil 3D has just ported to .NET last year, and .NET 8 is the first version.

Are you able to reproduce the problem?

Yes, it constantly crashes.

Does the problem go away if tiered compilation is disabled?

Yes, it goes away.

[AndyAyersMS]

From the files you shared above it looks like there is an assembly load notification that goes into native code and then calls back into managed which triggers more jitting, perhaps in the assembly that was just loaded (I can't tell).

It is not clear how tiering plays a role here. The error is on a foreground thread and the TC thread is apparently idle. We will likely need a more detailed dump to figure this out. Can you enable a full (or at least heap) dump and share that?

Also can you capture what happens with DOTNET_JitDisasmSummary=1?

FYI @mangod9 possibly a loader issue?

[angelowang]

@AndyAyersMS Sent you a OneDrive link via your MS email.

perhaps in the assembly that was just loaded

Yes. There are only two DLLs.
AeccDbMgd.dll contains Alignment and Corridor. They both inherit from Entity which is defined in AecBaseMgd.dll.
The two JITs happen when constructing Alignment and Corridor respectively.
Constructing Alignment triggers loading of AecBaseMgd.dll and this loading triggers construction of Corridor. 🤣

[AndyAyersMS]

Thanks -- I will try to look at this soon.

[AndyAyersMS]

Yeah I think somebody on the runtime or maybe interop team should take this one over. As you say, it looks like:

1. There's a request construct an Alignment object
2. Which triggers type load for Entity
3. Which triggers a custom assembly resolver
4. Which triggers an assembly load notification for AecBaseMgd.dll
5. Which triggers your assembly load event handler
6. Which triggers construction of Corridor (via COM\RCW)
7. Which attempts a type load for Entity
8. Which fails.

I suspect tiering's main role is to alter the order of class loading / initialization.

@mangod9 I am going to reassign this one to runtime.