dotnet.exe hangs on exit due to icu.dll

[joncham]

Description

I am seeing an intermittent hang in dotnet.exe when exiting. The icu.dll is attempting to acquire a lock and block indefinitely. Callstack of hung process is:

>	ntdll.dll!NtWaitForAlertByThreadId()	Unknown
 	ntdll.dll!RtlpRunOnceWaitForInit()	Unknown
 	ntdll.dll!RtlRunOnceExecuteOnce()	Unknown
 	KernelBase.dll!InitOnceExecuteOnce()	Unknown
 	icu.dll!std::call_once<void (&)(void)>(struct std::once_flag &,void (&)(void))	Unknown
 	icu.dll!icu::UMutex::getMutex(void)	Unknown
 	icu.dll!icu::UMutex::lock(void)	Unknown
 	icu.dll!u_cleanup()	Unknown
 	ucrtbase.dll!<lambda>(void)()	Unknown
 	ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda>(void) &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c>>()	Unknown
 	ucrtbase.dll!_execute_onexit_table()	Unknown
 	ucrtbase.dll!__crt_state_management::wrapped_invoke<int (*)(int *),int *,int>(int (*)(int *),int *)	Unknown
 	icu.dll!dllmain_crt_process_detach()	Unknown
 	icu.dll!dllmain_dispatch()	Unknown
 	ntdll.dll!LdrpCallInitRoutine()	Unknown
 	ntdll.dll!LdrShutdownProcess()	Unknown
 	ntdll.dll!RtlExitUserProcess()	Unknown
 	kernel32.dll!ExitProcessImplementation()	Unknown
 	ucrtbase.dll!exit_or_terminate_process()	Unknown
 	ucrtbase.dll!common_exit()	Unknown
 	dotnet.exe!__scrt_common_main_seh() Line 295	C++
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Only a single thread (main thread) is executing at this point in the process.

Reproduction Steps

No reproduction. I can produce this with a stress test under our CI environment and cause it to hang after a few hours.

Expected behavior

Process exits without hang.

Actual behavior

Process never exits.

Regression?

No response

Known Workarounds

No response

Configuration

.NET 7.0.102
Windows 10 Pro (10.0.19045)
x64

Other information

No response

[ericstj]

cc @tarekgh @jefgen

[TautvydasZilys]

From what we can tell, this is a bug in the icu.dll library that ships with the OS. icu.dll!u_cleanup accesses icu.dll!initFlag which is a global variable and then kernelbase.dll!InitOnceExecuteOnce thinks that it's locked. Given the callstack, I believe global variables might have already started being destroyed at that point. For comparison, the same code gets executed through DllMain instead on the Windows 11 version of the library:

icu.dll!icu_internal::UMutex::getMutex()
icu.dll!uprv_u_cleanup()
icu.dll!HandleIcuCleanUp()
icu.dll!DllMain()
icu.dll!dllmain_dispatch()
ntdll.dll!LdrpCallInitRoutine()
ntdll.dll!LdrShutdownProcess()
ntdll.dll!RtlExitUserProcess()
kernel32.dll!ExitProcessImplementation()
ucrtbased.dll!exit_or_terminate_process(const unsigned int return_code) Line 138	C++
ucrtbased.dll!common_exit(const int return_code, const _crt_exit_cleanup_mode cleanup_mode, const _crt_exit_return_mode return_mode) Line 274	C++
ucrtbased.dll!exit(int return_code) Line 288	C++
Project2.exe!__scrt_common_main_seh() Line 297	C++
Project2.exe!__scrt_common_main() Line 331	C++
Project2.exe!mainCRTStartup(void * __formal) Line 17	C++
kernel32.dll!BaseThreadInitThunk()	Unknown
ntdll.dll!RtlUserThreadStart()	Unknown

We have not confirmed whether this is reproducible on Windows 11 since our CI runs on Windows 10. Jonathan didn't mention this, but we have a full heap dump file that we could privately share if needed.

[jefgen]

cc @rp9-next

[tarekgh]

@rp9-next we may need to open a bug in Windows DB for ICU. Looks like this is a problem during the process shutdown. Do you know which ICU version installed on Windows 10 Pro (10.0.19045)?

@TautvydasZilys

but we have a full heap dump file that we could privately share if needed.

This will be very helpful. Is there a way you can run the CI on Windows 11 too just to see if the issue still there?

[jefgen]

I don't work on ICU anymore, but just to help out a bit...

For Windows 10 Pro (10.0.19045), it has ICU version 64.2 included in the OS.

Note that there are some changes made to the clean up functions for the OS version of the DLL, which aren't included upstream. See the comment here for details: https://github.com/microsoft/icu/blob/maint/maint-64/icu/icu4c/source/common/ucln_cmn.cpp#L66-L89

So this could be a bug in that patch perhaps. If you have a full heap dump file that would help. I think @rp9-next might need to take a look at the heap dump to investigate and see if this is a bug in those changes, or a bug in upstream ICU itself. There were also some additional changes made in this clean up code path in later versions. If you don't see this issue on Windows 11, then they might want to be backported.

Hope this helps.

[TautvydasZilys]

Note that there are some changes made to the clean up functions for the OS version of the DLL, which aren't included upstream. See the comment here for details: https://github.com/microsoft/icu/blob/maint/maint-64/icu/icu4c/source/common/ucln_cmn.cpp#L66-L89

From what I can tell, the DLL in Windows 10 does not have this patch:

u_cleanup:
  0000000180013130: mov         qword ptr [rsp+8],rbx
  0000000180013135: push        rdi
  0000000180013136: sub         rsp,20h
  000000018001313A: xor         ecx,ecx
  000000018001313C: call        umtx_lock
  0000000180013141: inc         dword ptr [?cleanup_count@@3HA]
  0000000180013147: lea         rcx,[?globalMutex@?A0x2c72d8bd@icu@@3VUMutex@2@A]
  000000018001314E: call        ?unlock@UMutex@icu@@QEAAXXZ
  0000000180013153: lea         rbx,[1802146F0h]
  000000018001315A: mov         edi,8
  000000018001315F: mov         rax,qword ptr [rbx]
  0000000180013162: test        rax,rax
  0000000180013165: je          0000000180013171
  0000000180013167: call        qword ptr [__guard_dispatch_icall_fptr]
  000000018001316D: and         qword ptr [rbx],0
  0000000180013171: add         rbx,8
  0000000180013175: sub         rdi,1
  0000000180013179: jne         000000018001315F
  000000018001317B: lea         rbx,[180214630h]
  0000000180013182: mov         edi,18h
  0000000180013187: mov         rax,qword ptr [rbx]
  000000018001318A: test        rax,rax
  000000018001318D: je          0000000180013199
  000000018001318F: call        qword ptr [__guard_dispatch_icall_fptr]
  0000000180013195: and         qword ptr [rbx],0
  0000000180013199: add         rbx,8
  000000018001319D: sub         rdi,1
  00000001800131A1: jne         0000000180013187
  00000001800131A3: and         qword ptr [180214740h],rdi
  00000001800131AA: and         qword ptr [180214730h],rdi
  00000001800131B1: and         qword ptr [180214738h],rdi
  00000001800131B8: and         qword ptr [180214748h],rdi
  00000001800131BF: and         qword ptr [1802145F8h],rdi
  00000001800131C6: and         qword ptr [180214600h],rdi
  00000001800131CD: and         qword ptr [1802145F0h],rdi
  00000001800131D4: or          dword ptr [180214610h],0FFFFFFFFh
  00000001800131DB: and         qword ptr [180214608h],rdi
  00000001800131E2: mov         rbx,qword ptr [rsp+30h]
  00000001800131E7: add         rsp,20h
  00000001800131EB: pop         rdi
  00000001800131EC: ret
  00000001800131ED: int         3

The one on Windows 11 does:

u_cleanup:
  0000000180040A60: ret         0
  0000000180040A63: int         3

Is there a way you can run the CI on Windows 11 too just to see if the issue still there?

Unfortunately that's surprisingly not trivial at this time. We will not be able to check that easily.

If you have a full heap dump file that would help.

What is a good way to share a dump with you privately? We don't want to put it on a public github issue :).

[tarekgh]

What is a good way to share a dump with you privately? We don't want to put it on a public github issue :).

I'll let @rp9-next advise about that as he will be the one to look at. @rp9-next let me know if you need help and I can provide that.

[rp9-next]

@joncham @TautvydasZilys Thanks for reporting this issue.
@tarekgh @jefgen Thanks for preliminary investigation and re-routing !

We will log a bug into our system for windows and investigate further..
Please share the full dumps to targetdev@microsoft.com (or you could file a bug with windows for ICU using Feedback Hub for tracking purposes.)

From the explanation above, it seems that the system does not contain the no-op patch on Windows 10 (vs Win 11).
@TautvydasZilys , could you consider updating the underlying ICU version for your project to a later version (perhaps 68.2) and confirm if you are still facing the same issue?

You can find the instructions for the same here