linux: `clrthreads` fails for AOT app

[am11]

Description

# alpine arm64
$ dotnet --version
9.0.202

$ dotnet-sos --version
9.0.607501+a651406e39038aef1dbc7c8097b52953284dba27

$ dotnet new console -n testsos
$ cd testsos
$ echo 'throw new Exception("doodlebug");' >> Program.cs
$ dotnet publish -p:PublishAot=true -o dist

$ lldb dist/testsos

Current symbol store settings:
-> Cache: /root/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
(lldb) target create "dist/testsos"
Current executable set to '/testsos/dist/testsos' (aarch64).

(lldb) r
Process 2592 launched: '/testsos/dist/testsos' (aarch64)
Hello, World!
Unhandled exception. System.Exception: doodlebug
   at Program.<Main>$(String[] args) + 0x3c
Process 2592 stopped
* thread #1, name = 'testsos', stop reason = signal SIGABRT
    frame #0: 0x0000fffff7f8eef0 ld-musl-aarch64.so.1`__restore_sigs(set=0x0000fffffffff240) at block.c:44:1

(lldb) clrthreads
Failed to find runtime module (libcoreclr.so), 0x80004002
Extension commands need it in order to have something to do.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
Threads  failed

(lldb) expression DotNetRuntimeDebugHeader
(DotNetRuntimeDebugHeader) $0 = {
  Cookie = "DNDH"
  MajorVersion = 4
  MinorVersion = 0
  Flags = 1
  ReservedPadding1 = 0
  DebugTypeEntries = 0x0000aaaaaac0bb80
  GlobalEntries = 0x0000aaaaaac0bb00
}

(lldb) expression *(DebugTypeEntry*)DotNetRuntimeDebugHeader.DebugTypeEntries
(DebugTypeEntry) $1 = (TypeName = "GcDacVars", FieldName = "SIZEOF", FieldOffset = 320, ResevedPadding = 0)

(lldb) expression *(DebugTypeEntry*)DotNetRuntimeDebugHeader.GlobalEntries
(DebugTypeEntry) $0 = (TypeName = "g_CrashInfoBuffer", FieldName = "{\"version\":\"1.0.0\",\"runtime_base\":\"0xAAAAAAAA0000\",\"runtime_type\":\"4\",\"runtime_version\":\"9.0.3\",\"reason\":\"1\",\"thread\":\"0xA99\",\"message\":\"Unhandled exception: a managed exception was not handled before reaching unmanaged code\",\"exception\":{\"address\":\"0xFFBFFA80AAC8\",\"hr\":\"0x80131500\",\"message\":\"doodlebug\",\"type\":\"System.Exception\",\"stack\":[{\"ip\":\"0xAAAAAAB5E3EC\",\"module\":\"0xAAAAAAAA0000\",\"offset\":\"0x3C\",\"name\":\"Program.<Main>$(String[] args)\"},{\"ip\":\"0xAAAAAAB70AC0\",\"module\":\"0xAAAAAAAA0000\",\"offset\":\"0xD0\"}]}}", FieldOffset = 2864448864, ResevedPadding = 43690)

Not sure how it is supposed to work, per docs: https://learn.microsoft.com/dotnet/core/diagnostics/debugger-extensions

Threads (clrthreads) [-live] [-special] 	Displays all managed threads in the process.
...
Supported for Native AOT applications.

Basically none of the "support for Native AOT" commands are working.

[tommcdon]

@mikem8361

[hoyosjs]

@am11 - can you please try https://www.nuget.org/packages/dotnet-debugger-extensions/9.0.607601? dotnet-sos doesn't have a lot of the nativeaot support.

[am11]

@hoyosjs, ah, I missed that dotnet-sos is replaced by dotnet-debugger-extensions, thanks for the info.

It is also fail the same way (one additional line Special diagnostics info read failed):

(lldb) clrthreads 
Special diagnostics info read failed
Failed to find runtime module (libcoreclr.so), 0x80004002
Extension commands need it in order to have something to do.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
Threads  failed

(lldb) expr DotNetRuntimeDebugHeader
(DotNetRuntimeDebugHeader) $0 = {
  Cookie = "DNDH"
  MajorVersion = 4
  MinorVersion = 0
  Flags = 1
  ReservedPadding1 = 0
  DebugTypeEntries = 0x0000aaaaaac0bb80
  GlobalEntries = 0x0000aaaaaac0bb00
}

(lldb) expr *(DebugTypeEntry*)DotNetRuntimeDebugHeader.DebugTypeEntries
(DebugTypeEntry) $1 = (TypeName = "GcDacVars", FieldName = "SIZEOF", FieldOffset = 320, ResevedPadding = 0)

(lldb) expr *(DebugTypeEntry*)DotNetRuntimeDebugHeader.GlobalEntries
(DebugTypeEntry) $2 = (TypeName = "g_CrashInfoBuffer", FieldName = "{\"version\":\"1.0.0\",\"runtime_base\":\"0xAAAAAAAA0000\",\"runtime_type\":\"4\",\"runtime_version\":\"9.0.3\",\"reason\":\"1\",\"thread\":\"0x19D9\",\"message\":\"Unhandled exception: a managed exception was not handled before reaching unmanaged code\",\"exception\":{\"address\":\"0xFFBFFA80AAC8\",\"hr\":\"0x80131500\",\"message\":\"doodlebug\",\"type\":\"System.Exception\",\"stack\":[{\"ip\":\"0xAAAAAAB5E3EC\",\"module\":\"0xAAAAAAAA0000\",\"offset\":\"0x3C\",\"name\":\"Program.<Main>$(String[] args)\"},{\"ip\":\"0xAAAAAAB70AC0\",\"module\":\"0xAAAAAAAA0000\",\"offset\":\"0xD0\"}]}}", FieldOffset = 2864448864, ResevedPadding = 43690)

[mikem8361]

What does sosstatus display? It shows the versions now of all the extensions loaded including the native AOT one.

[am11]

(lldb) sosstatus
Target OS: LINUX Architecture: Arm64 ProcessId: 306 (0x132)
SpecialDiagInfoHeader   : 00007FFFFFF10000 <NONE>
Current symbol store settings:
-> Cache: /root/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
Extensions loaded:
-> 9.0.607501+a651406e39038aef1dbc7c8097b52953284dba27 /root/.dotnet/sos/Microsoft.Diagnostics.ExtensionCommands.dll
-> 9.0.607601+1a717978b6db5d82880ea480ca7a863c23e22889 /root/.dotnet/sos/extensions/Microsoft.Diagnostics.DebuggerCommands.dll
-> 9.0.607601+1a717978b6db5d82880ea480ca7a863c23e22889 /root/.dotnet/sos/extensions/Microsoft.Diagnostics.DataContractReader.Extension.dll
-> 9.0.607601+1a717978b6db5d82880ea480ca7a863c23e22889 /root/.dotnet/sos/extensions/Microsoft.Diagnostics.DataContractReader.dll
GC memory usage for managed SOS components: 1,160,592 bytes

(lldb) clrthreads
Failed to find runtime module (libcoreclr.so), 0x80004002
Extension commands need it in order to have something to do.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
Threads  failed

[mikem8361]

I couldn't repro this on a Ubuntu 22.04 on an arm64 pi. Looks like you repro on an alpine (musl) arm64 linux machine. Not sure why that would make a difference.

(lldb) clrthreads
OSID Exception
1E6E 0000FFBFFA80C190 Unknown Type
1E72
(lldb) runtimes
#0 Data contract NativeAOT runtime at 0000AAAAAAAA0000 size 001C6678
    Exports:
        DotNetRuntimeInfo       : <NO SYMBOL>
        g_dacTable              : <NO SYMBOL>
        DotNetRuntimeDebugHeader: 0000AAAAAAC33728
DAC signature verification check enabled: False
(lldb) sosstatus
Target OS: LINUX Architecture: Arm64 ProcessId: 7790 (0x1E6E)
    #0 Data contract NativeAOT runtime at 0000AAAAAAAA0000 size 001C6678
        Exports:
            DotNetRuntimeInfo       : <NO SYMBOL>
            g_dacTable              : <NO SYMBOL>
            DotNetRuntimeDebugHeader: 0000AAAAAAC33728
SpecialDiagInfoHeader   : 00007FFFFFF10000 <NONE>
Current symbol store settings:
-> Cache: /home/ubuntu/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
Extensions loaded:
-> 9.0.607501+a651406e39038aef1dbc7c8097b52953284dba27 /home/ubuntu/.dotnet/sos/Microsoft.Diagnostics.ExtensionCommands.dll
-> 9.0.607601+1a717978b6db5d82880ea480ca7a863c23e22889 /home/ubuntu/.dotnet/sos/extensions/Microsoft.Diagnostics.DataContractReader.dll
-> 9.0.607601+1a717978b6db5d82880ea480ca7a863c23e22889 /home/ubuntu/.dotnet/sos/extensions/Microsoft.Diagnostics.DebuggerCommands.dll
-> 9.0.607601+1a717978b6db5d82880ea480ca7a863c23e22889 /home/ubuntu/.dotnet/sos/extensions/Microsoft.Diagnostics.DataContractReader.Extension.dll
GC memory usage for managed SOS components: 1,402,112 bytes

[am11]

Yup, mentioned alpine in top post.

Rereading previous comment #5364 (comment), this message was printed:

diagnostics/src/SOS/lldbplugin/services.cpp

Line 541 in c73305b

Output(DEBUG_OUTPUT_WARNING, "Special diagnostics info read failed\n");

so it's getting the lldb process (line 522) with valid thread (line 527), but fails to find the header; although expr DotNetRuntimeDebugHeader shows the header exists?

[mikem8361]

It fails there for live sessions or system core dumps because createdump is what adds this special memory region/block

[am11]

Here is the Dockerfile for repro:

FROM alpine

RUN apk update && apk add clang lldb py3-lldb dotnet-sdk-9.0

ENV PATH="$PATH:/root/.dotnet/tools"

RUN dotnet tool install --global dotnet-debugger-extensions && \
  dotnet-debugger-extensions install --accept-license-agreement 

RUN dotnet new console -n testsos && \
  echo 'throw new Exception("doodlebug");' >> testsos/Program.cs && \
  dotnet publish testsos -p:PublishAot=true -o dist

ENTRYPOINT ["lldb", "dist/testsos", "-o", "run", "-o", "sosstatus"]

Build: docker build . -t alpine-sos-test
Run: docker run --rm -it alpine-sos-test

[mikem8361]

Is there anyway you could send me a core dump of your test app? It will save time having to setup docker on our arm64 device, etc.