dotnet-symbol can't download symbols for dotnet 10.0.104 (source build by Ubuntu)

[zpodlovics]

Description

I originally installed dotnet-sdk-10.0 (10.0.104) via package manager (Ubuntu feed) but the dotnet symbol didn't work.

dotnet-symbol --symbols /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so -d -o .
Downloading from https://msdl.microsoft.com/download/symbols/
Writing files to .
Generating SymbolStore lookup keys for `/usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so`...
ERROR: Not Found: 1b770f2864b8d07e40344b727a61627d9672ac.debug - 'https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-bd1b770f2864b8d07e40344b727a61627d9672ac/_.debug'
Generated 1 SymbolStoreKeys for `/usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so`.

Expected behavior:
Download the symbol packages either from Microsoft or Canonical with or without specifying additional symbol server

Actual behavior:
The symbol download does not work.

How the interoperability story will work in this case? It seems that the Canonical Ubuntu source built symbols are not published to Microsoft dotnet side, and I am not aware of any symbol server available for dotnet symbol.

Configuration

OS/Distro: Ubuntu 24.04
Architecture: x64
Environment: Desktop, running on physical hardware

dotnet --info:

 Version:           10.0.104
 Commit:            80d3e14f5e
 Workload version:  10.0.100-manifests.c7707153
 MSBuild version:   18.0.11+80d3e14f5

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  24.04
 OS Platform: Linux
 RID:         ubuntu.24.04-x64
 Base Path:   /usr/lib/dotnet/sdk/10.0.104/

.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.

Host:
  Version:      10.0.4
  Architecture: x64
  Commit:       80d3e14f5e

.NET SDKs installed:
  8.0.125 [/usr/lib/dotnet/sdk]
  9.0.115 [/usr/lib/dotnet/sdk]
  10.0.104 [/usr/lib/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.25 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.14 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 10.0.4 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.25 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.14 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 10.0.4 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  arm64 [/usr/lib/dotnet]
    registered at [/etc/dotnet/install_location_arm64]

Environment variables:
  DOTNET_BUNDLE_EXTRACT_BASE_DIR           [/home/redacted/.cache/dotnet_bundle_extract]
  DOTNET_CLI_TELEMETRY_OPTOUT              [1]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Regression?

It's a regression, compared to the Microsoft provided packages. e.g.: #4481

Other information

The Canonical/Ubuntu provided source built "integration" tracked here:
dotnet/core#9258

[hoyosjs]

Microsoft .NET's build ends up in the microsoft symbol server. There's no equivalent for Canonical's build. Their story for this is https://documentation.ubuntu.com/ubuntu-for-developers/explanation/debugging-with-dotnet/#net-debug-symbols-packages

[richlander]

@dviererbe @mateusrodrigues

[mateusrodrigues]

@zpodlovics the managed symbols for the builds we distribute are available as packages and you can install them with sudo apt install dotnet-sdk-dbg-10.0 dotnet-runtime-dbg-10.0 aspnetcore-runtime-dbg-10.0. The PDB files will be placed alongside their respective DLLs.

For example:

# ls /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Threading.ThreadPool.*
/usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Threading.ThreadPool.dll
/usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Threading.ThreadPool.pdb

OTOH, the native symbols should be available from Ubuntu's debuginfod server and get automatically downloaded by GDB during a debug session.

[zpodlovics]

Thanks a lot for the quick response! Indeed, the managed debug packages are there and installed all of them. After I specified the debuginfod server it download something. The url probably https://debuginfod.ubuntu.com/download/symbols/_.debug/elf-buildid-sym-bd1b770f2864b8d07e40344b727a61627d9672ac/_.debug based on the earlier failed url.

Downloading from https://debuginfod.ubuntu.com/
Writing files to .
Generating SymbolStore lookup keys for `/usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so`...
Cached: /home/redacted/.dotnet/symbolcache/_.debug/elf-buildid-sym-bd1b770f2864b8d07e40344b727a61627d9672ac/_.debug
Writing: ./1b770f2864b8d07e40344b727a61627d9672ac.debug
Generated 1 SymbolStoreKeys for `/usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so`.

However the content is not what I expected for 1b770f2864b8d07e40344b727a61627d9672ac.debug:

<html>
    <head>
        <noscript>
            <meta http-equiv="refresh" content="0; url=../how-to/debugging/about-debuginfod/index.html"/>
        </noscript>
    <script async type="text/javascript" src="/_/static/javascript/readthedocs-addons.js"></script><meta name="readthedocs-project-slug" content="canonical-ubuntu-server-documentation" /><meta name="readthedocs-version-slug" content="latest" /><meta name="readthedocs-resolver-filename" content="/service-debuginfod/" /><meta name="readthedocs-http-status" content="200" /></head>
    <body>
        <script>
            window.location.href = '../how-to/debugging/about-debuginfod/index.html' + (window.location.search || '') + (window.location.hash || '');
        </script>
        <p>You should have been redirected.</p>
        <a href="../how-to/debugging/about-debuginfod/index.html">If not, click here to continue.</a>
    </body>
</html>

[hoyosjs]

debuginfod is not a classic windows debug server - their key schema is different. If you need symbols you have a few options:

• use gdb that automatically downloads them
• use debuginfod-find : DEBUGINFOD_URLS="https://debuginfod.ubuntu.com" debuginfod-find debuginfo <file path>

[mateusrodrigues]

If the symbols you need are missing from debuginfod, you can use ddebs (debug symbol packages) instead, along with GDB: https://ubuntu.com/server/docs/how-to/debugging/debug-symbol-packages/#getting-dbgsym-ddeb-packages

[zpodlovics]

Maybe I missed something, but how can I use any of the debuginfod / ddebs with dotnet-* tools?

export|grep DEBUG
declare -x DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"

export DEBUGINFOD_PROGRESS=1 
eu-srcfiles -e /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
Downloading -

eu-readelf -n /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so

Note section [ 1] '.note.gnu.build-id' of 36 bytes at offset 0x2a8:
  Owner          Data size  Type
  GNU                   20  GNU_BUILD_ID
    Build ID: bd1b770f2864b8d07e40344b727a61627d9672ac

gdb -q /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
Reading symbols from /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
(No debugging symbols found in /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so)

readelf -n /usr/bin/dotnet 
Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: d41880ed0c4d97ba9c2d46165abf8e558fec684f

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature: IBT, SHSTK

gdb -q /usr/bin/dotnet
Reading symbols from /usr/bin/dotnet...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/lib/dotnet/dotnet
(No debugging symbols found in /usr/bin/dotnet)                                       

getting a stack of a dotnet fsi process:

[sudo] password for redacted: 
PID 133176 - process
TID 133176:
#0  0x0000749643298d71     __futex_abstimed_wait_common64 - /usr/lib/x86_64-linux-gnu/libc.so.6
    ./nptl/futex-internal.c:57:12
#1  0x0000749643298d71     __futex_abstimed_wait_common - /usr/lib/x86_64-linux-gnu/libc.so.6
    ./nptl/futex-internal.c:87:9
#2  0x0000749643298d71     __GI___futex_abstimed_wait_cancelable64 - /usr/lib/x86_64-linux-gnu/libc.so.6
    ./nptl/futex-internal.c:139:10
#3  0x000074964329b7ed - 1 __pthread_cond_wait_common - /usr/lib/x86_64-linux-gnu/libc.so.6
    ./nptl/pthread_cond_wait.c:503:10
#4  0x000074964329b7ed - 1 ___pthread_cond_wait - /usr/lib/x86_64-linux-gnu/libc.so.6
    ./nptl/pthread_cond_wait.c:627:10
#5  0x000074964304cd22 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#6  0x000074964304c953 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#7  0x00007496430515cc - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#8  0x00007496430518a7 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#9  0x0000749642e2d3f8 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#10 0x0000749642e28378 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#11 0x0000749642e83c14 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#12 0x00007495c3c6b442 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Private.CoreLib.dll
#13 0x00007495c77545db - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/FSharp.Compiler.Interactive.Settings.dll
#14 0x0000749643013b1c - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#15 0x0000749642e65fc9 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#16 0x0000749642f0809f - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#17 0x00007495c3ce4d76 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Private.CoreLib.dll
#18 0x00007495c3ce4dea - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Private.CoreLib.dll
#19 0x00007495c3cf5a95 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/System.Private.CoreLib.dll
#20 0x00007495c5e79bfe - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/FSharp.Compiler.Service.dll
#21 0x00007495c556c890 - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/FSharp.Compiler.Service.dll
#22 0x00007495c556cad0 - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/FSharp.Compiler.Service.dll
#23 0x00007495c55888b6 - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/FSharp.Compiler.Service.dll
#24 0x00007495c4a25d77 - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/fsi.dll
#25 0x00007495c4a25f4f - 1 - /usr/lib/dotnet/sdk/10.0.104/FSharp/fsi.dll
#26 0x0000749643013b1c - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#27 0x0000749642e66a5a - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#28 0x0000749642d45475 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#29 0x0000749642d458f0 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#30 0x0000749642d6f88c - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#31 0x0000749642bc96f1 - 1 coreclr_execute_assembly - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libcoreclr.so
#32 0x00007496434a36b8 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libhostpolicy.so
#33 0x00007496434a3a44 - 1 - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libhostpolicy.so
#34 0x00007496434a4b0a - 1 corehost_main - /usr/lib/dotnet/shared/Microsoft.NETCore.App/10.0.4/libhostpolicy.so
#35 0x00007496434e5b7d - 1 - /usr/lib/dotnet/host/fxr/10.0.4/libhostfxr.so
#36 0x00007496434e477c - 1 - /usr/lib/dotnet/host/fxr/10.0.4/libhostfxr.so
#37 0x00007496434deea6 - 1 hostfxr_main_startupinfo - /usr/lib/dotnet/host/fxr/10.0.4/libhostfxr.so
#38 0x00005d02edb446f7 - 1 - /usr/lib/dotnet/dotnet
#39 0x00005d02edb44a0f - 1 - /usr/lib/dotnet/dotnet
#40 0x000074964322a1ca - 1 __libc_start_call_main - /usr/lib/x86_64-linux-gnu/libc.so.6
    ../sysdeps/nptl/libc_start_call_main.h:58:16
#41 0x000074964322a28b - 1 __libc_start_main_impl - /usr/lib/x86_64-linux-gnu/libc.so.6
    ../csu/libc-start.c:360:3
#42 0x00005d02edb439a5 - 1 - /usr/lib/dotnet/dotnet
[..]

[hoyosjs]

It looks like those symbols are not in debuginfod:

DEBUGINFOD_URLS="https://debuginfod.ubuntu.com" llvm-debuginfod-find-20 --debuginfo bd1b770f2864b8d07e40344b727a61627d9672ac

You'll need to use the ddeb packages.