Extend libproc's dbgsym search to include the guest zone; Fix bug where libproc generated dbgsym path with spaces rather than zeros

[smokris]

Summary

When various SmartOS GZ tools read a coredump from a process in an LX zone, they use libproc to try to find debug symbol files (dbgsyms).

libproc's Pbuild_file_symtab function reads the ELF header's .note.gnu.build-id value and tries to construct a pathname from it. However:

1. for certain specific build IDs, it constructs the path improperly (formatting bytes as hex with leading spaces rather than leading zeros)
2. it uses the build ID to search the global zone, but not the guest zone

…and thus can't find the dbgsym.

Steps to reproduce

On the GZ, create an LX zone from the latest Debian dataset:

imgadm import c5155873-e959-4dab-86a6-b8b5c69b082c
echo '{"brand":"lx","image_uuid":"c5155873-e959-4dab-86a6-b8b5c69b082c","kernel_version":"4.15","resolvers":["1.1.1.1"],"nics":[{"nic_tag":"admin","ips":["dhcp"]}]}' | vmadm create
zlogin $UUID

Inside the guest zone, install dbgsym and generate a coredump that refers to it:

echo 'deb http://debug.mirrors.debian.org/debian-debug/ bookworm-debug main' >> /etc/apt/sources.list
apt update
apt install gcc libuv1-dev libuv1-dbgsym
cat <<EOF >crash.c
#include <uv.h>
void crashFromLibUVCallback(uv_idle_t* handle) {
    ((char *)0)[0] = 0;
}
int main() {
    uv_idle_t idler;
    uv_idle_init(uv_default_loop(), &idler);
    uv_idle_start(&idler, crashFromLibUVCallback);
    uv_run(uv_default_loop(), UV_RUN_DEFAULT);
    uv_loop_close(uv_default_loop());
    return 0;
}
EOF
gcc crash.c -luv -o crash
./crash

Back on the GZ, examine the coredump:

cd /zones/$UUID/cores
LIBPROC_DEBUG=1 pstack core.crash.69857

It fails to find the dbgsym:

libproc DEBUG: Symbol table found for /zones/debf49d9-f31e-4e27-abd6-f93cb1be90b0/root/usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
libproc DEBUG: found .gnu_debuglink section for /zones/debf49d9-f31e-4e27-abd6-f93cb1be90b0/root/usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
libproc DEBUG: attempting to find build id alternate debug file at /usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a 57f53a815dbb0abe6b1.debug
libproc DEBUG: attempt failed

…and is thus unable to symbolicate stack frames involving that library:

core 'core.crash.16360' of 16360:       ./crash
 00007fffef451186 crashFromLibUVCallback () + d
 00007fffef045891 ???????? ()

Analysis

Looking at this line in the above output:

libproc DEBUG: attempting to find build id alternate debug file at /usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a 57f53a815dbb0abe6b1.debug
                                                     ----------------------->                                    ^ ⚠️

There's a space in the middle of the hex string! However, the actual dbgsym path has a zero in that position instead of a space:

# dpkg-query -L libuv1-dbgsym  | grep '\.debug$'
/usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a057f53a815dbb0abe6b1.debug
                                              ^

The hex string is generated by this line: https://github.com/illumos/illumos-gate/blob/a2e4f154b5ad0aae3da0ea3c730b6e993b11ba23/usr/src/lib/libproc/common/Psymtab.c#L2155

(void) snprintf(buf + bo, sizeof (buf) - bo, "%2x", *dp);

If I understand correctly, that format should instead be %02x, so that byte values less than 16 have a leading zero rather than a leading space. I changed that in eacd134.

After making that change, the path no longer contains any spaces:

libproc DEBUG: attempting to find build id alternate debug file at /usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a057f53a815dbb0abe6b1.debug
                                                     ----------------------->                                    ^ ✅

…and it exactly matches the guest zone path provided by package libuv1-dbgsym.

However, libproc still can't find it, since it checks for the dbgsym in the GZ but not the guest zone. In 808e147 I added a clause that checks for it in the guest zone. Now it finds the dbgsym:

libproc DEBUG: attempting to find build id alternate debug file in global zone at /usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a057f53a815dbb0abe6b1.debug
libproc DEBUG: attempt failed
libproc DEBUG: attempting to find build id alternate debug file in guest zone at /zones/9b7174a3-b573-4644-ba2b-fe570befc19c/root/usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a057f53a815dbb0abe6b1.debug
libproc DEBUG: found corresponding section in alternate file
libproc DEBUG: successfully loaded additional debug symbols for /zones/9b7174a3-b573-4644-ba2b-fe570befc19c/root/usr/lib/x86_64-linux-gnu/libuv.so.1.0.0 from /zones/9b7174a3-b573-4644-ba2b-fe570befc19c/root/usr/lib/debug/.build-id/f4/c8e3456a6c9d4da77a057f53a815dbb0abe6b1.debug
libproc DEBUG: attempt succeeded

…and it's able to successfully symbolicate stack frames involving that library:

core 'core.crash.8273' of 8273:	./crash
 00007fffef451186 crashFromLibUVCallback () + d
 00007fffef045891 uv__run_idle () + 91

[smokris]

I just rebased this on top of the latest master branch.

[danmcd]

Sorry for delay on this. I see where you're going here.

Must-ask-question: Any AI/LLM assistance on this? RFD 187 demands transparency.

Also, do you think this might have use outside LX? IF SO perhaps it's worth upstreaming? (I also wonder if OS-4573 might also belong to upstream?)

[danmcd]

This feels too-indenty, but I'm not sure if I can suggest anything better.