split debug symbols

[lrusak]

I've started to play with the idea of having separate debug symbols that are available without having to rebuild a package.

Using Yocto for work It's really nice that the debug symbols can be added to an image simply by requesting them to be added, no rebuilding is necessary. This is what I've tried to emulate here.

Basically after a package is built, we strip the binary/library and split the debug symbols into another directory rather than throwing them away. This of course means that every build will use the -g cflag.

The benefit here is that you can provide every debug symbol for every library/executable and have that available at request. This is also nice for remote debugging.

Basically this PR implements the minimum required to do this. There is a bunch of other cleanup that can happen around the debug functions as this makes a lot of the old debug function obsolete.

debug symbols are stored in $BUILD/install_debug_pkg/, They can then be added to an image with the normal DEBUG=mesa style option. The symbols are then added to the image in /usr/lib/debug/ This is the normal search directory for GDB.

I've also added the ability to pack all the debug symbols into a tarball at the end of the build. This can then be transferred to a running system and extracted to a directory (say /storage/debug). You can then use gdb with set debug-file-directory /storage/debug and exec-file /usr/bin/modetest. There may be a way to add another default dir to gdb (or patch it in) but I haven't bothered to look yet.

$ ls -lh target
-rw-r--r--. 1 lukas lukas 1.6G Apr 24 20:05 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486_debug_symbols.tar.gz
-rw-r--r--. 1 lukas lukas  24M Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.kernel
-rw-r--r--. 1 lukas lukas 185M Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.system
-rw-r--r--. 1 lukas lukas 210M Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.tar
-rw-r--r--. 1 lukas lukas  125 Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.tar.sha256

Here is an example from a build I did with DEBUG=libdrm

# gdb --args /usr/bin/modetest 
GNU gdb (GDB) 11.2
<snip>
Reading symbols from /usr/bin/modetest...
Reading symbols from /usr/lib/debug//usr/bin/modetest.debug...
(gdb) r
Starting program: /usr/bin/modetest

This is also possible with kodi, but it seems the symbol file needs to be loaded manually with symbol-file /storage/debug/usr/lib/kodi/kodi.bin.debug I'm not sure why at this time it's not auto loaded (maybe because it's not the default load path for gdb, this should be looked into).

This PR includes some changes to build cmake and meson packages with debug symbols included. I've also made it so that meson packages aren't stripped by the meson build system automatically (we want to control the stripping).

ref:
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

There may also be in the future the opportunity to use debuginfod to automatically pull debug symbols into a build
ref:
https://sourceware.org/elfutils/Debuginfod.html
https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server

Debugging is always a challenge on embedded systems so let's do what we can to improve that.

One downside is that the build folder becomes quite large as we keep all the debug symbols.

For this PR I'm looking to get some feedback about the following:

• if people think this is a good idea.
• the names of the variables/paths/files used.
• any other insight to splitting debug symbols that I've missed.

Thanks for reading if you got this far 😄

[vpeter4]

Looks very nice idea to just copy debug symbols to the device and use them without recompile the image itself. But do we really need debug symbols for all packages? Probably not. Then user list of debug packages would be more useful.

But what about the disk usage with this approach? Looks like now more than double the disk space is needed (from 27GB to 60GB)? And probably longer build time too?

In this case the generation of debug symbols should be optional (disabled by default) and used only on user request. Also build folder name should reflect if build is made with debug symbols or not.

[lrusak]

Looks very nice idea to just copy debug symbols to the device and use them without recompile the image itself. But do we really need debug symbols for all packages? Probably not. Then user list of debug packages would be more useful.

It's really not that easy to know what you need until you need it.

But what about the disk usage with this approach? Looks like now more than double the disk space is needed (from 27GB to 60GB)? And probably longer build time too?

Yep the disk usage is much higher. I didn't notice any difference in build time (but I didn't time it).

In this case the generation of debug symbols should be optional (disabled by default) and used only on user request. Also build folder name should reflect if build is made with debug symbols or not.

I disagree. I think in order for this to be useful we need to build with debug symbols enabled by default. If they are disabled by default then there is no difference to what we are doing now really and you need to rebuild when you think you need debug symbols. I am for adding a switch to disable the debug symbol generation but I would like it to be enabled by default.

[HiassofT]

Please provide some more data so we can estimate the impact of this proposed change.

Check the following, both with LE master and LE master plus this PR, eg when building for RPi4:

• how long does a clean image build take (sources pre-populated, build, ccache and target folders removed)
• how big is the size of build, target and ccache folders
• how long does a rebuild take

[lrusak]

Here are some stats for RPi4:

Time is taken from the LE build output (wall clock time)

	With Debug Symbols	Without Debug Symbols
No ccache	01:14:45.297	01:13:04.631
With ccache	00:19:08.751	00:18:52.932
build folder size	33982128KB (33.98GB)	28741188KB (28.74GB)
target tarball	146483200B (146.48MB)	141690880B (141.69MB)
target img.gz	127548940B (127.54MB)	122752150B (122.75MB)
debug symbol tarball	613801765B (613.8MB)	N/A

Would have to look at where the difference is in the tarball size, likely something in the image is not being stripped.

Built on my NUC

CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz

[vpeter4]

Make comparison for Generic too. Because I got double the size (from 27GB to 60GB).

[lrusak]

Make comparison for Generic too. Because I got double the size (from 27GB to 60GB).

I can do that, but the size change really shouldn't matter as the value of the of the PR is much greater than having a bigger build folder (IMO). You can also disable it if you want.

[vpeter4]

as the value of the of the PR is much greater than having a bigger build folder (IMO)

Yes, this is only your opinion.
And I wonder how many self builders will actually use this feature. IMO not many. That's why it should be off by default and turned on who actually know how to use it. IMO

[chewitt]

I can see the use-case for this and I'm all for having the feature in the buildsystem, but I'd like it "off by default" so we don't impose extra resource needs on the moderately large number of casual image builders (and the smaller number of mid-tier contributors) who have no day-to-day need for symbols. Capability in the build-system is a positive. Simplicity (in the low-bar to access sense) of the build-system is also a positive.

[lrusak]

I can see the use-case for this and I'm all for having the feature in the buildsystem, but I'd like it "off by default" so we don't impose extra resource needs on the moderately large number of casual image builders (and the smaller number of mid-tier contributors) who have no day-to-day need for symbols. Capability in the build-system is a positive. Simplicity (in the low-bar to access sense) of the build-system is also a positive.

Defaults should be what we intend to build releases with. I would think that we would like to allow downloading debug symbols for releases. So that's why it should be on by default. Yocto also does this by default. Self builders can easily turn them off as they are likely using their own options file anyways.

[vpeter4]

Self builders can easily turn them off as they are likely using their own options file anyways.

Sorry but I can't resists - same applies for doing official builds: turn them on :)

[lrusak]

Self builders can easily turn them off as they are likely using their own options file anyways.

Sorry but I can't resists - same applies for doing official builds: turn them on :)

We shouldn't be changing options from what is checked in when doing a release.

[HiassofT]

While there's some merit in having symbols for official builds it's hardly helpful for actually debugging things - -O2/O3/Os builds with debug info are pretty unusable as too much stuff is optimized out.

When I'm actually debugging I use DEBUG on select packages (which selects -Og so debug info is actually useful) together with SPLIT_DEBUG_INFO so I can cross-debug with source and full debug info without blowing up image size.

So I'd also say make that option off by default and maybe enable it on official CI builds - developers who want to have debugging symbol of optimized binary builds can then also op-tin to get them if they see any advantage in having them.

[mglae]

As test I've build Generic/x11 in the default configuration release with external symbols. Some experiences/notes:

Image full

A 733MB .system file was created being to large for the image. Most
size wa occupied by an 1.6GB libLLVM-13.so. With the executable flag not set
it was not stripped. I'll add a suggestion.

cmake build type

We have to be careful which cmake build type to choose. Not only that
packages do not allow default build type with CMAKE_BUILD_TYPE=""
they can very picky what is allowed.

See #6422 for media-driver only allowing Relese/Debug and the work around.

Another example is kodi only allowing LTO for release types but forcing stripping for Release.

#6422 using Release build type without having cmake options set is the most reliable default way for me. Getting symbols is only a minor feature.

external symbols

Handling of external symbols can be simplified by creating a squashfs image and mounting it. Even /usr/lib/debug can be used when created in the image.

gdb OOM

When using full external symbols gdb -p $(pidof kodi.bin) in a 4GB VM is terminated by OOM-Killer. It may be nice to have a feature to only create a subset of external symbols.

ccache

I'm wondering if default ccache size has to be increased after much larger object files are created.

[mglae]

Suggestion to strip all libs even without executable flag:

Suggested change

	-type f -executable \
	! -name "*.ko" \
	! -name "ld.so" \
	! -name "ld-*.so" \
	! -name "libc-*.so" \
	! -name "libpthread-*.so" \
	! -name "libthread_db-*so" \
	-type f \( -executable -o -name "*.so*" \) \
	! -name "*.ko" \

[heitbaum]

cmake build type

We have to be careful which cmake build type to choose. Not only that packages do not allow default build type with CMAKE_BUILD_TYPE="" they can very picky what is allowed.

See #6422 for media-driver only allowing Relese/Debug and the work around.

Another example is kodi only allowing LTO for release types but forcing stripping for Release.

#6422 using Release build type without having cmake options set is the most reliable default way for me. Getting symbols is only a minor feature.

#6641 is now another package that is picky of the CMAKE_BUILD_TYPE