-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
split debug symbols #6426
base: master
Are you sure you want to change the base?
split debug symbols #6426
Conversation
Looks very nice idea to just copy debug symbols to the device and use them without recompile the image itself. But do we really need debug symbols for all packages? Probably not. Then user list of debug packages would be more useful. But what about the disk usage with this approach? Looks like now more than double the disk space is needed (from 27GB to 60GB)? And probably longer build time too? In this case the generation of debug symbols should be optional (disabled by default) and used only on user request. Also build folder name should reflect if build is made with debug symbols or not. |
It's really not that easy to know what you need until you need it.
Yep the disk usage is much higher. I didn't notice any difference in build time (but I didn't time it).
I disagree. I think in order for this to be useful we need to build with debug symbols enabled by default. If they are disabled by default then there is no difference to what we are doing now really and you need to rebuild when you think you need debug symbols. I am for adding a switch to disable the debug symbol generation but I would like it to be enabled by default. |
Please provide some more data so we can estimate the impact of this proposed change. Check the following, both with LE master and LE master plus this PR, eg when building for RPi4:
|
Here are some stats for RPi4: Time is taken from the LE build output (wall clock time)
Would have to look at where the difference is in the tarball size, likely something in the image is not being stripped. Built on my NUC
|
Make comparison for Generic too. Because I got double the size (from 27GB to 60GB). |
I can do that, but the size change really shouldn't matter as the value of the of the PR is much greater than having a bigger build folder (IMO). You can also disable it if you want. |
Yes, this is only your opinion. |
I can see the use-case for this and I'm all for having the feature in the buildsystem, but I'd like it "off by default" so we don't impose extra resource needs on the moderately large number of casual image builders (and the smaller number of mid-tier contributors) who have no day-to-day need for symbols. Capability in the build-system is a positive. Simplicity (in the low-bar to access sense) of the build-system is also a positive. |
Defaults should be what we intend to build releases with. I would think that we would like to allow downloading debug symbols for releases. So that's why it should be on by default. Yocto also does this by default. Self builders can easily turn them off as they are likely using their own options file anyways. |
Sorry but I can't resists - same applies for doing official builds: turn them on :) |
We shouldn't be changing options from what is checked in when doing a release. |
While there's some merit in having symbols for official builds it's hardly helpful for actually debugging things - -O2/O3/Os builds with debug info are pretty unusable as too much stuff is optimized out. When I'm actually debugging I use DEBUG on select packages (which selects -Og so debug info is actually useful) together with SPLIT_DEBUG_INFO so I can cross-debug with source and full debug info without blowing up image size. So I'd also say make that option off by default and maybe enable it on official CI builds - developers who want to have debugging symbol of optimized binary builds can then also op-tin to get them if they see any advantage in having them. |
As test I've build Generic/x11 in the default configuration release with external symbols. Some experiences/notes: Image fullA 733MB .system file was created being to large for the image. Most cmake build typeWe have to be careful which cmake build type to choose. Not only that See #6422 for media-driver only allowing Relese/Debug and the work around. Another example is kodi only allowing LTO for release types but forcing stripping for Release. #6422 using Release build type without having cmake options set is the most reliable default way for me. Getting symbols is only a minor feature. external symbolsHandling of external symbols can be simplified by creating a squashfs image and mounting it. Even /usr/lib/debug can be used when created in the image. gdb OOMWhen using full external symbols ccacheI'm wondering if default ccache size has to be increased after much larger object files are created. |
-type f -executable \ | ||
! -name "*.ko" \ | ||
! -name "ld.so" \ | ||
! -name "ld-*.so" \ | ||
! -name "libc-*.so" \ | ||
! -name "libpthread-*.so" \ | ||
! -name "libthread_db-*so" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggestion to strip all libs even without executable flag:
-type f -executable \ | |
! -name "*.ko" \ | |
! -name "ld.so" \ | |
! -name "ld-*.so" \ | |
! -name "libc-*.so" \ | |
! -name "libpthread-*.so" \ | |
! -name "libthread_db-*so" \ | |
-type f \( -executable -o -name "*.so*" \) \ | |
! -name "*.ko" \ |
#6641 is now another package that is picky of the CMAKE_BUILD_TYPE |
I've started to play with the idea of having separate debug symbols that are available without having to rebuild a package.
Using Yocto for work It's really nice that the debug symbols can be added to an image simply by requesting them to be added, no rebuilding is necessary. This is what I've tried to emulate here.
Basically after a package is built, we strip the binary/library and split the debug symbols into another directory rather than throwing them away. This of course means that every build will use the
-g
cflag.The benefit here is that you can provide every debug symbol for every library/executable and have that available at request. This is also nice for remote debugging.
Basically this PR implements the minimum required to do this. There is a bunch of other cleanup that can happen around the debug functions as this makes a lot of the old debug function obsolete.
debug symbols are stored in
$BUILD/install_debug_pkg/
, They can then be added to an image with the normalDEBUG=mesa
style option. The symbols are then added to the image in/usr/lib/debug/
This is the normal search directory for GDB.I've also added the ability to pack all the debug symbols into a tarball at the end of the build. This can then be transferred to a running system and extracted to a directory (say
/storage/debug
). You can then use gdb withset debug-file-directory /storage/debug
andexec-file /usr/bin/modetest
. There may be a way to add another default dir to gdb (or patch it in) but I haven't bothered to look yet.Here is an example from a build I did with
DEBUG=libdrm
This is also possible with kodi, but it seems the symbol file needs to be loaded manually with
symbol-file /storage/debug/usr/lib/kodi/kodi.bin.debug
I'm not sure why at this time it's not auto loaded (maybe because it's not the default load path for gdb, this should be looked into).This PR includes some changes to build cmake and meson packages with debug symbols included. I've also made it so that meson packages aren't stripped by the meson build system automatically (we want to control the stripping).
ref:
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
There may also be in the future the opportunity to use debuginfod to automatically pull debug symbols into a build
ref:
https://sourceware.org/elfutils/Debuginfod.html
https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server
Debugging is always a challenge on embedded systems so let's do what we can to improve that.
One downside is that the build folder becomes quite large as we keep all the debug symbols.
For this PR I'm looking to get some feedback about the following:
Thanks for reading if you got this far 😄