Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split debug symbols #6426

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

split debug symbols #6426

wants to merge 5 commits into from

Conversation

lrusak
Copy link
Member

@lrusak lrusak commented Apr 25, 2022

I've started to play with the idea of having separate debug symbols that are available without having to rebuild a package.

Using Yocto for work It's really nice that the debug symbols can be added to an image simply by requesting them to be added, no rebuilding is necessary. This is what I've tried to emulate here.

Basically after a package is built, we strip the binary/library and split the debug symbols into another directory rather than throwing them away. This of course means that every build will use the -g cflag.

The benefit here is that you can provide every debug symbol for every library/executable and have that available at request. This is also nice for remote debugging.

Basically this PR implements the minimum required to do this. There is a bunch of other cleanup that can happen around the debug functions as this makes a lot of the old debug function obsolete.

debug symbols are stored in $BUILD/install_debug_pkg/, They can then be added to an image with the normal DEBUG=mesa style option. The symbols are then added to the image in /usr/lib/debug/ This is the normal search directory for GDB.

I've also added the ability to pack all the debug symbols into a tarball at the end of the build. This can then be transferred to a running system and extracted to a directory (say /storage/debug). You can then use gdb with set debug-file-directory /storage/debug and exec-file /usr/bin/modetest. There may be a way to add another default dir to gdb (or patch it in) but I haven't bothered to look yet.

$ ls -lh target
-rw-r--r--. 1 lukas lukas 1.6G Apr 24 20:05 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486_debug_symbols.tar.gz
-rw-r--r--. 1 lukas lukas  24M Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.kernel
-rw-r--r--. 1 lukas lukas 185M Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.system
-rw-r--r--. 1 lukas lukas 210M Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.tar
-rw-r--r--. 1 lukas lukas  125 Apr 24 20:04 LibreELEC-gbm.x86_64-11.0-devel-20220424200357-1e3c486.tar.sha256

Here is an example from a build I did with DEBUG=libdrm

# gdb --args /usr/bin/modetest 
GNU gdb (GDB) 11.2
<snip>
Reading symbols from /usr/bin/modetest...
Reading symbols from /usr/lib/debug//usr/bin/modetest.debug...
(gdb) r
Starting program: /usr/bin/modetest

This is also possible with kodi, but it seems the symbol file needs to be loaded manually with symbol-file /storage/debug/usr/lib/kodi/kodi.bin.debug I'm not sure why at this time it's not auto loaded (maybe because it's not the default load path for gdb, this should be looked into).

This PR includes some changes to build cmake and meson packages with debug symbols included. I've also made it so that meson packages aren't stripped by the meson build system automatically (we want to control the stripping).

ref:
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

There may also be in the future the opportunity to use debuginfod to automatically pull debug symbols into a build
ref:
https://sourceware.org/elfutils/Debuginfod.html
https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server

Debugging is always a challenge on embedded systems so let's do what we can to improve that.

One downside is that the build folder becomes quite large as we keep all the debug symbols.

For this PR I'm looking to get some feedback about the following:

  • if people think this is a good idea.
  • the names of the variables/paths/files used.
  • any other insight to splitting debug symbols that I've missed.

Thanks for reading if you got this far 😄

@vpeter4
Copy link
Contributor

vpeter4 commented Apr 26, 2022

Looks very nice idea to just copy debug symbols to the device and use them without recompile the image itself. But do we really need debug symbols for all packages? Probably not. Then user list of debug packages would be more useful.

But what about the disk usage with this approach? Looks like now more than double the disk space is needed (from 27GB to 60GB)? And probably longer build time too?

In this case the generation of debug symbols should be optional (disabled by default) and used only on user request. Also build folder name should reflect if build is made with debug symbols or not.

@lrusak
Copy link
Member Author

lrusak commented May 5, 2022

Looks very nice idea to just copy debug symbols to the device and use them without recompile the image itself. But do we really need debug symbols for all packages? Probably not. Then user list of debug packages would be more useful.

It's really not that easy to know what you need until you need it.

But what about the disk usage with this approach? Looks like now more than double the disk space is needed (from 27GB to 60GB)? And probably longer build time too?

Yep the disk usage is much higher. I didn't notice any difference in build time (but I didn't time it).

In this case the generation of debug symbols should be optional (disabled by default) and used only on user request. Also build folder name should reflect if build is made with debug symbols or not.

I disagree. I think in order for this to be useful we need to build with debug symbols enabled by default. If they are disabled by default then there is no difference to what we are doing now really and you need to rebuild when you think you need debug symbols. I am for adding a switch to disable the debug symbol generation but I would like it to be enabled by default.

@HiassofT
Copy link
Member

HiassofT commented May 6, 2022

Please provide some more data so we can estimate the impact of this proposed change.

Check the following, both with LE master and LE master plus this PR, eg when building for RPi4:

  • how long does a clean image build take (sources pre-populated, build, ccache and target folders removed)
  • how big is the size of build, target and ccache folders
  • how long does a rebuild take

@lrusak
Copy link
Member Author

lrusak commented May 7, 2022

Here are some stats for RPi4:

Time is taken from the LE build output (wall clock time)

With Debug Symbols Without Debug Symbols
No ccache 01:14:45.297 01:13:04.631
With ccache 00:19:08.751 00:18:52.932
build folder size 33982128KB (33.98GB) 28741188KB (28.74GB)
target tarball 146483200B (146.48MB) 141690880B (141.69MB)
target img.gz 127548940B (127.54MB) 122752150B (122.75MB)
debug symbol tarball 613801765B (613.8MB) N/A

Would have to look at where the difference is in the tarball size, likely something in the image is not being stripped.

Built on my NUC

CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz

@lrusak lrusak changed the title [RFC] split debug symbols split debug symbols May 7, 2022
@vpeter4
Copy link
Contributor

vpeter4 commented May 7, 2022

Make comparison for Generic too. Because I got double the size (from 27GB to 60GB).

@lrusak
Copy link
Member Author

lrusak commented May 7, 2022

Make comparison for Generic too. Because I got double the size (from 27GB to 60GB).

I can do that, but the size change really shouldn't matter as the value of the of the PR is much greater than having a bigger build folder (IMO). You can also disable it if you want.

@vpeter4
Copy link
Contributor

vpeter4 commented May 7, 2022

as the value of the of the PR is much greater than having a bigger build folder (IMO)

Yes, this is only your opinion.
And I wonder how many self builders will actually use this feature. IMO not many. That's why it should be off by default and turned on who actually know how to use it. IMO

@chewitt
Copy link
Member

chewitt commented May 9, 2022

I can see the use-case for this and I'm all for having the feature in the buildsystem, but I'd like it "off by default" so we don't impose extra resource needs on the moderately large number of casual image builders (and the smaller number of mid-tier contributors) who have no day-to-day need for symbols. Capability in the build-system is a positive. Simplicity (in the low-bar to access sense) of the build-system is also a positive.

@lrusak
Copy link
Member Author

lrusak commented May 9, 2022

I can see the use-case for this and I'm all for having the feature in the buildsystem, but I'd like it "off by default" so we don't impose extra resource needs on the moderately large number of casual image builders (and the smaller number of mid-tier contributors) who have no day-to-day need for symbols. Capability in the build-system is a positive. Simplicity (in the low-bar to access sense) of the build-system is also a positive.

Defaults should be what we intend to build releases with. I would think that we would like to allow downloading debug symbols for releases. So that's why it should be on by default. Yocto also does this by default. Self builders can easily turn them off as they are likely using their own options file anyways.

@vpeter4
Copy link
Contributor

vpeter4 commented May 9, 2022

Self builders can easily turn them off as they are likely using their own options file anyways.

Sorry but I can't resists - same applies for doing official builds: turn them on :)

@lrusak
Copy link
Member Author

lrusak commented May 9, 2022

Self builders can easily turn them off as they are likely using their own options file anyways.

Sorry but I can't resists - same applies for doing official builds: turn them on :)

We shouldn't be changing options from what is checked in when doing a release.

@HiassofT
Copy link
Member

HiassofT commented May 9, 2022

While there's some merit in having symbols for official builds it's hardly helpful for actually debugging things - -O2/O3/Os builds with debug info are pretty unusable as too much stuff is optimized out.

When I'm actually debugging I use DEBUG on select packages (which selects -Og so debug info is actually useful) together with SPLIT_DEBUG_INFO so I can cross-debug with source and full debug info without blowing up image size.

So I'd also say make that option off by default and maybe enable it on official CI builds - developers who want to have debugging symbol of optimized binary builds can then also op-tin to get them if they see any advantage in having them.

@mglae
Copy link
Contributor

mglae commented May 28, 2022

As test I've build Generic/x11 in the default configuration release with external symbols. Some experiences/notes:

Image full

A 733MB .system file was created being to large for the image. Most
size wa occupied by an 1.6GB libLLVM-13.so. With the executable flag not set
it was not stripped. I'll add a suggestion.

cmake build type

We have to be careful which cmake build type to choose. Not only that
packages do not allow default build type with CMAKE_BUILD_TYPE=""
they can very picky what is allowed.

See #6422 for media-driver only allowing Relese/Debug and the work around.

Another example is kodi only allowing LTO for release types but forcing stripping for Release.

#6422 using Release build type without having cmake options set is the most reliable default way for me. Getting symbols is only a minor feature.

external symbols

Handling of external symbols can be simplified by creating a squashfs image and mounting it. Even /usr/lib/debug can be used when created in the image.

gdb OOM

When using full external symbols gdb -p $(pidof kodi.bin) in a 4GB VM is terminated by OOM-Killer. It may be nice to have a feature to only create a subset of external symbols.

ccache

I'm wondering if default ccache size has to be increased after much larger object files are created.

Comment on lines +498 to +504
-type f -executable \
! -name "*.ko" \
! -name "ld.so" \
! -name "ld-*.so" \
! -name "libc-*.so" \
! -name "libpthread-*.so" \
! -name "libthread_db-*so" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion to strip all libs even without executable flag:

Suggested change
-type f -executable \
! -name "*.ko" \
! -name "ld.so" \
! -name "ld-*.so" \
! -name "libc-*.so" \
! -name "libpthread-*.so" \
! -name "libthread_db-*so" \
-type f \( -executable -o -name "*.so*" \) \
! -name "*.ko" \

@heitbaum
Copy link
Contributor

cmake build type

We have to be careful which cmake build type to choose. Not only that packages do not allow default build type with CMAKE_BUILD_TYPE="" they can very picky what is allowed.

See #6422 for media-driver only allowing Relese/Debug and the work around.

Another example is kodi only allowing LTO for release types but forcing stripping for Release.

#6422 using Release build type without having cmake options set is the most reliable default way for me. Getting symbols is only a minor feature.

#6641 is now another package that is picky of the CMAKE_BUILD_TYPE

@CvH CvH added LE 13.0 and removed LE 12.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants