Documentation and examples for various forms of split debug info

[VorpalBlade]

From what I can tell there are at least three types of split debug info:

1. Loose .dwo files
2. Packed .dwp files
3. Split after linking, using .gnu_debuglink and/or .note.gnu.build-id. This is commonly used with debuginfod for example. (Is there a DWARF 5 version of these attributes?)

Then how this works is different between DWARF 4 (GNU extensions) and DWARF 5 (in the standard).

However, the docs for how to properly handle this when parsing debug info with gimli are lacking and the types are confusing

• There is DwarfPackage, presumably only relevant to case 2 above.
• There is:
  • DW_AT_dwo_name (DW_AT_GNU_dwo_name)
  • DW_AT_GNU_dwo_id (but I don't see any such ID in DWARF 5 in Rust built skeleton units, not even for case 2).
  • Dwarf::set_sup etc. But I don't see any examples of how these are supposed to be used. Is all I need to do to handle case 3 to call set_sup?

It would be good to have better docs, or an example, that show how to handle all of these cases. For example, since at least case 1 & 2 are not relocated, is there are need to relocate address info?

[bjorn3]

I believe set_sup is for .dwo/.dwp. For debuginfo split after linking you would load the debuginfo as if the separate debuginfo object file is an executable/shared library with regular debuginfo sections. Splitting debuginfo after linking works by copying the debuginfo sections out of the linked executable/shared library verbatim.

For example, since at least case 1 & 2 are not relocated, is there are need to relocate address info?

For cases 1 & 2 only the half that remains inside the executable/shared library contains addresses (that are thus relocated). The .dwo/.dwp files don't contain addresses.

[VorpalBlade]

set_sup doesn't really seem to make sense for dwo to me:

1. You can only set one. But there are many dwo files.
2. Looking through the code at how sup is actually used, it seems to only be used for sup_string. So this must be for something entirely different (string table in a different file?)

And it doesn't make sense for dwp since there is DwarfPackage for that. Which is a different type than what set_sup expects (a normal Dwarf).

So set_sup is a mystery to me.

EDIT: There is also Dwarf::make_dwo, which seems to indicate you are supposed to open a Dwarf per dwo and set the parent of it using that method. It copies the sup from the parent, which again suggests that is unrelated to dwo.

[VorpalBlade]

I would really prefer a interface where I can throw all DWOs & DWPs in a pile, and have the library transparently load from the appropriate source as I iterate over DIEs, line tables etc.

@bjorn3 wrote:

For cases 1 & 2 only the half that remains inside the executable/shared library contains addresses (that are thus relocated). The .dwo/.dwp files don't contain addresses.

Ah I see, while DW_TAG_subprogram end up in dwo files; the DW_AT_low_pc/DW_AT_high_pc/DW_AT_ranges point back into sections of the main binary. This seems to be what make_dwo handles. And for DwarfPackage it seems you need to extract a Dwarf for the section you are interested in, then process that.

It isn't clear either how you are supposed to know if you should look for a dwp or dwo on the file system. Also, Rustc doesn't put any DwoId into the binary when using DWARF 5, meaning DwarfPackage::find_cu is useless. In fact there doesn't seem to be any way to iterate over DWOs in a DWP to handle this case. That seems fairly flawed. EDIT: My bad, it is a header field in DWARF 5, not an attribute.

[VorpalBlade]

Half an hour of research later: Supplementary object files (set_sup) is completely unrelated to dwo/dwp. See Section 7.3.6 of the DWARF 5 spec, and this blog post by maskray. It seems to be a way to optimise the debug info after linking (possibly after splitting as well, not sure) to deduplicate the data. (I hope I never come across it as I don't want to deal with it.)

Since dwarf data can also be compressed with normal compressors, this doesn't seem that useful.

[philipc]

backtrace-rs is the most complete example of loading all of these. The addr2line loader has some support but is still missing parts. wholesym should support all of these too, but I'm not familiar with its code.

For example, since at least case 1 & 2 are not relocated, is there are need to relocate address info?

Relocations only need to be handled for relocatable object files (.o files, this excludes .dwo). It is unusual to need this.

Supplementary object files (set_sup) is completely unrelated to dwo/dwp.

Yes, set_sup is for supplementary object files created by dwz. I think Fedora uses this still.

I would really prefer a interface where I can throw all DWOs & DWPs in a pile, and have the library transparently load from the appropriate source as I iterate over DIEs, line tables etc.

I agree there is a need to make this simpler for users. However, this is closely intertwined with the executable file parsing, which gimli has explicitly chosen to be independent from, so I don't think it would be possible to do exactly what you suggest here. I'm open to suggestions if you can see a way to implement it though. And certainly the documentation needs improving.

locate-dwarf is meant to help somewhat with this, but I've not used it. One of my long term goals is to use this to complete the support in addr2line.

[VorpalBlade]

locate-dwarf is meant to help somewhat with this, but I've not used it. One of my long term goals is to use this to complete the support in addr2line.

I had a look at it, and it only seems to have reasonable code for MacOS X unfortunately (though I'm not sure about this, as I don't know MacOS X at all):

• There is no support for debuginfod which has become the defacto standard on Linux in recent years.
• There is windows PDB support, but symbol server support is missing as well from what I can tell (though I haven't used windows in over a decade for development, I could be missing something).
• It seems to have some hard coded behaviour for the Rust target/ directory structure, making it fairly useless if you are debugging something other than Rust code (I want to be able to debug C++, C and Rust). It also isn't useful if you are installing the binary rather than running it from your target directory. I'm interested in debugging on deployed systems (embedded/industrial Linux computers where you deploy code to over the network).
• No support for supplementary files.

Looking at it, backtrace-rs has much more reasonable code for the Linux case, though it still lacks debuginfod support for debug info for system libraries. But given the use case (in-process back traces) that might be sensible.

So neither of these is appropriate to the debugger use case, but backtrace-rs is a better starting point (and I'm borrowing code from it).

One thing I have no clue about yet though is how to locate .dwp vs .dwo. With .dwo the paths to the .dwo files are embedded in the skeleton units / special GNU extensions for DWARF 4. But those are still there and unchanged when using dwp. So it seems I need to somehow locate the dwp and determine I should use that instead of the dwo?

[VorpalBlade]

wholesym seems more complete, but I haven't had time to dive deep into it. From a quick glance it seems to only expose looking up symbols given addresses though. I need full access to the underlying DWARF, including the dreaded dwarf expressions (which I will need to symbolically execute to generate the equivalent eBPF code, the gimli approach as a co-routine program is not really suitable for my use case, but at least I can get at the raw opcodes).

Interestingly it does not depend on gimli, so I assume it uses it's own parsing.

[philipc]

So it seems I need to somehow locate the dwp and determine I should use that instead of the dwo?

I don't know a better way than: use the dwp if you have it, otherwise try to find the dwo.

Interestingly it does not depend on gimli, so I assume it uses it's own parsing.

It uses addr2line for the DWARF parsing, but not for the loading. It also has PDB support via pdb-addr2line.