interpreter: add Golang #408

florianl · 2025-03-17T17:15:36Z

Add symbolization functionality for Go executables.

fabled

Some comments added. But before just blindly starting mixing Rust, I'd like a larger discussion a comments from all maintainers on feasibility of this.

I understand using existing Rust code might be faster, and the Rust code is likely good and tested.

On the other hand this the down sides are:

core maintainers should also know Rust (possibly true?)
CGO is slower than regular Go calls. not sure of the overhead.
there is room for non-trivial errors as the calls need to use unsafe Go, potentially in non-trivial ways

Additionally go symbolization could be done directly using Go standard runtime libraries, or by extending the existing elfgopclntab code we have intree.

Could you elaborate why embedded Rust was chosen instead of the two above mentioned approaches? Do these reasons justify the overhead that mixing Go and Rust brings both in maintanenance, complexity and debugging?

interpreter/golang/golang.go

fabled · 2025-03-17T19:21:01Z

processmanager/manager.go

@@ -267,6 +267,21 @@ func (pm *ProcessManager) ConvertTrace(trace *host.Trace) (newTrace *libpf.Trace
 				}
 			}

+			if pm.eim.IsGolang(frame.File) {


This is fairly ugly way to plug this in. Makes me wonder if the integration should be done in some other way. Perhaps split the interpreter into unwinder / symbolization stages. Or have some other hooking mechanism to tag the frames that should get Go symbolized. Perhaps add a mechanism to specify the ebpf sent frame type on per-DSO or mapping range basis.

Happy to discuss alternatives. For the moment, I think, this is the least intrusive way. This current approach should also be easily adaptable if the interpreter interface differentiates at some point between unwinding and symbolization.

To clarify, I really don't like the special golang check here. As minimum, I'd completely remove the IsGolang call above (And introduction if it in eim). We can just directly call symbolizeFrame here. And make sure the existing interpreters ignore native frame (should be the case).

Or do you see some practical reasons why this check should be here?

Your suggestion sounds reasonable to me. It just needs some logic on the eBPF side to report Go frames for the respective executables. And I do think introducing a dedicated Go unwinder in eBPF doesn't make sense but the native unwinder needs to be extended in some way to switch the frame type at some point.

Applied the suggestion with 7128306

christos68k · 2025-03-17T19:47:42Z

Could you elaborate why embedded Rust was chosen instead of the two above mentioned approaches? Do these reasons justify the overhead that mixing Go and Rust brings both in maintanenance, complexity and debugging?

Besides the main reasons you outlined (Rust code performant/safe/well tested), using symblib allows us to bring this feature to production quickly, and solves a number of issues we have with deployed Go binaries. We do plan to also use symblib for extracting/uploading symbols from other native (non-Go) binaries, albeit this doesn't need to take place in-process in the agent, and can be a separate utility.

Regardless, I'm not opposed to doing Go symbolization in Go, assuming someone implements it we could evaluate and transition. But we don't have that right now (and at least at Elastic, have no plans to work on it) and we do have symblib.

fabled · 2025-03-17T20:54:29Z

using symblib allows us to bring this feature to production quickly, and solves a number of issues we have with deployed Go binaries

If going this way, I'd rather then use symblib as the native code dynamic symbolizer. And have it symbolize native code. We could limit to using dynamic symbols only if wanted. But getting some symbols for C-code with this would make sense. Rather than special case this for Golang only.

Regardless, I'm not opposed to doing Go symbolization in Go, assuming someone implements it we could evaluate and transition. But we don't have that right now (and at least at Elastic, have no plans to work on it) and we do have symblib.

Also looking at Rust code a bit, it seems to mmap files which is sort of good, but also previously unused. Also since Go binaries can contain C-code and Dwarf, the symblib dwarf code will get used for these executable bringing a lot of potentially new unexpected things: like automatically uncompressing the compressed dwarf to large temporary files. Edit: I missed that the Rust API used only looks at gopclntab.

IMHO the symblib code would probably need a mode the host agent can safely use for all binaries to do dynamic/elf/gopclntab symbolization but skip dwarf and compressed things. The integration as "native symbolizer" would be much cleaner than trying to special Golang.

florianl · 2025-03-18T06:59:07Z

IMHO the symblib code would probably need a mode the host agent can safely use for all binaries to do dynamic/elf/gopclntab symbolization but skip dwarf and compressed things. The integration as "native symbolizer" would be much cleaner than trying to special Golang.

That is the general idea going forward. At the moment the scope is limited and focused on Go as from this ecosystem there is the highest demand for Symbols.

florianl · 2025-03-18T08:32:47Z

interpreter/golang/golang.go

+	}
+
+	frameID := libpf.NewFrameID(libpf.NewFileID(uint64(frame.File), uint64(frame.File)),
+		libpf.AddressOrLineno(symbolsSlice[0].line_number))


I switched the AddressOrLineno part of frameID with 5bf7fd7. Previously pc was used, but this generated too much different frame IDs for the same function/source file/source line combination.

This does prevent using the data for PGO. Do you have statistics?

I didn't collect statistics. Like everywhere else for the on CPU sampling approach, leave frames are different which causes a high number of variance and differences in frame IDs.

fabled

Some additional new comments to clean up a bit.

Also, how much the profiler executable size goes up with this now that it pulls static Rust stuff?

interpreter/golang/golang.go

processmanager/manager.go

fabled · 2025-03-18T10:00:03Z

interpreter/golang/golang.go

+	frameID := libpf.NewFrameID(libpf.NewFileID(uint64(frame.File), uint64(frame.File)),
+		libpf.AddressOrLineno(symbolsSlice[0].line_number))
+
+	trace.AppendFrameID(libpf.GolangFrame, frameID)


Should this provide the mapping information similar as the non-symbolized native code is done?

At this point Go frames are treated as interpreter and no longer as native code. So I guess the answer here is no, and mapping information is not provided similar to other native code.

fabled · 2025-03-18T10:00:46Z

interpreter/golang/golang.go

+	}
+
+	frameID := libpf.NewFrameID(libpf.NewFileID(uint64(frame.File), uint64(frame.File)),
+		libpf.AddressOrLineno(symbolsSlice[0].line_number))


This does prevent using the data for PGO. Do you have statistics?

fabled · 2025-03-18T10:35:24Z

A little note that also #407 adds golang interpreter that does quite different things.

florianl · 2025-03-18T10:50:36Z

Also, how much the profiler executable size goes up with this now that it pulls static Rust stuff?

#408 @ 7616b6c

$ ls -lia ebpf-profiler 
42649440 -rwxr-xr-x 1 user user 32535648 Mar 18 11:46 ebpf-profiler

main @ 422bd6b

$ ls -lia ebpf-profiler 
42649394 -rwxr-xr-x 1 user user 30313560 Mar 18 11:48 ebpf-profiler

Add symbolization functionality for Go executables. Signed-off-by: Florian Lehner <[email protected]>

Signed-off-by: Florian Lehner <[email protected]>

Co-authored-by: Timo Teräs <[email protected]>

Signed-off-by: Florian Lehner <[email protected]>

florianl · 2025-03-20T11:08:44Z

Force pushed to resolve merge conflict with main branch.

florianl force-pushed the interpreter-golang branch from 5ad982b to 9078329 Compare March 17, 2025 18:22

florianl marked this pull request as ready for review March 17, 2025 18:32

florianl requested review from a team as code owners March 17, 2025 18:32

fabled reviewed Mar 17, 2025

View reviewed changes

florianl commented Mar 18, 2025

View reviewed changes

fabled reviewed Mar 18, 2025

View reviewed changes

fabled mentioned this pull request Mar 18, 2025

Custom labels for native and go #407

Closed

florianl and others added 12 commits March 20, 2025 12:01

interpreter: add Golang

43e608a

Add symbolization functionality for Go executables. Signed-off-by: Florian Lehner <[email protected]>

fixup: add rust-components to more targets

0be1e34

Signed-off-by: Florian Lehner <[email protected]>

fixup: just log errors in executablemanager

76d4975

Signed-off-by: Florian Lehner <[email protected]>

fixup: use native handling as fallback if Go symbolization fails

002c023

Signed-off-by: Florian Lehner <[email protected]>

fixup: drop IsGolang()

3310143

Signed-off-by: Florian Lehner <[email protected]>

fixup: use interpreter.InstanceStubs

1c10cc0

Signed-off-by: Florian Lehner <[email protected]>

fixup: drop LRU and use resolved line number instead of pc for frameID

20d0f13

Signed-off-by: Florian Lehner <[email protected]>

Update processmanager/manager.go

2274e1c

Co-authored-by: Timo Teräs <[email protected]>

fixup: drop NOP comment

fb5490b

Signed-off-by: Florian Lehner <[email protected]>

fixup: reword Golang to Go

c577720

Signed-off-by: Florian Lehner <[email protected]>

fixup: s/Golang/Go/

b4dd87e

Signed-off-by: Florian Lehner <[email protected]>

fixup: fix frame ID

9a26452

Signed-off-by: Florian Lehner <[email protected]>

florianl force-pushed the interpreter-golang branch from 3222c8d to 9a26452 Compare March 20, 2025 11:08

florianl force-pushed the interpreter-golang branch 3 times, most recently from 8b5ea8d to 77bd162 Compare March 21, 2025 11:04

florianl force-pushed the interpreter-golang branch 8 times, most recently from 8be0993 to 9a26452 Compare March 21, 2025 11:27

florianl mentioned this pull request Mar 21, 2025

Rust: add binary blobs #415

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

interpreter: add Golang #408

interpreter: add Golang #408

florianl commented Mar 17, 2025

fabled left a comment

fabled Mar 17, 2025

florianl Mar 18, 2025

fabled Mar 18, 2025

florianl Mar 18, 2025

florianl Mar 18, 2025

christos68k commented Mar 17, 2025 •

edited

Loading

fabled commented Mar 17, 2025 •

edited

Loading

florianl commented Mar 18, 2025

florianl Mar 18, 2025

fabled Mar 18, 2025

florianl Mar 18, 2025

fabled left a comment

fabled Mar 18, 2025

florianl Mar 18, 2025

fabled Mar 18, 2025

fabled commented Mar 18, 2025

florianl commented Mar 18, 2025

florianl commented Mar 20, 2025

interpreter: add Golang #408

Are you sure you want to change the base?

interpreter: add Golang #408

Conversation

florianl commented Mar 17, 2025

fabled left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christos68k commented Mar 17, 2025 • edited Loading

fabled commented Mar 17, 2025 • edited Loading

florianl commented Mar 18, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabled left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabled commented Mar 18, 2025

florianl commented Mar 18, 2025

florianl commented Mar 20, 2025

christos68k commented Mar 17, 2025 •

edited

Loading

fabled commented Mar 17, 2025 •

edited

Loading