Beyla causing SIGSEGV and SIGILL in Go applications

[alexchowle]

What's wrong?

I have multiple applications (HTTP servers) compiled against Go v1.19.6 and am using Beyla to instrument them.

When Beyla is instrumenting the applications it can cause any one of them to receive either a SIGSEGV or a SIGILL. When Beyla is not instrumenting, there is no issue. The application it affects can be different each time I experience the problem. The problem is not 100% reproducible.

These applications have a bespoke deployment and startup process wherein, for a short period of time, there will be a version-A and version-B running side-by-side (from different base directories), each with a different TCP port. So there will be 2x PIDs for the same process name. The problem occurs as part of my deployment process; if I was not to release any new versions of a service and Beyla was already successfully monitoring the required processes I would not experience the problem.

Steps to reproduce

Proprietary deployment process - Beyla is already running at the time of the issue.

System information

Linux 6.8.0-85-generic #85-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 18 15:26:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Software version

Grafana Beyla v2.7.0

Configuration

log_level: INFO
discovery:
  services:
    - exe_path: ${SERVICES_INSTALLATION_BASE_DIR}
      namespace: ${NAMESPACE}
 
ebpf:
  context_propagation: headers
  track_request_headers: true
 
otel_metrics_export:
  endpoint: ${METRICS_HTTP_ENDPOINT}
  interval: 30s
  allow_service_graph_self_references: true
  reporters_cache_len: 50
  instrumentations: ["*"]
 
otel_traces_export:
  endpoint: ${TRACES_HTTP_ENDPOINT}

Logs

App #1
======
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x6ad227]goroutine 55 [running]:
runtime.throw({0x88bcf6?, 0x10?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc0002079a0 sp=0xc000207970 pc=0x43925d
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:842 +0x2c5 fp=0xc0002079f0 sp=0xc0002079a0 pc=0x44fa85
net/http.(*Server).readHeaderTimeout(...)
        /usr/local/go/src/net/http/server.go:3195
net/http.(*conn).readRequest(0xc0000005a0, {0x944788, 0xc0000a5dc0})
        /usr/local/go/src/net/http/server.go:975 +0x87 fp=0xc000207b78 sp=0xc0002079f0 pc=0x6ad227
net/http.(*conn).serve(0xc0000005a0, {0x944830, 0xc000148fc0})
        /usr/local/go/src/net/http/server.go:1916 +0x345 fp=0xc000207fb8 sp=0xc000207b78 pc=0x6b1925
net/http.(*Server).Serve.func3()
        /usr/local/go/src/net/http/server.go:3102 +0x2e fp=0xc000207fe0 sp=0xc000207fb8 pc=0x6b6e4e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000207fe8 sp=0xc000207fe0 pc=0x46cfa1
created by net/http.(*Server).Serve


App #2
======
unexpected fault address 0xbfcc0be4b8
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xbfcc0be4b8 pc=0x6796fa]

goroutine 66 [running]:
runtime.throw({0x88bce7?, 0xca0?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc0000a7858 sp=0xc0000a7828 pc=0x43925d
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:842 +0x2c5 fp=0xc0000a78a8 sp=0xc0000a7858 pc=0x44fa85
compress/gzip.(*Writer).Close(0xc0000be420)
        /usr/local/go/src/compress/gzip/gzip.go:243 +0xba fp=0xc0000a78e0 sp=0xc0000a78a8 pc=0x6796fa
runtime/pprof.(*profileBuilder).build(0xc00022fce0)
        /usr/local/go/src/runtime/pprof/proto.go:391 +0x379 fp=0xc0000a79f0 sp=0xc0000a78e0 pc=0x741639
runtime/pprof.writeHeapProto({0x949e60?, 0xc00026a540?}, {0xc000418000, 0x8, 0x0?}, 0x0?, {0x0, 0x0})
        /usr/local/go/src/runtime/pprof/protomem.go:65 +0x305 fp=0xc0000a7bf8 sp=0xc0000a79f0 pc=0x746b45
runtime/pprof.writeHeapInternal({0x949e60?, 0xc00026a540?}, 0x0, {0x0, 0x0})
        /usr/local/go/src/runtime/pprof/pprof.go:575 +0x305 fp=0xc0000a9650 sp=0xc0000a7bf8 pc=0x73bea5
runtime/pprof.writeHeap({0x949e60?, 0xc00026a540?}, 0xc00023a270?)
        /usr/local/go/src/runtime/pprof/pprof.go:535 +0x27 fp=0xc0000a9688 sp=0xc0000a9650 pc=0x73bb07
runtime/pprof.(*Profile).WriteTo(0x85d320?, {0x949e60?, 0xc00026a540?}, 0x13?)
        /usr/local/go/src/runtime/pprof/pprof.go:330 +0x14b fp=0xc0000a97b0 sp=0xc0000a9688 pc=0x73a10b
net/http/pprof.handler.ServeHTTP({0xc000276051, 0x4}, {0x94c708, 0xc00026a540}, 0x736f686c61636f6c?)
        /usr/local/go/src/net/http/pprof/pprof.go:253 +0x4a5 fp=0xc0000a9888 sp=0xc0000a97b0 pc=0x7492e5
net/http/pprof.Index({0x94c708?, 0xc00026a540}, 0xc000242c00)
        /usr/local/go/src/net/http/pprof/pprof.go:371 +0x13e fp=0xc0000a9a50 sp=0xc0000a9888 pc=0x749e7e
net/http.HandlerFunc.ServeHTTP(0xc000058af0?, {0x94c708?, 0xc00026a540?}, 0x0?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f fp=0xc0000a9a78 sp=0xc0000a9a50 pc=0x6d5c2f
net/http.(*ServeMux).ServeHTTP(0x0?, {0x94c708, 0xc00026a540}, 0xc000242c00)
        /usr/local/go/src/net/http/server.go:2487 +0x149 fp=0xc0000a9ac8 sp=0xc0000a9a78 pc=0x6d75c9
net/http.serverHandler.ServeHTTP({0xc000226b40?}, {0x94c708, 0xc00026a540}, 0xc000242c00)
        /usr/local/go/src/net/http/server.go:2947 +0x30c fp=0xc0000a9b78 sp=0xc0000a9ac8 pc=0x6d902c
net/http.(*conn).serve(0xc0002403c0, {0x94cca8, 0xc000226300})
        /usr/local/go/src/net/http/server.go:1991 +0x607 fp=0xc0000a9fb8 sp=0xc0000a9b78 pc=0x6d4707
net/http.(*Server).Serve.func3()
        /usr/local/go/src/net/http/server.go:3102 +0x2e fp=0xc0000a9fe0 sp=0xc0000a9fb8 pc=0x6d996e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a9fe8 sp=0xc0000a9fe0 pc=0x46cfa1
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3102 +0x4db


App #3
======
SIGILL: illegal instruction
PC=0x6b61e0 m=3 sigcode=2
instruction bytes: 0x4c 0x8d 0xcc 0x24 0x10 0xff 0xff 0xff 0x4d 0x3b 0x66 0x10 0xf 0x86 0xa1 0x4

goroutine 285 gp=0xc000485880 m=3 mp=0xc000057008 [running]:
runtime/pprof.allFrames(0x79881a25fec8)
        /usr/local/go/src/runtime/pprof/proto.go:209 fp=0xc002619b08 sp=0xc002619b00 pc=0x6b61e0
runtime/pprof.(*profileBuilder).appendLocsForStack(0xc00010ef20, {0x0, 0x0, 0x0}, {0x79881a25fec8, 0xc, 0xc})
        /usr/local/go/src/runtime/pprof/proto.go:451 +0x36f fp=0xc002619c90 sp=0xc002619b08 pc=0x6b794f
runtime/pprof.writeHeapProto({0xa6bf20?, 0xc002449420?}, {0xc00261c008, 0x92, 0x0?}, 0x80000, {0x0, 0x0})
        /usr/local/go/src/runtime/pprof/protomem.go:47 +0x66e fp=0xc002619e10 sp=0xc002619c90 pc=0x6bca8e
runtime/pprof.writeHeapInternal({0xa6bf20, 0xc002449420}, 0x0, {0x0, 0x0})
        /usr/local/go/src/runtime/pprof/pprof.go:634 +0x2a5 fp=0xc00261b758 sp=0xc002619e10 pc=0x6b1a25
runtime/pprof.writeHeap({0xa6bf20?, 0xc002449420?}, 0x29?)
        /usr/local/go/src/runtime/pprof/pprof.go:593 +0x25 fp=0xc00261b790 sp=0xc00261b758 pc=0x6b16e5
runtime/pprof.(*Profile).WriteTo(0xd5e540?, {0xa6bf20?, 0xc002449420?}, 0x13?)
        /usr/local/go/src/runtime/pprof/pprof.go:377 +0x14b fp=0xc00261b890 sp=0xc00261b790 pc=0x6af94b
net/http/pprof.handler.ServeHTTP({0xc0004805b1, 0x4}, {0xa6edb8, 0xc002449420}, 0xc00240d900)
        /usr/local/go/src/net/http/pprof/pprof.go:272 +0x52f fp=0xc00261b948 sp=0xc00261b890 pc=0x6c918f
net/http/pprof.Index({0xa6edb8, 0xc002449420}, 0xc00240d900?)
        /usr/local/go/src/net/http/pprof/pprof.go:389 +0xde fp=0xc00261baf0 sp=0xc00261b948 pc=0x6c9c3e
net/http.HandlerFunc.ServeHTTP(0xd701e0?, {0xa6edb8?, 0xc002449420?}, 0x652eb6?)
        /usr/local/go/src/net/http/server.go:2294 +0x29 fp=0xc00261bb18 sp=0xc00261baf0 pc=0x65a929
net/http.(*ServeMux).ServeHTTP(0x473059?, {0xa6edb8, 0xc002449420}, 0xc00240d900)
        /usr/local/go/src/net/http/server.go:2822 +0x1c4 fp=0xc00261bb68 sp=0xc00261bb18 pc=0x65c7c4
net/http.serverHandler.ServeHTTP({0xc000278d80?}, {0xa6edb8?, 0xc002449420?}, 0x1?)
        /usr/local/go/src/net/http/server.go:3301 +0x8e fp=0xc00261bb98 sp=0xc00261bb68 pc=0x67810e
net/http.(*conn).serve(0xc0024d8e10, {0xa6f2c8, 0xc0024f0510})
        /usr/local/go/src/net/http/server.go:2102 +0x625 fp=0xc00261bfb8 sp=0xc00261bb98 pc=0x659085
net/http.(*Server).Serve.gowrap3()
        /usr/local/go/src/net/http/server.go:3454 +0x28 fp=0xc00261bfe0 sp=0xc00261bfb8 pc=0x65e548
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00261bfe8 sp=0xc00261bfe0 pc=0x47e221
created by net/http.(*Server).Serve in goroutine 1
        /usr/local/go/src/net/http/server.go:3454 +0x485

[grcevski]

Thanks for reporting this @alexchowle! I'm thinking that somehow we mess up when we attach the uprobes to the process, because they do patch the binary to inject the hooks.

So just to confirm, these two programs are named the same, but they are from two different binaries, e.g. they are in two different directories, but the binary is absolutely a different file? So I'm guessing you are waiting for the second binary to start-up and become active and then you kill the old version?

Do you know if the binary that crashes is the old one or the new one?

One thing we can try is to see if we can workaround the problem for the time being with delaying or not delaying instrumentation. Can you try to adjust our min_process_age setting to see if the problem goes away?

One option to try is to increase this time from the default of 5 seconds to something like 15 or 30 seconds, like this:

discovery:
  min_process_age: 15s

Another thing to try is to see if you can set it to 0s, like this:

discovery:
  min_process_age: 0s

Please let us know how it goes.

[alexchowle]

Thanks for reporting this @alexchowle! I'm thinking that somehow we mess up when we attach the uprobes to the process, because they do patch the binary to inject the hooks.

So just to confirm, these two programs are named the same, but they are from two different binaries, e.g. they are in two different directories, but the binary is absolutely a different file? So I'm guessing you are waiting for the second binary to start-up and become active and then you kill the old version?

Do you know if the binary that crashes is the old one or the new one?

One thing we can try is to see if we can workaround the problem for the time being with delaying or not delaying instrumentation. Can you try to adjust our min_process_age setting to see if the problem goes away?

One option to try is to increase this time from the default of 5 seconds to something like 15 or 30 seconds, like this:

discovery:
  min_process_age: 15s

Another thing to try is to see if you can set it to 0s, like this:

discovery:
  min_process_age: 0s

Please let us know how it goes.

Thanks for the quick reply, @grcevski.

• The two programs are named the same, are ran from two different locations, and are different files.
• The binary that crashes is the new one. You are correct in that the second binary starts and, once up, the original is killed.

[alexchowle]

@grcevski - I've set min_process_age: 15s and am unable to reproduce the issue. Would you be able to explain the significance of this setting, and how to set it optimally?

[grcevski]

Thanks for reporting back on this @alexchowle. So this option for delay is something we use to avoid instrumenting processes that won't live very long. It's an optimisation. For example if there was a short lived, we have to do a lot of work to parse executable symbols, attach probes etc, just to have the process terminate before we are able to finish the work. So we introduced a delay of 5 seconds, that we wait to see if the process is still live after we've discovered it.

So it seems that there's a problem in when we attach the probes into the new executable. Adding a bit more delay means we attach a bit later presumably after the executable is now in a good shape.

I don't know what would cause the failure yet, it's like the code patching that the ebpf infrastructure does is unsafe at certain points. We'll investigate this.

[grcevski]

Forgot to answer the optimal value question. I think it depends on how fast do you expect the new program to start and become available. I would try lowering the value to 10 seconds. There has to be a specific moment where things are unsafe to patch with probes.

Anything more you can tell us about how these are launched, a script or a launcher program? I see pprof being enabled, perhaps that pays a roles somehow, any other tools or perhaps multiple Beyla installations?

[grcevski]

I think I may have an idea about what's happening. We are setting the probes in a wrong location of the file, which means we somehow have offsets of the symbols where we attach from a stale file. We keep track of this by the system inode number, which is unique (for a device at least) and we clean-up this information when the last executable of a given program dies. However, eyeballing the code with @rafaelroquetto we noticed that we are not cleaning a symbols cache, which was added for performance reasons.

Namely I think the following scenario might happen:

1. You have one current version with inode = 1 and you deploy a new executable with inode = 2. So far so good, I don't think we'll crash here.
2. Next your version with inode = 2 becomes current, and I'm assuming you are deleting the old binary now, which means inode = 1 can be recycled by the system.
3. Then you deploy one more version of your program to replace inode = 2 and the new version now gets again inode = 1. If we haven't cleaned up the symbols cache, we could potentially pick the symbol offsets of the executable 2 versions ago. This will cause the crash.

The fix is relatively straightforward. I'll start working on this tomorrow and we'll release a patch version.