Beyla interferes with IPv6 SSL connectivity with `context_propagation: all` enabled

[paweljw]

What's wrong?

When context_propagation: all is enabled, certain SSL connections may fail with Connection reset by peer, despite the peer being up and responsive. This seems to be worsened/primarily occur for hosts with ~moderate latency (e.g. connections from US to EU).

It appears to specifically occur when these items are loaded:

perf_event name pyperf_collect
perf_event name read_python_stack
sched_cls name obi_app_egress
sched_cls name obi_app_ingress
sk_msg name obi_packet_extender
sk_msg name obi_packet_extender_write_msg_tp
sock_ops name obi_sockmap_tracker

No apparent error/related stdout logs from Beyla.

Steps to reproduce

• obtain a host that's in the US (I used a cheap Hetzner host to reproduce)
• deploy Beyla with Docker (see attached configuration)
• curl --fail -6 https://mc.pjw.lol -o /dev/null <- this host is physically located in Europe
  • may take a couple restarts of Beyla or a few tries (some timing bug?)
  • feel free to use this host in testing!
• does NOT reproduce with curl -4 ...

System information

Linux 6.8.0-79-generic #79-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 12 14:42:46 UTC 2025 x86_64 GNU/Linux

Software version

Grafana Beyla 2.7.5 (official Docker image)

Configuration

discovery:
  poll_interval: 30s
  min_process_age: 30s

  instrument:
    - open_ports: "1-65535"
  
  exclude_instrument:
    # some irrelevant exclusions omitted

otel_metrics_export:
  features: []
  instrumentations: []

prometheus_export:
  port: 0

network:
  enable: false

otel_traces_export:
  instrumentations:
    - "*"

ebpf:
  context_propagation: all
  track_request_headers: true

Logs

curl -6 --fail https://mc.pjw.lol -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) Recv failure: Connection reset by peer

curl -4 --fail https://mc.pjw.lol -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2386    0  2386    0     0   3692      0 --:--:-- --:--:-- --:--:--  3687

[grcevski]

Thanks for reporting this! Can you confirm if it's possible that you don't see the problem with context_propagation: headers, rather than all?

I'm suspecting that we have a bug with the injection of the trace context in the IP options (since that's our only option for SSL) when IPv6 is enabled. Maybe what we picked as an option to store is not correct, or some of the flags calculations are incorrect.

I'll give this a shot shortly if you don't mind keeping the server on for a bit :)

[paweljw]

@grcevski

Can you confirm if it's possible that you don't see the problem with context_propagation: headers, rather than all?

Yes - it appears that with headers or disabled this no longer reproduces.

I'll give this a shot shortly if you don't mind keeping the server on for a bit :)

No worries at all! It's a non-essential machine and it's on ~permanently, feel free to poke it whenever handy. There generally shouldn't be mechanisms that'd rate-limit incoming connections of a semi-reasonable volume :)

Thank you for the prompt response! <3

[paweljw]

One more note! Since I have a source/target setup where I can reproduce this pretty reliably, I'm also happy to test any patches. E.g. pulling a branch, building a Docker image and verifying it with my existing setup is not a problem.

[rafaelroquetto]

Hey @paweljw are you able to test the latest OBI main image? open-telemetry/opentelemetry-ebpf-instrumentation#933 should have resolved it.

The images are here: https://hub.docker.com/r/otel/ebpf-instrument/tags

Let me know, otherwise I will see if I get to reproduce it locally. Thanks!

[paweljw]

Hi @rafaelroquetto ! Thank you for reaching out. I tested it. Unfortunately I don't have great news :(

Same testing instance as in the original report, nothing else running on the host.

image version (otel/ebpf-instrument:main):

"otel/ebpf-instrument@sha256:b15033e17907391cf945a54584013d398ea78c49bfd198ef0922349251f35308"

otel.yaml:

discovery:
  instrument:
  - open_ports: 1-65535
trace_printer: text
ebpf:
  context_propagation: all

running with:

docker run --rm -v ./otel.yaml:/otel.yaml --network=host --pid=host --privileged otel/ebpf-instrument:main -config /otel.yaml

test loop:

while true; do 
	date
	(set -o pipefail; docker ps | grep -e otel | awk '{print $2}' || echo "otel off")
	curl -f -6 https://mc.pjw.lol -o /dev/null || true
	sleep 1
done

result:

Fri Nov 28 06:59:44 PM UTC 2025
otel/ebpf-instrument:main
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) Recv failure: Connection reset by peer

It's unfortunately as persistent as it was - 95-100% connections are reset, and it goes away the moment I kill the ebpf-instrument container. 😢

Let me know if there's something else you'd like me to try please 🙏

[rafaelroquetto]

@paweljw thank you so much for the swift response and testing. I will continue to investigate this further (I wonder if there's something affecting the IP header checksum and if that could be causing the problem).

In the meantime, for your needs, you can try context_propagation: "headers, tcp" - the TCP context propagation was introduced a few days ago as a (hopefully) better alternative to the IP one. If you do try it out, could you let me know if that works fine for you?
Thanks again!

[paweljw]

@rafaelroquetto I can confirm that with:

discovery:
  instrument:
  - open_ports: 1-65535
trace_printer: text
ebpf:
  context_propagation: "headers,tcp"

I can no longer reproduce the connection resets on otel/ebpf-instrument:main. 🎉

Do you have a sense for when these options will be supported in Beyla, rather than the "raw" OBI image?

Also is there documentation you could point me towards (or code I could read) to understand the differences between these context propagation options?

Being able to use more than headers would be a massive unlock, but I'd like to understand the difference in capability between all and headers,tcp in detail before I proceed. Thank you! 🙏

[rafaelroquetto]

Hey @paweljw, glad to hear this worked around for you - and thanks again for helping us with testing.

We are about to release Beyla 2.8 so that functionality has missed the cut for it, but I am hoping we can include it on Beyla 2.9 in the next weeks - although I don't have an exact time frame yet, but I will keep you posted here.

With regards to documentation, https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/main/devdocs/context-propagation.md is a good starting point.

In a nutshell though:

• headers try to inject Traceparent: in the HTTP headers - as such, it only works with plaintext HTTP - it doesn't work with SSL.
• tcp injects the traceparent in the TCP header - that means we can support SSL.
• ip is the fallback when tcp fails - it's the least reliable option as routers and proxies tend to strip IP options (and as you noticed, it's currently broken).

[rafaelroquetto]

Hey @paweljw - just an update, I am hoping that open-telemetry/opentelemetry-ebpf-instrumentation#991 will address the issue. This PR is still WIP and I still need to do proper IPv6 testing - I thought I'd let you know just in case - if you happen to have spare time (but please, do not feel obliged to), I've pushed a docker image to https://hub.docker.com/r/rafaelroquetto/obi/tags containing the changes.

I will continue to keep you posted.