Skip to content

Conversation

@pinoOgni
Copy link

@pinoOgni pinoOgni commented Jan 27, 2026

This PR introduces the appnetworktracer, which allows you to attach custom ebpf probes to calculate network metrics related to an instrumented service.

The first metric implemented is the TCP RTT using tcp_close.
If the event is of interest to us, it is sent to userspace using a different ringbuf than the generic one.
The event is then converted to Span and the already established pipeline (decorator and others) is used.

I still have a few things to fix, but this is the data flow from kernel -> user -> export.

This is an example of metric:

# HELP obi_net_tcp_rtt_seconds mearures the smoothed TCP RTT as calculated by the kernel in seconds
# TYPE obi_net_tcp_rtt_seconds histogram
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.0005"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.001"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.002"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.005"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.01"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.025"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.05"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.1"} 0
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.25"} 5
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="0.5"} 5
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="1"} 5
obi_net_tcp_rtt_seconds_bucket{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace="",le="+Inf"} 5
obi_net_tcp_rtt_seconds_sum{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace=""} 1.0150000000000001
obi_net_tcp_rtt_seconds_count{instance="lima-ubuntu-ebpf:1671691",job="main",netns="4026531840",port="9092",service_name="main",service_namespace=""} 5

@NimrodAvni78
Copy link

can we introduce another label of destination_service and destination_namespace (maybe think of naming)?
its interesting to see this network data related to which service we are sending data to

@pinoOgni
Copy link
Author

can we introduce another label of destination_service and destination_namespace (maybe think of naming)? its interesting to see this network data related to which service we are sending data to

destination_service may be something that can be extracted from the various decorator, not sure about destination_namespace but I'll check

@pinoOgni pinoOgni force-pushed the pgn/netmetrics branch 4 times, most recently from 765fa58 to b44c080 Compare January 28, 2026 10:20
return segment, bit
}

func (p *Tracer) buildPidFilter() []uint64 {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need all this pidfilter related code? aren't we reusing the generictracer's one?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed, the appnetworktracer and the generictracer modify the same PID map and can be attacked together or alone, so I simply added a check in the attacher: I check that the generictracer is present and pass the flag to the appnetworktracer. This way, the various methods that work on the PID map check whether the generictracer is already present.
Not use if this is the best idea

exported with prom and otel

Signed-off-by: Giuseppe Ognibene <[email protected]>
add iters in the appnetworktracer

Signed-off-by: Giuseppe Ognibene <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants