Unable to Perform eBPF Profiling in Showcase Demo with Latest OAP, UI, and Rover Image

[KHADEPALLAVI975]

I'm deploying the Apache SkyWalking Showcase demo using the following setup command:

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover,events,als,cilium

This brings up the environment with mesh observability, Cilium, ALS, and Rover enabled. The deployment is successful and I’m seeing normal service mesh metrics like:

Load , Success rate , Latency

All Kubernetes pods are running, and the OAP, UI, and Rover components are using the latest images.

❌ What's not working:

No processes are listed under the eBPF profiling section.
Flame graphs do not appear in the UI.

My Setup:

OAP TAG : e820c75
UI TAG : e820c75
Rover TAG : 67622c352b98fd32782a3e7afc7d3fbd6d6ec8e3
Kubernetes environment: KIND
Kernel version: 5.15.167-0515167-generic
Rover deployment: DaemonSet

Questions:

Are there extra configuration steps needed for eBPF profiling to work in the showcase?
Has anyone managed to get flame graphs showing up with the latest Rover + OAP combo?
Could this be a known issue or compatibility mismatch between components?

Happy to provide logs or config snippets if needed — any help or pointers would be much appreciated!

[mrproliu]

Are there extra configuration steps needed for eBPF profiling to work in the showcase?

Are you using the Cilium services in your service? If not, please remove it. I'm not sure the Envoy will be installed when using Cilium.

Has anyone managed to get flame graphs showing up with the latest Rover + OAP combo?

It has already be used in the skywalking demo: https://demo.skywalking.apache.org/dashboard/MESH/Service/bWVzaC1zdnI6OnNvbmdzLnNhbXBsZS1zZXJ2aWNlcw==.1/Mesh-Service

[KHADEPALLAVI975]

Hi @mrproliu @wu-sheng , Any Update on this issue?

[mrproliu]

I am getting an output like without cycles this for perf . Is this affect ebpf profiling ?

We are using the PERF_COUNT_SW_CPU_CLOCK for the CPU profiling, It should not be the cause of the CPU Profiling not working, but you can have a try to open it to check the profiling is working or not.

You could add some log into the BPF to check the program is running successfully or not.

I am facing an issue argument list too long: BPF program is too large

It's often happen when using the access log module, if it's no need to run, you can disable it temprory.

[KHADEPALLAVI975]

Here, i am facing issues like "the process has been recognized as dead, so deleted."
Could there be issues with the sample services?

[KHADEPALLAVI975]

total found stacks: 0 This could mean the rover cannot find any stack executing in the program. That's strange. You could check the following ways:

1. checking the program running status in the host that which rover belongs to: cat /proc/{rover pid}/fdinfo/{program fd}. The program fd should be one of the fd from success running on cpu profiling, perf event fds log.
2. Check the CPU lock is available in the host that which rover belongs to(for On CPU Profiling): perf stat -e cpu-clock -a sleep 1.
3. Check the program attachment success or not by the command: bpftool prog show
4. Add some logs into the BPF program to check the program's running status in this file(for on CPU profiling): https://github.com/apache/skywalking-rover/blob/main/bpf/profiling/oncpu.c

HI @mrproliu

I added bpf_trace_printk in oncpu.c, and I can see that the PID is not matching. How should I debug this further? Am I missing something?

Here is code

int do_perf_event(struct pt_regs *ctx) {
    int monitor_pid;
    asm("%0 = MONITOR_PID ll" : "=r"(monitor_pid));

    // only match the same pid
    __u64 id = bpf_get_current_pid_tgid();
    __u32 tgid = id >> 32;

    if (tgid != monitor_pid) {
        bpf_printk("PID %d does not match monitored PID %d\n", tgid, monitor_pid);
        return 0;
    }

    bpf_printk("PID %d matches monitored PID\n", tgid);

And here is the logs which i got

kworker/u24:13-2822105 [006] d.h2. 158785.412853: bpf_trace_printk: PID 2822105 does not match monitored PID 1044983

 DataCarrier.MET-1939979 [007] d.h1. 158785.412885: bpf_trace_printk: PID 1939606 does not match monitored PID 1947667

 DataCarrier.MET-1939979 [007] d.h1. 158785.412896: bpf_trace_printk: PID 1939606 does not match monitored PID 1044983

          <idle>-0       [011] dNh2. 158785.412905: bpf_trace_printk: PID 0 does not match monitored PID 1947675

   kworker/u24:5-2842377 [008] d.h2. 158785.412931: bpf_trace_printk: PID 2842377 does not match monitored PID 1947667

   kworker/u24:5-2842377 [008] d.h2. 158785.412934: bpf_trace_printk: PID 2842377 does not match monitored PID 1044983

            sudo-2867268 [009] d.h1. 158785.412958: bpf_trace_printk: PID 2867268 does not match monitored PID 1947667

            sudo-2867268 [009] d.h1. 158785.412970: bpf_trace_printk: PID 2867268 does not match monitored PID 1044983

            sshd-2867206 [010] d.h1. 158785.412997: bpf_trace_printk: PID 2867206 does not match monitored PID 1947667

            sshd-2867206 [010] d.h1. 158785.413003: bpf_trace_printk: PID 2867206 does not match monitored PID 1044983

 DataCarrier.MET-1939963 [011] d.h1. 158785.413032: bpf_trace_printk: PID 1939606 does not match monitored PID 1947667

[mrproliu]

Could you help check these two PIDs:

1. the rover logs detected process PID(in the detect process related logs)
2. The process ID in the host(which rover deployed, not execute ps inside the rover container).

If they are not the same one, please confirm that your Kubernetes node /proc/ is the same as the host /proc are same or not, If not, please add the column mount in your Kubernetes platform, such as in the kind env:

1. Added this column mount in each node configuration:

  - hostPath: /proc
    containerPath: /host/proc

2. Bind the /host/proc directory into the rover daemonset, and update the env ROVER_HOST_PROC_MAPPING as /host/proc.