Request for Updated Blogs on SkyWalking Rover for Kubernetes Application Profiling

[KHADEPALLAVI975]

Hi community,
I'm currently exploring CPU profiling for Kubernetes applications using Rover, and I'm particularly interested in integrating the latest version of Apache SkyWalking to enhance observability in production environments.
Based on concepts from Brendan Gregg's work, we're looking to implement:
On-CPU Profiling: Understanding where threads spend time actively running.
Off-CPU Profiling: Tracking time spent waiting (I/O, locks, timers, etc.).
We're also delving into eBPF-based profiling of Envoy, especially in service mesh setups like Istio, where Envoy acts as a sidecar proxy. Since performance issues in Envoy can impact all mesh traffic, this type of profiling is crucial for uncovering hidden bottlenecks.
What I need help with:
I'm looking for updated blogs, guides, or examples that demonstrate how to use the latest version of SkyWalking in conjunction with Rover for profiling in Kubernetes environments.
If anyone has:
Recent blog posts or tutorials
GitHub repos
Real-world case studies or experience reports
…please share them here! I'd greatly appreciate any direction or resources you can offer 🙏
Thanks in advance!

[wu-sheng]

There may help.

• https://skywalking.apache.org/blog/2023-06-25-intruducing-continuous-profiling-skywalking-with-ebpf/
• https://skywalking.apache.org/blog/diagnose-service-mesh-network-performance-with-ebpf/
• https://skywalking.apache.org/blog/2022-07-05-pinpoint-service-mesh-critical-performance-impact-by-using-ebpf/

[wu-sheng]

There is no newer blog. Blogs are not written for every release/update.

[wu-sheng]

The UI interaction is similar, and showcase has all deployment automatically, as same as our demo.

I am not sure what you are looking, there is not much manually setup.

[wu-sheng]

https://skywalking.apache.org/docs/skywalking-showcase/next/readme/

These work on GCP k8s simply. All things are set.

[KHADEPALLAVI975]

I have tried every Helm setup as well as the SkyWalking showcase setup, but I am still unable to get the CPU profiling graphs using the latest version of SkyWalking Rover.

There are no clear guides or blogs that explain the dependencies or prerequisites. Additionally, there are no proper blogs related to the latest release of eBPF profiling, which makes it difficult to debug and understand the setup.

The Rover logs are attached here.

[wu-sheng]

We don't do any extra step in the GCP env. We are still not sure why this happens at your end.

[wu-sheng]

The https://skywalking.apache.org#demo has the showcase deployment, and profiling is automatically triggered and running.

You could see details from this page(k8s service monitoring powered by Rover latest version).
https://demo.skywalking.apache.org/dashboard/MESH/Service/bWVzaC1zdnI6OnNvbmdzLnNhbXBsZS1zZXJ2aWNlcw==.1/Mesh-Service

songs.sample-service is being profiled a lot of times.

[mrproliu]

Sorry, I just saw the comment. Could you just remove all the scaling jobs, then check the rover pod logs is there still have the process dead logs?

[KHADEPALLAVI975]

HI @mrproliu

I deployed it after removing scaling jobs, But still i am facing the same process dead issue in the rover pod logs.

[mrproliu]

It's strange, is the same process always detected and dead?

[KHADEPALLAVI975]

Hi @mrproliu

From the logs processes are getting detected and dead

Why this behaviour is there ?

time="2025-04-29T10:51:46Z" level=info msg="detected new process by sync all: pid: **557650**, entity: {\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::nginx.skywalking-showcase\",\"InstanceName\":\"nginx-deployment-b758dcbf8-9m28b\",\"ProcessName\":\"sleep\",\"Labels\":[\"k8s-service\"]}" module=process.finder
time="2025-04-29T10:51:46Z" level=info msg="detected new process by sync all: pid: 557668, entity: {\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::demo-etcd.skywalking-showcase\",\"InstanceName\":\"demo-etcd-0\",\"ProcessName\":\"healthcheck.sh\",\"Labels\":[\"k8s-service\"]}" module=process.finder
time="2025-04-29T10:51:46Z" level=info msg="detected new process by sync all: pid: 557679, entity: {\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::demo-etcd.skywalking-showcase\",\"InstanceName\":\"demo-etcd-0\",\"ProcessName\":\"etcdctl\",\"Labels\":[\"k8s-service\"]}" module=process.finder
time="2025-04-29T10:51:46Z" level=info msg="detected new process by sync all: pid: 557692, entity: {\"Layer\":\"MESH\",\"ServiceName\":\"mesh-svr::r3-load.sample-services\",\"InstanceName\":\"r3-load-29098730-jjnlf\",\"ProcessName\":\"sleep\",\"Labels\":[\"mesh-application\"]}" module=process.finder
time="2025-04-29T10:51:46Z" level=info msg="the process has been recognized as dead, so deleted. pid: 557440, entity: {\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::nginx.skywalking-showcase\",\"InstanceName\":\"nginx-deployment-b758dcbf8-9m28b\",\"ProcessName\":\"sleep\",\"Labels\":[\"k8s-service\"]}, id: " module=process.finder
time="2025-04-29T10:51:46Z" level=info msg="the process has been recognized as dead, so deleted. pid: 557449, entity: {\"Layer\":\"MESH\",\"ServiceName\":\"mesh-svr::r3-load.sample-services\",\"InstanceName\":\"r3-load-29098730-jjnlf\",\"ProcessName\":\"sleep\",\"Labels\":[\"mesh-application\"]}, id: " module=process.finder
time="2025-04-29T10:51:51Z" level=info msg="detected new process by sync all: pid: 557830, entity: {\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::nginx.skywalking-showcase\",\"InstanceName\":\"nginx-deployment-b758dcbf8-9m28b\",\"ProcessName\":\"sleep\",\"Labels\":[\"k8s-service\"]}" module=process.finder
time="2025-04-29T10:51:51Z" level=info msg="detected new process by sync all: pid: 557839, entity: {\"Layer\":\"MESH\",\"ServiceName\":\"mesh-svr::r3-load.sample-services\",\"InstanceName\":\"r3-load-29098730-jjnlf\",\"ProcessName\":\"sleep\",\"Labels\":[\"mesh-application\"]}" module=process.finder
time="2025-04-29T10:51:51Z" level=info msg="the process has been recognized as dead, so deleted. pid: **557650**, entity: {\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::nginx.skywalking-showcase\",\"InstanceName\":\"nginx-deployment-b758dcbf8-9m28b\",\"ProcessName\":\"sleep\",\"Labels\":[\"k8s-service\"]}, id: " module=process.finder

[mrproliu]

It looks like the nginx pod has an executed sleep command. It can be ignored. Is there another longlive process getting dead?