Issues with SkyWalking Setup on Kubernetes: Elasticsearch Cluster Configuration, Flamegraph Visualization, and Component Build Steps

[hsrakesh]

Hello,

I’m currently setting up SkyWalking Rover and APM (Application Performance Management) for my project and would appreciate some help with the following:

System Requirements:
Could anyone provide detailed hardware and software requirements for SkyWalking, specifically for the following components:

UI (User Interface)

OAP (Observability Analysis Platform)

Elasticsearch (ES)

Elasticsearch Configuration Issues:
I am running Elasticsearch on a Kubernetes cluster and have tested versions 7.14.0, 7.17.3, and 8.5.1, but am encountering issues with cluster stability and performance. Has anyone experienced similar problems with Elasticsearch on Kubernetes, or could you provide any best practices for configuring ES in a distributed setup?

Flamegraph Issue:
When using the skywalking-helm chart for deployment, I am unable to generate flamegraphs in the UI. The flamegraph visualization does not appear, and I am unsure whether it's a configuration issue with SkyWalking or with the Helm deployment.

Build Steps:
I would also appreciate any guidance or detailed steps on how to build and configure the following components correctly:

UI

OAP

Elasticsearch

I’m working on Ubuntu 22.04 (Kernel 6.8), and I would appreciate any insights, especially regarding proper version compatibility between Elasticsearch and SkyWalking for optimal performance.

Looking forward to your assistance!

[wu-sheng]

Flamegraph Issue:
When using the skywalking-helm chart for deployment, I am unable to generate flamegraphs in the UI. The flamegraph visualization does not appear, and I am unsure whether it's a configuration issue with SkyWalking or with the Helm deployment.

Which widget you are asking?

[mrproliu]

These programs seem like belong to the access log module. You could disable it and take a look at the BPF program list again to check are they still exist. And your attached image seems like there is no rover related program?

Additionally, I noticed that the rover logs print the local address along with a PID, but I'm unclear which process or socket this PID and local address belong to.

It still belongs to the access log module.

[hsrakesh]

We verified with the profiling tasks and the attached output. We can see that the profiling task has started, but the stacks are still showing as 0.

[hsrakesh]

After disabling the access log module and rebuilding the cluster, we encountered a warning message: 'Could not execute task, ignored.'

[hsrakesh]

Hi @mrproliu, @wu-sheng

Any update on this? I also noticed that the process has been recognized as dead.
Could this be an issue with a kernel or a process running on the cluster?

Additionally, I see the following error related to pod metrics:

The watchlist request for pods ended with an error, falling back to the standard LIST semantics.
Error: pods.metrics.k8s.io is forbidden: User "system:serviceaccount:kube-system:horizontal-pod-autoscaler" cannot watch resource "pods" in API group "metrics.k8s.io" in the namespace "skywalking-showcase"

**E0423 06:45:09.499384 1 horizontal.go:275] "Unhandled Error" err="failed to compute desired number of replicas based on listed metrics for Deployment/skywalking-showcase/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)" logger="UnhandledError"**

Could you please check and let me know how we should proceed?

[hsrakesh]

Hi @wu-sheng,

Any update on this error?

[wu-sheng]

Elasticsearch Configuration Issues:
I am running Elasticsearch on a Kubernetes cluster and have tested versions 7.14.0, 7.17.3, and 8.5.1, but am encountering issues with cluster stability and performance. Has anyone experienced similar problems with Elasticsearch on Kubernetes, or could you provide any best practices for configuring ES in a distributed setup?

We don't have that easily. I think mostly due to lack of CPU and memory.
You should open replication(x2) to increase performance.

[hsrakesh]

We are working with 10 CPU cores and 50GB of memory.