missing some metrics information such as node-cpu-usage when try to monitor k8s metrics by oap-9.2.0

[arielwei123]

I try to monitor K8S metrics by oap-9.2.0 following the setup steps got from guide document. Almost all the metrics could be seen on UI, but  node-cpu-usage、node-network I/O、pod-cpu and pod-memory。
The OpenTelemetry Collector and RABC config shown below。
If i miss something for the monitor? Very appreciate if any suggestion!

Setup steps,
1.Setup [kube-state-metric]
2.cAdvisor is integrated into kubelet by default.
3.Set up [OpenTelemetry Collector ]
4.Config SkyWalking [OpenTelemetry receiver]

cluster-role.yaml for kube-state-metrics,

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.6.0
name: kube-state-metrics
rules:

• apiGroups:
  • ""
    resources:
  • configmaps
  • secrets
  • nodes
  • pods
  • services
  • serviceaccounts
  • resourcequotas
  • replicationcontrollers
  • limitranges
  • persistentvolumeclaims
  • persistentvolumes
  • namespaces
  • endpoints
  • nodes/metrics
  • nodes/proxy
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • apps
    resources:
  • statefulsets
  • daemonsets
  • deployments
  • replicasets
    verbs:
  • list
  • watch
• apiGroups:
  • batch
    resources:
  • cronjobs
  • jobs
    verbs:
  • list
  • watch
• apiGroups:
  • autoscaling
    resources:
  • horizontalpodautoscalers
    verbs:
  • list
  • watch
• apiGroups:
  • authentication.k8s.io
    resources:
  • tokenreviews
    verbs:
  • create
• apiGroups:
  • authorization.k8s.io
    resources:
  • subjectaccessreviews
    verbs:
  • create
• apiGroups:
  • policy
    resources:
  • poddisruptionbudgets
    verbs:
  • list
  • watch
• apiGroups:
  • certificates.k8s.io
    resources:
  • certificatesigningrequests
    verbs:
  • list
  • watch
• apiGroups:
  • storage.k8s.io
    resources:
  • storageclasses
  • volumeattachments
    verbs:
  • list
  • watch
• apiGroups:
  • admissionregistration.k8s.io
    resources:
  • mutatingwebhookconfigurations
  • validatingwebhookconfigurations
    verbs:
  • list
  • watch
• apiGroups:
  • networking.k8s.io
    resources:
  • networkpolicies
  • ingresses
    verbs:
  • list
  • watch
• apiGroups:
  • coordination.k8s.io
    resources:
  • leases
    verbs:
  • list
  • watch
• apiGroups:
  • rbac.authorization.k8s.io
    resources:
  • clusterrolebindings
  • clusterroles
  • rolebindings
  • roles
    verbs:
  • list
  • watch
• apiGroups:
  • extensions
    resources:
  • daemonsets
  • deployments
  • replicasets
    verbs:
  • list
  • watch

otel-config-collector.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-conf
namespace: skywalking
#namespace: kube-system
labels:
app: opentelemetry
component: otel-collector-conf
data:
otel-collector-config: |
receivers:
prometheus:
config:
global:
scrape_interval: 1m
scrape_timeout: 10s
#global:
#keep_dropped_targets: 100
scrape_configs:
- job_name: 'kubernetes-cadvisor'
#metrics_path: /metrics/cadvisor
scheme: https
#scheme: http
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: _meta_kubernetes_node_label(.+)
- source_labels: [ ]
target_label: cluster
replacement: skywalking-k8s
- target_label: address
replacement: kubernetes.default.svc:443
#replacement: kubernetes.default.svc.cluster.local:443
- source_labels: [ __meta_kubernetes_node_name ]
regex: (.+)
target_label: metrics_path
replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
- source_labels: [ instance ]
separator: ;
regex: (.+)
target_label: node
replacement: $$1
action: replace

        - job_name: kube-state-metrics
          metrics_path: /metrics
          kubernetes_sd_configs:
            - role: endpoints
          relabel_configs:
            - source_labels: [ __meta_kubernetes_service_label_app_kubernetes_io_name ]
              regex: kube-state-metrics
              replacement: $$1
              action: keep
            - action: labelmap
              regex: __meta_kubernetes_service_label_(.+)
            - source_labels: [ ]
              target_label: cluster
              replacement: skywalking-k8s

processors:
  batch:
exporters:
  otlp:
    endpoint: 'skywalking-oap.skywalking.svc.cluster.local:11800'
    tls:
      insecure: true
  logging:
    verbosity: detailed

service:
  pipelines:
    metrics/1:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp]

---

apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: skywalking
#namespace: kube-system
labels:
app: opentelemetry
component: otel-collector
spec:
ports:

• name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
  port: 4317
  protocol: TCP
  targetPort: 4317
• name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
  port: 4318
  protocol: TCP
  targetPort: 4318
• name: metrics # Default endpoint for querying metrics.
  port: 8888
  selector:
  component: otel-collector

---

apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
namespace: skywalking
#namespace: kube-system
labels:
app: opentelemetry
component: otel-collector
spec:
selector:
matchLabels:
app: opentelemetry
component: otel-collector
minReadySeconds: 5
progressDeadlineSeconds: 120
replicas: 1 #TODO - adjust this to your own requirements
template:
metadata:
labels:
app: opentelemetry
component: otel-collector
spec:
nodeName: k8s-node3
automountServiceAccountToken: true
serviceAccountName: kube-state-metrics
containers:
- command:
- "/otelcol"
- "--config=/conf/otel-collector-config.yaml"
image: otel/opentelemetry-collector:0.78.0
name: otel-collector
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
ports:
- containerPort: 55679 # Default endpoint for ZPages.
- containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
- containerPort: 14250 # Default endpoint for Jaeger gRPC receiver.
- containerPort: 14268 # Default endpoint for Jaeger HTTP receiver.
- containerPort: 9411 # Default endpoint for Zipkin receiver.
- containerPort: 8888 # Default endpoint for querying metrics.
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
volumeMounts:
- name: otel-collector-config-vol
mountPath: /conf
volumes:
- configMap:
name: otel-collector-conf
items:
- key: otel-collector-config
path: otel-collector-config.yaml
name: otel-collector-config-vol

OTEL_RECEIVER

       - name: SW_OTEL_RECEIVER_ENABLED_OTEL_RULES
          value: istio-controlplane,k8s-node,k8s-cluster,k8s-service,k8s-instance,oap,vm,mysql,postgresql

oap rabc

apiVersion: v1
kind: ServiceAccount
metadata:
labels:
#app: skywalking
app: skywalking-oap
name: skywalking-oap
namespace: skywalking

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: skywalking-oap
namespace: skywalking
labels:
#app: skywalking
app: skywalking-oap
rules:

• apiGroups:
  • ""
    resources:
  • configmaps
  • secrets
  • nodes
  • pods
  • services
  • serviceaccounts
  • resourcequotas
  • replicationcontrollers
  • limitranges
  • persistentvolumeclaims
  • persistentvolumes
  • namespaces
  • endpoints
  • instance
  • nodes/metrics
  • nodes/proxy
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • apps
    resources:
  • statefulsets
  • daemonsets
  • deployments
  • replicasets
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • batch
    resources:
  • cronjobs
  • jobs
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • autoscaling
    resources:
  • horizontalpodautoscalers
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • authentication.k8s.io
    resources:
  • tokenreviews
    verbs:
  • create
• apiGroups:
  • authorization.k8s.io
    resources:
  • subjectaccessreviews
    verbs:
  • create
• apiGroups:
  • policy
    resources:
  • poddisruptionbudgets
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • certificates.k8s.io
    resources:
  • certificatesigningrequests
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • storage.k8s.io
    resources:
  • storageclasses
  • volumeattachments
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • admissionregistration.k8s.io
    resources:
  • mutatingwebhookconfigurations
  • validatingwebhookconfigurations
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • networking.k8s.io
    resources:
  • networkpolicies
  • ingresses
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • coordination.k8s.io
    resources:
  • leases
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • rbac.authorization.k8s.io
    resources:
  • clusterrolebindings
  • clusterroles
  • rolebindings
  • roles
    verbs:
  • get
  • list
  • watch
• apiGroups:
  • extensions
    resources:
  • daemonsets
  • deployments
  • replicasets
    verbs:
  • get
  • list
  • watch

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: skywalking-oap
namespace: skywalking
labels:
app: skywalking-oap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: skywalking-oap
subjects:

• kind: ServiceAccount
  name: skywalking-oap
  namespace: skywalking

[wu-sheng]

It is nearly impossible to discuss this in this way.

We have the demo setup in the showcase and demo, showcase source codes are available for you, https://skywalking.apache.org/docs/skywalking-showcase/next/readme/

Please try to address the issue, rather than posting crazy YAMLs, and expect us to diagnose for you.