No OTLP custom metrics data in UI

[lxx95]

Hi all,

I have a problem during custom metrics visualization. I used OTLP to send metrics to OAP (v9.4.0) with the steps provided in the dedicated documentation page.

I created two different MAL file:
adservice.yaml

filter: "{ tags -> tags.job_name == 'adservice' }" # The OpenTelemetry job name
expSuffix: service(['node_identifier_host_name'] , Layer.GENERAL)
metricPrefix: adservice
metricsRules:
  - name: process_runtime_jvm_system_cpu_utilization
    exp: process_runtime_jvm_system_cpu_utilization
  - name: process_runtime_jvm_system_cpu_load_1m
    exp: process_runtime_jvm_system_cpu_load_1m

recommendationservice.yaml

filter: "{ tags -> tags.job_name == 'recommendationservice' }" # The OpenTelemetry job name
expSuffix: service(['node_identifier_host_name'] , Layer.GENERAL)
metricPrefix: recommendation
metricsRules:
  - name: app_recommendations_counter
    exp: app_recommendations_counter

We have also UI evidences about metrics, we can see the metrics creation result in the metrics list:

Unfortunately the metrics values are always 0 but we can confirm (via OTLP logs) that we pass also the Value field (different from 0):

2023-06-28T06:55:46.846Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 17, "data points": 54}
2023-06-28T06:55:46.847Z	info	ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.18.0
Resource attributes:
     -> container.id: Str(a3e780026dc1106d2e262daa74aa562dab5673d361f86e29dde11a5f7946cf46)
     -> host.arch: Str(amd64)
     -> host.name: Str(opentelemetry-demo-adservice-5cccc4fbfc-f87mp)
     -> k8s.namespace.name: Str(skywalking)
     -> k8s.node.name: Str(minikube)
     -> k8s.pod.name: Str(opentelemetry-demo-adservice-5cccc4fbfc-f87mp)
     -> os.description: Str(Linux 3.10.0-1160.90.1.el7.x86_64)
     -> os.type: Str(linux)
     -> process.command_line: Str(/opt/java/openjdk/bin/java -javaagent:/usr/src/app/opentelemetry-javaagent.jar)
     -> process.executable.path: Str(/opt/java/openjdk/bin/java)
     -> process.pid: Int(1)
     -> process.runtime.description: Str(Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.6+10)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(17.0.6+10)
     -> service.instance.id: Str(5bd54f4e-24ae-422f-b877-3649a045a2c5)
     -> service.name: Str(adservice)
     -> service.namespace: Str(opentelemetry-demo)
     -> telemetry.auto.version: Str(1.24.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.24.0)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope io.opentelemetry.runtime-metrics 1.24.0-alpha
...
Metric #3
Descriptor:
     -> Name: process_runtime_jvm_system_cpu_utilization
     -> Description: Recent cpu utilization for the whole system
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-06-28 06:54:46.62662756 +0000 UTC
Timestamp: 2023-06-28 06:55:46.644576946 +0000 UTC
Value: 0.004612
...
Metric #5
Descriptor:
     -> Name: process_runtime_jvm_system_cpu_load_1m
     -> Description: Average CPU load of the whole system for the last minute
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-06-28 06:54:46.62662756 +0000 UTC
Timestamp: 2023-06-28 06:55:46.644576946 +0000 UTC
Value: 5.404297

Also for recommendationservice:

ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope recommendationservice 
Metric #0
Descriptor:
     -> Name: app_recommendations_counter
     -> Description: Counts the total number of given recommendations
     -> Unit: recommendations
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> recommendation.type: Str(catalog)
StartTimestamp: 2023-06-28 06:57:57.16267253 +0000 UTC
Timestamp: 2023-06-28 06:58:57.169712625 +0000 UTC
Value: 80

Of course the ENV OAP variables was set:

    SW_OTEL_RECEIVER: default
    SW_RECEIVER_ZIPKIN: default
    SW_QUERY_ZIPKIN: default
    SW_OTEL_RECEIVER_ENABLED_HANDLERS: "otlp"
    SW_OTEL_RECEIVER_ENABLED_OTEL_RULES: "istio-controlplane,oap,vm,k8s/k8s-cluster,k8s/k8s-service,k8s/k8s-node,k8s/k8s-instance,recommendationservice,adservice"

I have no idea where the problem could be.

Thank you.

[lxx95]

Anyone with the same problem?

I tried also with MAL creation in $CLASSPATH/meter-analyzer-config like described here.

In that case these ENV OAP variables was set:

SW_RECEIVER_METER: default
SW_AGENT_ANALYZER: default
SW_METER_ANALYZER_ACTIVE_FILES: "recommendationservice,adservice"

Also in that case we have metrics structures but without data, the OTLP exporter was skywalking, like described here:

exporters:
  skywalking:
    endpoint: <my_endpoint>
    tls:
      insecure: true

Thank you.

[wu-sheng]

Only one thing, maybe relavent, #11020

[lxx95]

HI @wu-sheng yes maybe it could be related, but I used v9.4.0 of OAP and UI that should not be affected by the problem.

[wu-sheng]

I just know, 9.6.0-dev fixed this. I don't know when it exists.
But, you should be able to fix by yourself through reviewing the application.yml