Closed
Description
OpenTelemetry gRPC metrics sent to the APM Server OTLP endpoint are not displayed correctly in Kibana when originating from the Java OpenTelemetry Agent.
APM Server version (apm-server version):
8.10.4
Kibana version
8.13.2
OTEL Java Agent version:
2.10.0
Expected Behavior:
Should show metrics correctly in Kibana
Steps to reproduce:
Instrument an java application using the OTEL java agent - opentelemetry-java-instrumentation
Metrics Context from OTEL collector
collector-1 | Metric #9
collector-1 | Descriptor:
collector-1 | -> Name: jvm.gc.duration
collector-1 | -> Description: Duration of JVM garbage collection actions.
collector-1 | -> Unit: s
collector-1 | -> DataType: Histogram
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | HistogramDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> jvm.gc.action: Str(end of minor GC)
collector-1 | -> jvm.gc.name: Str(PS Scavenge)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Count: 2615
collector-1 | Sum: 5.732000
collector-1 | Min: 0.000000
collector-1 | Max: 0.050000
collector-1 | ExplicitBounds #0: 0.010000
collector-1 | ExplicitBounds #1: 0.100000
collector-1 | ExplicitBounds #2: 1.000000
collector-1 | ExplicitBounds #3: 10.000000
collector-1 | Buckets #0, Count: 2602
collector-1 | Buckets #1, Count: 13
collector-1 | Buckets #2, Count: 0
collector-1 | Buckets #3, Count: 0
collector-1 | Buckets #4, Count: 0
collector-1 | HistogramDataPoints #1
collector-1 | Data point attributes:
collector-1 | -> jvm.gc.action: Str(end of major GC)
collector-1 | -> jvm.gc.name: Str(PS MarkSweep)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Count: 4
collector-1 | Sum: 0.930000
collector-1 | Min: 0.074000
collector-1 | Max: 0.362000
collector-1 | ExplicitBounds #0: 0.010000
collector-1 | ExplicitBounds #1: 0.100000
collector-1 | ExplicitBounds #2: 1.000000
collector-1 | ExplicitBounds #3: 10.000000
collector-1 | Buckets #0, Count: 0
collector-1 | Buckets #1, Count: 1
collector-1 | Buckets #2, Count: 3
collector-1 | Buckets #3, Count: 0
collector-1 | Buckets #4, Count: 0
collector-1 | Metric #10
collector-1 | Descriptor:
collector-1 | -> Name: jvm.cpu.time
collector-1 | -> Description: CPU time used by the process as reported by the JVM.
collector-1 | -> Unit: s
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: true
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 155.370000
collector-1 | Metric #11
collector-1 | Descriptor:
collector-1 | -> Name: jvm.thread.count
collector-1 | -> Description: Number of executing platform threads.
collector-1 | -> Unit: {thread}
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: false
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> jvm.thread.daemon: Bool(true)
collector-1 | -> jvm.thread.state: Str(runnable)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 11
collector-1 | NumberDataPoints #1
collector-1 | Data point attributes:
collector-1 | -> jvm.thread.daemon: Bool(true)
collector-1 | -> jvm.thread.state: Str(waiting)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 8
collector-1 | NumberDataPoints #2
collector-1 | Data point attributes:
collector-1 | -> jvm.thread.daemon: Bool(false)
collector-1 | -> jvm.thread.state: Str(timed_waiting)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 2
collector-1 | NumberDataPoints #3
collector-1 | Data point attributes:
collector-1 | -> jvm.thread.daemon: Bool(false)
collector-1 | -> jvm.thread.state: Str(waiting)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 1
collector-1 | NumberDataPoints #4
collector-1 | Data point attributes:
collector-1 | -> jvm.thread.daemon: Bool(false)
collector-1 | -> jvm.thread.state: Str(runnable)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 1
collector-1 | NumberDataPoints #5
collector-1 | Data point attributes:
collector-1 | -> jvm.thread.daemon: Bool(true)
collector-1 | -> jvm.thread.state: Str(timed_waiting)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 13
collector-1 | Metric #12
collector-1 | Descriptor:
collector-1 | -> Name: jvm.memory.used_after_last_gc
collector-1 | -> Description: Measure of memory used, as measured after the most recent garbage collection event on this pool.
collector-1 | -> Unit: By
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: false
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(PS Eden Space)
collector-1 | -> jvm.memory.type: Str(heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 0
collector-1 | NumberDataPoints #1
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(PS Survivor Space)
collector-1 | -> jvm.memory.type: Str(heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 98304
collector-1 | NumberDataPoints #2
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(PS Old Gen)
collector-1 | -> jvm.memory.type: Str(heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 97485888
collector-1 | Metric #13
collector-1 | Descriptor:
collector-1 | -> Name: jvm.memory.limit
collector-1 | -> Description: Measure of max obtainable memory.
collector-1 | -> Unit: By
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: false
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(PS Eden Space)
collector-1 | -> jvm.memory.type: Str(heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 958398464
collector-1 | NumberDataPoints #1
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(CodeHeap 'profiled nmethods')
collector-1 | -> jvm.memory.type: Str(non_heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 122912768
collector-1 | NumberDataPoints #2
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(CodeHeap 'non-nmethods')
collector-1 | -> jvm.memory.type: Str(non_heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 5828608
collector-1 | NumberDataPoints #3
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(CodeHeap 'non-profiled nmethods')
collector-1 | -> jvm.memory.type: Str(non_heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 122916864
collector-1 | NumberDataPoints #4
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(Compressed Class Space)
collector-1 | -> jvm.memory.type: Str(non_heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 1073741824
collector-1 | NumberDataPoints #5
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(PS Survivor Space)
collector-1 | -> jvm.memory.type: Str(heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 524288
collector-1 | NumberDataPoints #6
collector-1 | Data point attributes:
collector-1 | -> jvm.memory.pool.name: Str(PS Old Gen)
collector-1 | -> jvm.memory.type: Str(heap)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 1919942656
collector-1 | Metric #14
collector-1 | Descriptor:
collector-1 | -> Name: jvm.class.unloaded
collector-1 | -> Description: Number of classes unloaded since JVM start.
collector-1 | -> Unit: {class}
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: true
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 1
collector-1 | Metric #15
collector-1 | Descriptor:
collector-1 | -> Name: jvm.class.count
collector-1 | -> Description: Number of classes currently loaded.
collector-1 | -> Unit: {class}
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: false
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 32949
collector-1 | Metric #16
collector-1 | Descriptor:
collector-1 | -> Name: jvm.buffer.memory.limit
collector-1 | -> Description: Measure of total memory capacity of buffers.
collector-1 | -> Unit: By
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: false
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> jvm.buffer.pool.name: Str(mapped - 'non-volatile memory')
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 0
collector-1 | NumberDataPoints #1
collector-1 | Data point attributes:
collector-1 | -> jvm.buffer.pool.name: Str(mapped)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 0
collector-1 | NumberDataPoints #2
collector-1 | Data point attributes:
collector-1 | -> jvm.buffer.pool.name: Str(direct)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 8436561
collector-1 | Metric #17
collector-1 | Descriptor:
collector-1 | -> Name: jvm.class.loaded
collector-1 | -> Description: Number of classes loaded since JVM start.
collector-1 | -> Unit: {class}
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: true
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 32950
collector-1 | ScopeMetrics #1
collector-1 | ScopeMetrics SchemaURL:
collector-1 | InstrumentationScope io.opentelemetry.sdk.logs
collector-1 | Metric #0
collector-1 | Descriptor:
collector-1 | -> Name: queueSize
collector-1 | -> Description: The number of items queued
collector-1 | -> Unit: 1
collector-1 | -> DataType: Gauge
collector-1 | NumberDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> processorType: Str(BatchLogRecordProcessor)
collector-1 | StartTimestamp: 2024-12-13 09:42:20.625079679 +0000 UTC
collector-1 | Timestamp: 2024-12-13 09:55:20.664590596 +0000 UTC
collector-1 | Value: 0
