ServiceGraph Exponential Histograms Client metrics no time stamp

[owenhaynes]

Component(s)

otelcol.connector.servicegraph

What's wrong?

When sending exponential histograms to Grafana cloud the client metric traces_service_graph_request_client_seconds for the service graph fails to send because of no timestamp

The traces_service_graph_request_server_seconds has no issues and is is successfully sent.

Steps to reproduce

Use the following in your alloy config

otelcol.connector.servicegraph "default" {
	dimensions = [
		"service.namespace",
		"service.version",
		"deployment.environment",
		"k8s.cluster.name",
		"k8s.namespace.name",
		"team",
	]

	exponential_histogram_max_size = 20

	store {
		ttl = "2s"
	}

	output {
		metrics = [otelcol.processor.transform.trace_metrics.input]
	}
}

System information

K8s 1.33.*

Software version

Grafana Alloy 1.12.0

Configuration

Logs

ts=2025-12-04T18:10:02.156939816Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.lgtm error="not retryable error: Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to https://otlp-gateway-****.grafana.net/otlp/v1/metrics responded with HTTP Status Code 400, Message=received a sample whose timestamp is too far in the past, timestamp: 0 series: 'traces_service_graph_request_client_seconds' (err-mimir-too-far-in-past). To adjust the related per-tenant limit, configure -validation.past-grace-period, or contact your service administrator., Details=[]" dropped_items=407

Tip

_{React with 👍 if this issue is important to you.}