feat: emit OpenTelemetry exemplars from HTTP duration histogram

[graceanya796-maker]

feat: emit OpenTelemetry exemplars from HTTP duration histogram

Closes #792

Summary

Wire OpenTelemetry exemplars on the http_request_duration_seconds Prometheus histogram so operators can jump directly from Grafana dashboards to a specific Tempo trace for any slow request, eliminating the manual context-switch that was previously required.

Motivation

The Prometheus histograms emitted by the HTTP middleware did not carry exemplars. Operators had to manually copy a request ID, search Tempo, and hope they found the right trace. With exemplars, every histogram bucket observation is tagged with trace_id and span_id, enabling one-click drill-down from metric panels to distributed traces.

Implementation

internal/tracing/tracing.go — canonical exemplar extraction

Added ExemplarLabels(ctx context.Context) prometheus.Labels — a reusable, well-documented helper that:

• Extracts the active span from the context via trace.SpanFromContext
• Validates the span context (IsValid, IsSampled, IsRecording)
• Returns nil for unsampled, non-recording, invalid, or absent spans
• Returns prometheus.Labels{"trace_id": ..., "span_id": ...} for active spans

This centralises the trace→exemplar conversion in the tracing package so any downstream consumer can use it without importing OTel internals.

internal/metrics/metrics.go — exemplar-aware HTTP histogram

Refactored MetricsMiddleware to call tracing.ExemplarLabels() instead of the now-removed private spanExemplar(). The middleware:

1. Observes the request duration on HTTPRequestDuration
2. Calls tracing.ExemplarLabels(ctx) to extract exemplar labels
3. If exemplars are available and the observer implements prometheus.ExemplarObserver, calls ObserveWithExemplar(duration, exemplars) — attaching trace_id and span_id
4. Falls back to plain Observe(duration) when exemplars are unavailable (unsampled spans, no active span)

This preserves full backward compatibility: Prometheus scrapes that don't enable exemplar storage see no change.

internal/middleware/middleware.go — exemplar-aware middleware

Added ExemplarAwareMiddleware — a contract-checkpoint middleware that:

• Verifies the OTel span context is propagated through the request lifecycle
• Sets X-Exemplar-Available: true response header when the span is sampled and recording
• Helps downstream services understand tracing state without re-extracting the span context
• Must be placed after the OTel tracing middleware (e.g. otelgin.Middleware) in the handler chain

Files Changed

File	Change
internal/tracing/tracing.go	Added ExemplarLabels() helper + imports (prometheus, trace)
internal/metrics/metrics.go	Replaced private spanExemplar() with tracing.ExemplarLabels() call; removed direct otel/trace import
internal/middleware/middleware.go	Added ExemplarAwareMiddleware + trace import
internal/tracing/exemplar_test.go	New — 12 focused tests for ExemplarLabels
internal/tracing/sampler.go	Fixed pre-existing compile error (return type mismatch)
internal/tracing/sampler_test.go	Fixed pre-existing compile error (shutdown return value)
internal/tracing/tail_sampling_test.go	Fixed pre-existing compile error (undefined struct fields)
internal/metrics/metrics_test.go	Updated exemplar tests to use tracing.ExemplarLabels; added 10 new tests
internal/middleware/exemplar_test.go	New — 7 focused tests for ExemplarAwareMiddleware

Test Coverage

internal/tracing/exemplar_test.go (12 tests)

Test	Scenario
TestExemplarLabels_SampledRecording	Happy path — sampled+recording span returns valid labels
TestExemplarLabels_Unsampled	NeverSample provider returns nil
TestExemplarLabels_NoSpan	Background context (no span) returns nil
TestExemplarLabels_EndedSpan	Ended span (not recording) returns nil
TestExemplarLabels_InvalidSpanContext	Zero TraceID returns nil
TestExemplarLabels_SampledButNotRecording	Sampled flag without recording returns nil
TestExemplarLabels_VerifyHexFormat	Labels are valid lowercase hex
TestExemplarLabels_CorruptContext	Non-OTel value in context returns nil
TestExemplarLabels_ConcurrentSafety	100 goroutines call concurrently — no race
TestExemplarLabels_ConsistencyAcrossCalls	Multiple calls return identical labels
TestExemplarLabels_TraceIDMatchesSpanContext	Labels match span.SpanContext() values
TestExemplarLabels_NeverSampleProvider	NeverSample with explicit span context returns nil

internal/metrics/metrics_test.go (10 exemplar-specific tests)

Test	Scenario
TestExemplarLabels_SampledRecording	Delegates to tracing.ExemplarLabels — happy path
TestExemplarLabels_Unsampled	Returns nil for unsampled span
TestExemplarLabels_NoSpan	Returns nil for background context
TestExemplarLabels_EndedSpan	Returns nil for ended span
TestExemplarLabels_EmptyTraceID	Returns nil for invalid (zero) trace ID
TestExemplarLabels_CorruptContext	Returns nil for non-OTel context
TestMetricsMiddleware_ExemplarAttachedOnSampledRequest	Sampled request records metrics
TestMetricsMiddleware_NoExemplarOnUnsampledRequest	Unsampled request records metrics (no exemplars)
TestMetricsMiddleware_ExemplarTraceIDAndSpanIDInLabels	Exemplar labels match span's trace_id/span_id
TestMetricsMiddleware_ExemplarFallbackToPlainObserve	Falls back to Observe when ExemplarObserver unavailable
TestMetricsMiddleware_BackwardCompatibility	Standard histogram buckets in /metrics output without exemplars
TestExemplarLabels_ConcurrentSafety	100 goroutines — no race condition
TestExemplarLabels_MultipleRequestsEachGetUniqueTraceID	Each trace produces a unique trace_id
TestExemplarLabels_VerifyLabelValues	All label values are valid hex
TestMetricsMiddleware_ExemplarDoesNotBreakDBTimer	DB metrics path unaffected by exemplar changes

internal/middleware/exemplar_test.go (7 tests)

Test	Scenario
TestExemplarAwareMiddleware_SampledSpan	Sets X-Exemplar-Available: true header
TestExemplarAwareMiddleware_NoSpan	No header when context has no span
TestExemplarAwareMiddleware_UnsampledSpan	No header when span is unsampled
TestExemplarAwareMiddleware_EndedSpan	No header for ended (non-recording) span
TestExemplarAwareMiddleware_PreservesNextHandlerStatus	Response status from next handler is preserved
TestExemplarAwareMiddleware_BackwardCompatibility	Requests without OTel context work unchanged
TestExemplarAwareMiddleware_InvalidSpanContext	No header for invalid span context

Security & Data-Integrity

• No PII in exemplars: Only trace_id and span_id (opaque 128-bit and 64-bit identifiers) are attached — no user data, tenant IDs, or request bodies.
• No cardinality bloat: Exemplars are only attached when the span is sampled and recording. Unsampled requests never produce exemplar labels.
• Backward compatible: The prometheus.ExemplarObserver type assertion ensures the code falls back to plain Observe() if the histogram doesn't support exemplars. No existing scrape configurations break.

Failure Modes

Scenario	Behavior
No OTel span in context	ExemplarLabels returns nil → plain Observe()
Span not sampled	ExemplarLabels returns nil → plain Observe()
Span ended (not recording)	ExemplarLabels returns nil → plain Observe()
Invalid span context (zero TraceID)	ExemplarLabels returns nil → plain Observe()
Histogram doesn't implement ExemplarObserver	Type assertion fails → plain Observe()
Concurrent requests	ExemplarLabels is goroutine-safe (read-only on immutable span context)

Backward Compatibility

• Prometheus scrape: Standard histogram bucket output is unchanged. Exemplars are only visible when the Prometheus server has --enable-feature=exemplar-storage enabled.
• No API changes: All public function signatures are unchanged. The only additions are tracing.ExemplarLabels() and middleware.ExemplarAwareMiddleware().
• No dependency changes: Uses existing prometheus/client_golang and go.opentelemetry.io/otel versions already in go.mod.

Scrape Configuration

To expose exemplars in Prometheus/Grafana, ensure your scrape config includes:

scrape_configs:
  - job_name: 'stellabill-backend'
    metrics_path: '/metrics'
    # Exemplars are exposed automatically when --enable-feature=exemplar-storage
    # is set on the Prometheus server.

Grafana panels can then use the exemplar link to jump directly to Tempo traces.

Regression Coverage

• Empty context → nil exemplars (no crash)
• Invalid/zero trace ID → nil exemplars (no crash)
• Duplicate request IDs → unique trace IDs per request
• Long-running requests → duration histogram still accurate with exemplars
• DB metrics path → unaffected by exemplar changes (verified by TestMetricsMiddleware_ExemplarDoesNotBreakDBTimer)
• Multiple concurrent requests → no race conditions (verified by TestExemplarLabels_ConcurrentSafety)

[drips-wave]

@graceanya796-maker Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits