[LFXV2-609] Add OpenTelemetry tracing support

[bramwelt]

This pull request introduces comprehensive OpenTelemetry (OTel) support to the lfx-v2-query-service, enabling distributed tracing, metrics, and log correlation for improved observability. It adds OTel configuration options to the Helm chart and application, instruments the HTTP server, and enhances logging with trace context. The most significant changes are grouped below.

OpenTelemetry integration and configuration:

• Added a new utility package (pkg/utils/otel.go) that provides OTel configuration via environment variables, sets up OTel SDK (tracing, metrics, logs), and configures exporters and propagators. This enables flexible OTel setup based on deployment needs.
• Updated the Helm chart (values.yaml and deployment.yaml) to allow configuring OTel environment variables such as service name, protocol, endpoint, exporters, and propagators, making OTel setup declarative and deployment-friendly. [1] [2]
• Bumped chart version to 0.4.10 to reflect these configuration changes.

Application instrumentation and startup:

• Modified cmd/main.go to initialize the OTel SDK on startup using the new utility, ensuring that all telemetry is exported as configured and that shutdown is handled gracefully. [1] [2]
• Wrapped the HTTP server handler in OTel instrumentation (otelhttp.NewHandler) to automatically generate trace spans for incoming HTTP requests. [1] [2]

Logging improvements for trace correlation:

• Enhanced the logging system to inject trace and span IDs into log records using the slog-otel handler and a new helper in pkg/logging/context.go, allowing logs to be correlated with distributed traces. [1] [2] [3]

Dependency updates:

• Added and updated dependencies in go.mod to include OTel SDKs, exporters, and the slog-otel handler, ensuring all new functionality is supported.

These changes collectively provide robust observability for the service, making it easier to monitor, debug, and analyze system behavior in production environments.

[coderabbitai]

Walkthrough

This pull request adds comprehensive OpenTelemetry observability instrumentation to the project. Changes include: OTEL SDK initialization with configurable exporters for traces, metrics, and logs; HTTP instrumentation wrapping handlers and clients; environment-driven configuration through Kubernetes values and environment variables; logging integration with trace context propagation; build-time variable injection via ldflags; and CI/CD pipeline updates to capture build metadata for binary distribution.

Changes

Cohort / File(s)	Summary
OpenTelemetry Configuration Framework pkg/utils/otel.go, pkg/utils/otel_test.go	New self-contained OTEL bootstrapper supporting OTLP over HTTP/gRPC, resource creation, propagator composition, and per-signal (traces/metrics/logs) provider setup with configurable exporters and sampling. Comprehensive test coverage for default/custom configurations and SDK initialization.
Application Startup & Initialization cmd/main.go, cmd/http.go	Adds OTEL SDK initialization at startup with environment-based config, graceful shutdown handler, and build-time variables (Version, BuildTime, GitCommit). HTTP handler wrapped with otelhttp for request tracing.
Structured Logging Integration pkg/log/log.go	Replaces multi-map config with environment-driven approach. Wraps JSON slog handler with OtelHandler to propagate trace_id/span_id, adds context-aware handler wrapper, and logs final configuration.
HTTP Client Instrumentation pkg/httpclient/client.go, internal/infrastructure/opensearch/searcher.go, internal/infrastructure/opensearch/client.go	Wraps HTTP transport with otelhttp.NewTransport to enable tracing on outgoing requests. Removes redundant httpClient field from OpenSearch client struct.
Kubernetes Deployment Configuration charts/lfx-v2-query-service/templates/deployment.yaml, charts/lfx-v2-query-service/values.yaml	Adds OTEL environment variables (service name/version, endpoint, protocol, exporters, sampling, propagators) with conditional rendering. Introduces extraEnv field for custom environment injection in deployment template.
Dependency Management go.mod	Adds OpenTelemetry v1.40.0 ecosystem (SDK, exporters for traces/metrics/logs, HTTP instrumentation, Jaeger propagator, slog-otel integration). Updates golang.org/x and gRPC-related indirect dependencies to align with OTEL lineage.
Build & Release Pipeline .github/workflows/ko-build-main.yaml, .github/workflows/ko-build-tag.yaml, .ko.yaml	Adds VERSION, GIT_COMMIT, and BUILD_TIME environment variables to build steps. New ko.yaml configuration injects build metadata via ldflags (main.Version, main.BuildTime, main.GitCommit). Enables SBOM generation.
Test Formatting internal/infrastructure/opensearch/searcher_test.go	Minor field alignment and whitespace cleanup; no functional changes.

Sequence Diagram

sequenceDiagram
    participant App as Application
    participant OTEL as OTEL SDK
    participant Exporter as OTLP Exporter
    participant Logger as Structured Logger
    participant Tracer as Tracer Provider

    App->>OTEL: OTelConfigFromEnv()
    OTEL-->>App: OTelConfig
    App->>OTEL: SetupOTelSDKWithConfig(ctx, cfg)
    OTEL->>OTEL: newResource(cfg)
    OTEL->>OTEL: newPropagator(cfg)
    OTEL->>OTEL: newTraceProvider(ctx, cfg, resource)
    OTEL->>OTEL: newMetricsProvider(ctx, cfg, resource)
    OTEL->>OTEL: newLoggerProvider(ctx, cfg, resource)
    OTEL->>Exporter: Register OTLP exporters (traces/metrics/logs)
    Exporter-->>OTEL: Exporter initialized
    OTEL-->>App: shutdown func
    
    App->>Logger: InitStructureLogConfig()
    Logger->>Logger: Wrap with OtelHandler
    Logger->>Logger: Set as default slog
    
    App->>App: Start HTTP Server
    
    rect rgba(0, 150, 136, 0.5)
    Note over App,Tracer: Request processing
    App->>Tracer: otelhttp wraps handler
    Tracer->>Logger: Extract trace context
    Logger-->>Tracer: Emit logs with trace_id/span_id
    Tracer->>Exporter: Export trace spans
    end
    
    App->>OTEL: defer shutdown(gracefulTimeout)
    OTEL->>Exporter: Flush and close exporters
    Exporter-->>OTEL: Shutdown complete

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding OpenTelemetry tracing support to the service, which matches the primary objective across all modified files.
Description check	✅ Passed	The description comprehensively explains the OTel integration, covering configuration, instrumentation, logging improvements, and dependency updates with specific file references.
Linked Issues check	✅ Passed	The PR successfully addresses all LFXV2-609 objectives: added OTel dependencies in go.mod, configured SDK via environment variables in pkg/utils/otel.go with Helm chart integration, instrumented HTTP handlers with otelhttp.NewHandler, and enhanced logging with trace ID/span ID injection via slog-otel.
Out of Scope Changes check	✅ Passed	All changes are scoped to OTel integration objectives. Minor formatting fixes in searcher_test.go and httpClient struct cleanup are incidental adjustments aligned with the instrumentation work.
Docstring Coverage	✅ Passed	Docstring coverage is 88.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch bramwelt/otel-implementation

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This pull request introduces comprehensive OpenTelemetry (OTel) support to the lfx-v2-query-service, enabling distributed tracing, metrics, and log correlation. The implementation includes a new OTel utility package, HTTP instrumentation, enhanced logging with trace context, and declarative Helm chart configuration.

Changes:

• Added OTel SDK setup with configurable exporters for traces, metrics, and logs
• Instrumented HTTP server with automatic span generation for incoming requests
• Enhanced logging to inject trace and span IDs for correlation with distributed traces

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/utils/otel.go	New utility package providing OTel configuration and SDK initialization with support for OTLP exporters
pkg/utils/otel_test.go	Comprehensive test coverage for OTel configuration parsing, SDK setup, and provider functions
pkg/logging/context.go	New helpers to extract trace context from context and add to log attributes
pkg/log/log.go	Wrapped logging handler with slog-otel to automatically inject trace context into logs
cmd/main.go	Initialize OTel SDK on startup with proper shutdown handling
cmd/http.go	Wrap HTTP handler with OTel instrumentation for automatic trace span generation
charts/lfx-v2-query-service/values.yaml	Added OTel configuration options with sensible defaults
charts/lfx-v2-query-service/templates/deployment.yaml	Inject OTel environment variables into deployment based on Helm values
charts/lfx-v2-query-service/Chart.yaml	Bump chart version to 0.4.10
go.mod, go.sum	Added OTel SDK dependencies and updated related packages

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@pkg/log/log.go`:
- Around line 67-105: InitStructureLogConfig currently builds a JSON handler
(h), wraps it with slogotel.OtelHandler (otelHandler) and a contextHandler
(logger) but returns the raw h, so callers miss trace/span and context attribute
enrichment; update the function to return the wrapped handler (logger) instead
of h (ensure contextHandler implements slog.Handler) so callers receive the
fully enriched handler while keeping the existing
slog.SetDefault(slog.New(logger)) call.