Add OpenTelemetry tracing provider to the container (disabled by default) (#37227)

* Add OpenTelemetry tracing provider to the container (disabled by default)

Introduces the no-op-by-default infrastructure for native OpenTelemetry
(OTel) tracing in the container. No tracing happens until explicitly
enabled via the opentelemetry-sdk feature flag.

- container-opentelemetry: new Felix bundle embedding the OTel SDK + OTLP
  HTTP exporter (jdk sender, no okhttp/kotlin); exports the api/context/sdk
  packages. Pre-installed into the container.
- OpenTelemetryProvider (container-disc): Provider<OpenTelemetry> registered
  in all container types. Hands out OpenTelemetry.noop() when disabled;
  builds the real SDK (OTLP/HTTP, batching, parent-based ratio sampling,
  W3C propagation) only when enabled. Builds the OTel Resource from the
  resource-attribute map.
- telemetry.def: enabled / endpoint / samplingRatio + a resourceAttribute
  map. ContainerCluster fills the map with deployment identity available in
  the model (application, tenant, cluster.type, cluster.id).
- opentelemetry-sdk JSON feature flag (enabled/endpoint/samplingRatio),
  threaded through ModelContext -> ContainerCluster.getConfig. Takes effect
  at redeployment.
- dependency-enforcer + abi-spec updated accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Make OpenTelemetry classes available to in-process container tests; avoid null endpoint

- container-dev: add the OpenTelemetry SDK + OTLP exporter artifacts so the in-process
  container test harness (StandaloneContainerApplication) can load OpenTelemetryProvider,
  which is now registered in every container. In production these come from the
  pre-installed container-opentelemetry bundle via OSGi; container-dev is the flat
  test classpath, mirroring vespa-3party-bundles / container-onnxruntime.
- Never hand back a null endpoint when disabled: the telemetry.def endpoint field is
  mandatory, so the generated config builder rejects null. OpenTelemetryConfiguration.disabled()
  and OpenTelemetrySettings now use an empty string (still no localhost default, so nothing
  is mistakenly sent).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Expose only the OpenTelemetry API on the public container classpath

The SDK must not leak onto the 3rd-party container classpath (container-dependencies-enforcer).
Keep the public surface to the OTel API only:

- Split OpenTelemetrySdkBuilder out of OpenTelemetryProvider. The provider now references only the OTel
  API; the SDK-building class is loaded lazily, only when tracing is enabled. So the in-process container
  test harness needs only the API to instantiate the disabled, no-op provider.
- container-dev: depend on opentelemetry-api only (was sdk + exporter). The SDK is supplied at runtime by
  the pre-installed container-opentelemetry bundle.
- container-dependencies-enforcer: whitelist opentelemetry-api/context/common (provided). These are
  genuinely provided at runtime by the bundle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Fix container-opentelemetry OSGi resolution: mark unused imports optional

The bundle failed to resolve in a real container (system tests): bnd had generated mandatory
Import-Package entries for packages the container does not provide (io.grpc, guava, the OTel
incubator/autoconfigure SPI, jspecify, sun.misc). The earlier assumption that bnd marks these
optional automatically was wrong.

Everything the bundle actually uses at runtime is embedded (private) in the jar. Add an explicit
Import-Package keeping only com.fasterxml.jackson.core mandatory (provided by the container) and
marking all other computed imports resolution:=optional, so the bundle resolves without the
unavailable deps it never exercises.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Address review: export ai.vespa.telemetry, keep OTel off the public container classpath

- configdefinitions: export the generated ai.vespa.telemetry package (@ExportPackage package-info)
  so container-disc can import TelemetryConfig at runtime in a real container (OSGi).
- container: exclude io.opentelemetry:* from the container-dev dependency so the public 3rd-party
  container classpath does not expose OpenTelemetry. It stays internal to the platform (container-disc),
  provided at runtime by the container-opentelemetry bundle.
- container-dependencies-enforcer: drop the OpenTelemetry whitelist entries (no longer exposed).

container-dev keeps opentelemetry-api so the in-process container test harness still resolves the
no-op OpenTelemetryProvider.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Remove optional imports

* Fix bundle embedding: declare all OpenTelemetry modules as direct dependencies

The maven-bundle-plugin did not reliably embed deep transitive OTel modules across build
environments (e.g. opentelemetry-common, 4 levels deep via sdk->api->context->common). When a
module was not embedded, bnd emitted a mandatory versioned Import-Package for it, so the bundle
failed OSGi resolution in the real container (system tests), cascading to container-disc and
standalone-container.

Declare all 12 OTel modules directly so Embed-Dependency embeds them as first-level deps instead
of relying on Embed-Transitive. Same workaround pattern as container-apache-http-client-bundle.
Verified: the built bundle imports only com.fasterxml.jackson.core and no io.opentelemetry packages.

Also keep Import-Package limited to com.fasterxml.jackson.core (everything else is embedded).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: strip stray OSGI-INF/MANIFEST.MF from container-opentelemetry fat bundle

Inlined upstream OpenTelemetry MR-JARs carry their own OSGI-INF/MANIFEST.MF
(notably under META-INF/versions/9/OSGI-INF/), which survive into the
assembled fat bundle. OSGi resolvers only read META-INF/MANIFEST.MF, so the
extra files are inert noise that can mislead anyone inspecting the bundle.

Replace the built-in jar-with-dependencies descriptorRef with a custom
descriptor that mirrors the built-in but excludes OSGI-INF/MANIFEST.MF and
META-INF/versions/*/OSGI-INF/MANIFEST.MF when unpacking dependencies.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Bjørn Christian Seime <bjorn.christian@seime.no>

Author: 3 people

CMakeLists.txt

@@ -117,6 +117,7 @@ add_subdirectory(container-apache-http-client-bundle)
 add_subdirectory(container-disc)
 add_subdirectory(container-messagebus)
 add_subdirectory(container-onnxruntime)
+add_subdirectory(container-opentelemetry)
 add_subdirectory(container-llama)
 add_subdirectory(container-search)
 add_subdirectory(container-search-and-docproc)

config-model-api/abi-spec.json

@@ -1415,6 +1415,7 @@
       "public double feedNiceness()",
       "public int maxUnCommittedMemory()",
       "public boolean containerDumpHeapOnShutdownTimeout()",
+      "public com.yahoo.config.provision.OpenTelemetryConfiguration opentelemetrySdk()",
       "public int heapSizePercentage(java.util.Optional)",
       "public java.util.List allowedAthenzProxyIdentities()",
       "public int maxActivationInhibitedOutOfSyncGroups()",

config-model-api/src/main/java/com/yahoo/config/model/api/ModelContext.java

@@ -14,6 +14,7 @@
 import com.yahoo.config.provision.DockerImage;
 import com.yahoo.config.provision.HostName;
 import com.yahoo.config.provision.NodeResources.Architecture;
+import com.yahoo.config.provision.OpenTelemetryConfiguration;
 import com.yahoo.config.provision.SharedHosts;
 
 import java.io.File;
@@ -96,6 +97,7 @@ interface FeatureFlags {
         @ModelFeatureFlag(owners = {"hmusum"}) default double feedNiceness() { return 0.0; }
         @ModelFeatureFlag(owners = {"hmusum"}) default int maxUnCommittedMemory() { return 130000; }
         @ModelFeatureFlag(owners = {"bjorncs"}) default boolean containerDumpHeapOnShutdownTimeout() { return false; }
+        @ModelFeatureFlag(owners = {"onur"}) default OpenTelemetryConfiguration opentelemetrySdk() { return OpenTelemetryConfiguration.disabled(); }
         @ModelFeatureFlag(owners = {"hmusum"}) default int heapSizePercentage(Optional<String> clusterId) { return  0;}
         @ModelFeatureFlag(owners = {"bjorncs"}) default List<String> allowedAthenzProxyIdentities() { return List.of(); }
         @ModelFeatureFlag(owners = {"vekterli"}) default int maxActivationInhibitedOutOfSyncGroups() { return 0; }

config-model/src/main/java/com/yahoo/vespa/model/container/ContainerCluster.java

@@ -1,6 +1,7 @@
 // Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
 package com.yahoo.vespa.model.container;
 
+import ai.vespa.telemetry.TelemetryConfig;
 import com.yahoo.cloud.config.ClusterInfoConfig;
 import com.yahoo.cloud.config.ConfigserverConfig;
 import com.yahoo.cloud.config.CuratorConfig;
@@ -13,6 +14,7 @@
 import com.yahoo.config.model.producer.AnyConfigProducer;
 import com.yahoo.config.model.producer.TreeConfigProducer;
 import com.yahoo.config.provision.ClusterSpec;
+import com.yahoo.config.provision.OpenTelemetryConfiguration;
 import com.yahoo.config.provision.Zone;
 import com.yahoo.container.ComponentsConfig;
 import com.yahoo.container.QrSearchersConfig;
@@ -109,7 +111,8 @@ public abstract class ContainerCluster<CONTAINER extends Container>
         ClusterInfoConfig.Producer,
         ConfigserverConfig.Producer,
         CuratorConfig.Producer,
-        SchemaInfoConfig.Producer
+        SchemaInfoConfig.Producer,
+        TelemetryConfig.Producer
 {
 
     /**
@@ -168,6 +171,8 @@ public abstract class ContainerCluster<CONTAINER extends Container>
 
     private volatile boolean deferChangesUntilRestart = false;
     private final boolean applyOnRestartForApplicationMetadataConfigEnabled;
+    private final OpenTelemetryConfiguration opentelemetrySdk;
+    private final Map<String, String> telemetryResourceAttributes;
     private boolean clientsLegacyMode;
     private List<Client> clients = List.of();
 
@@ -179,7 +184,9 @@ public ContainerCluster(TreeConfigProducer<?> parent, String configSubId, String
         this.zooKeeperLocalhostAffinity = zooKeeperLocalhostAffinity;
         this.compressionType = "zstd";
         applyOnRestartForApplicationMetadataConfigEnabled = deployState.featureFlags().applyOnRestartForApplicationMetadataConfig();
-        
+        opentelemetrySdk = deployState.featureFlags().opentelemetrySdk();
+        telemetryResourceAttributes = telemetryResourceAttributes(deployState, clusterId);
+
         componentGroup = new ComponentGroup<>(this, "component");
 
         addCommonVespaBundles();
@@ -199,6 +206,8 @@ public ContainerCluster(TreeConfigProducer<?> parent, String configSubId, String
         addSimpleComponent(com.yahoo.container.handler.ClustersStatus.class.getName());
         addSimpleComponent("com.yahoo.container.jdisc.DisabledConnectionLogProvider");
         addSimpleComponent(com.yahoo.jdisc.http.server.jetty.Janitor.class);
+        // OpenTelemetry tracing provider: present in all container types; hands out a no-op instance unless enabled.
+        addSimpleComponent("com.yahoo.container.jdisc.telemetry.OpenTelemetryProvider");
     }
 
     protected abstract boolean messageBusEnabled();
@@ -566,6 +575,32 @@ public void getConfig(QrSearchersConfig.Builder builder) {
         if (containerSearch != null) containerSearch.getConfig(builder);
     }
 
+    @Override
+    public void getConfig(TelemetryConfig.Builder builder) {
+        builder.enabled(opentelemetrySdk.enabled())
+               .endpoint(opentelemetrySdk.endpoint())
+               .samplingRatio(opentelemetrySdk.samplingRatio());
+        builder.resourceAttribute.putAll(telemetryResourceAttributes);
+    }
+
+    /**
+     * Resource attributes describing this container service, filled from the deployment identity available
+     * in the model. Used by the OpenTelemetry provider to build the OTel {@code Resource}.
+     */
+    private Map<String, String> telemetryResourceAttributes(DeployState deployState, String clusterId) {
+        var applicationId = deployState.getProperties().applicationId();
+
+        Map<String, String> attributes = new LinkedHashMap<>();
+        attributes.put("application", applicationId.application().value());
+        attributes.put("tenant", applicationId.tenant().value());
+        attributes.put("zone", this.zone.systemLocalValue());
+        attributes.put("environment", this.zone.environment().value());
+        attributes.put("cloud", this.zone.cloud().toString());
+        attributes.put("cluster.type", "container");
+        attributes.put("cluster.id", clusterId);
+        return attributes;
+    }
+
     @Override
     public void getConfig(QrStartConfig.Builder builder) {
         builder.jvm

config-provisioning/src/main/java/com/yahoo/config/provision/OpenTelemetryConfiguration.java

@@ -0,0 +1,25 @@
+// Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
+package com.yahoo.config.provision;
+
+/**
+ * Settings for Vespa's OpenTelemetry SDK (tracing), controlled by the {@code opentelemetry-sdk}
+ * feature flag and produced into the container's telemetry config.
+ *
+ * @author onur
+ */
+public interface OpenTelemetryConfiguration {
+
+    boolean enabled();
+    String endpoint();
+    double samplingRatio();
+
+    /** The default, disabled configuration: produces a no-op OpenTelemetry. */
+    static OpenTelemetryConfiguration disabled() {
+        return new OpenTelemetryConfiguration() {
+            @Override public boolean enabled() { return false; }
+            @Override public String endpoint() { return ""; }
+            @Override public double samplingRatio() { return 1.0; }
+        };
+    }
+
+}

configdefinitions/src/main/java/ai/vespa/telemetry/package-info.java

@@ -0,0 +1,5 @@
+// Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
+@ExportPackage
+package ai.vespa.telemetry;
+
+import com.yahoo.osgi.annotation.ExportPackage;

configdefinitions/src/vespa/telemetry.def

@@ -0,0 +1,12 @@
+# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
+# Configuration for the container OpenTelemetry tracing provider.
+# Disabled by default: when enabled=false the provider hands out a no-op OpenTelemetry that produces nothing.
+package=ai.vespa.telemetry
+
+enabled            bool
+endpoint           string
+samplingRatio      double
+
+# Resource attributes describing the emitting service, filled by the config-model
+# from whatever deployment identity is available (application, tenant, cluster type, cluster id).
+resourceAttribute{} string

configserver/src/main/java/com/yahoo/vespa/config/server/deploy/ModelContextImpl.java

@@ -28,6 +28,7 @@
 import com.yahoo.config.provision.DockerImage;
 import com.yahoo.config.provision.HostName;
 import com.yahoo.config.provision.NodeResources.Architecture;
+import com.yahoo.config.provision.OpenTelemetryConfiguration;
 import com.yahoo.config.provision.SharedHosts;
 import com.yahoo.vespa.flags.Flag;
 import com.yahoo.vespa.flags.FlagSource;
@@ -223,6 +224,7 @@ private <T, F extends Flag<T, F>, U extends UnboundFlag<T, F, U>> ModelContext.F
         @Override public double feedConcurrency() { return flag(PermanentFlags.FEED_CONCURRENCY).value(); }
         @Override public double feedNiceness() { return flag(PermanentFlags.FEED_NICENESS).value(); }
         @Override public int mbusNetworkThreads() { return flag(Flags.MBUS_NUM_NETWORK_THREADS).value(); }
+        @Override public OpenTelemetryConfiguration opentelemetrySdk() { return flag(Flags.OPENTELEMETRY_SDK).value(); }
         @Override public List<String> allowedAthenzProxyIdentities() { return flag(PermanentFlags.ALLOWED_ATHENZ_PROXY_IDENTITIES).value(); }
         @Override public int maxActivationInhibitedOutOfSyncGroups() { return flag(PermanentFlags.MAX_ACTIVATION_INHIBITED_OUT_OF_SYNC_GROUPS).value(); }
         @Override public double resourceLimitDisk() { return flag(PermanentFlags.RESOURCE_LIMIT_DISK).value(); }

container-dependency-versions/pom.xml

@@ -36,6 +36,13 @@
 
     <dependencyManagement>
         <dependencies>
+            <dependency>
+                <groupId>io.opentelemetry</groupId>
+                <artifactId>opentelemetry-bom</artifactId>
+                <version>${opentelemetry.vespa.version}</version>
+                <type>pom</type>
+                <scope>import</scope>
+            </dependency>
             <dependency>
                 <groupId>aopalliance</groupId>
                 <artifactId>aopalliance</artifactId>

container-dev/pom.xml

@@ -187,6 +187,14 @@
       <version>${project.version}</version>
       <type>pom</type>
     </dependency>
+    <!-- OpenTelemetry API only: this is what the in-process container test harness needs to instantiate the
+         (disabled, no-op) OpenTelemetryProvider, and all that 3rd-party container projects should compile
+         against. The SDK is loaded lazily by OpenTelemetrySdkBuilder and supplied at runtime by the
+         pre-installed container-opentelemetry bundle. -->
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-api</artifactId>
+    </dependency>
     <dependency>
       <groupId>com.yahoo.vespa</groupId>
       <artifactId>vespajlib</artifactId>