Exporter for Metrics Defaults to gRPC Even When Explicitly Set

[jeffNedley]

Describe the bug
Using the Spring Boot Zero-Code Implementation and following the documentation closely, I cannot seem to get the implementation to work without seeing this error message.

From what it seems, even after setting the endpoint and protocol values for what my team needs (http/protobuf), we always receive the error saying, "Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317". This is incorrect for two reasons. One, because I have set the endpoint using the application.yml and two, because I have set the protocol using the application.yml.

Here is our config (application.yml) below:

otel:
  service:
    name: <redacted-for-security>
  exporter:
    otlp:
      traces:
        endpoint:  http://<redacted-for-security>:4318/v1/traces
        protocol: http/protobuf
      metrics:
        endpoint:  http://<redacted-for-security>:4318/v1/metrics 
        protocol: http/protobuf
      logs:
        endpoint:  http://<redacted-for-security>:4318/v1/logs
        protocol: http/protobuf
  resource:
    attributes:
      deployment:
        environment: <redacted-for-security>

Here is our dependency config (pom.xml):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>3.1.12</version>
		<relativePath/>
	</parent>
	<groupId><redacted-for-security></groupId>
	<artifactId><redacted-for-security></artifactId>
	<version>0.1.6-SNAPSHOT</version>
	<name><redacted-for-security></name>
	<properties>
		<project.version>0.1.8-SNAPSHOT</project.version>
		<java.version>21</java.version>
		<argLine>-Dnet.bytebuddy.experimental=true</argLine>
		<!-- Remove when Mockito gets updated
        for Java 21 -->
		<confluent.version>7.4.0</confluent.version>
		<avro.version>1.11.3</avro.version>
		<avro.base.directory>src/main/resources/avro/</avro.base.directory>
		<avro.schema.directory>src/main/resources/avro/schemas</avro.schema.directory>
		<avro.idl.directory>src/main/resources/avro/idl</avro.idl.directory>
		<redacted-for-security>
	</properties>
	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>io.opentelemetry.instrumentation</groupId>
				<artifactId>opentelemetry-instrumentation-bom</artifactId>
				<version>2.11.0</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>
	<dependencies>
		<!-- OTEL DEPENDENCY -->
		<dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-spring-boot-starter</artifactId>
		</dependency>
		
		<!-- INTERNAL DEPENDENCIES /////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.32</version>
			<scope>provided</scope>
		</dependency>
		<!-- SPRING DEPENDENCIES ///////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.retry</groupId>
			<artifactId>spring-retry</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-aspects</artifactId>
		</dependency>
		<!-- ELASTICSEARCH DEPENDENCIES ////////////////////////////////////////// -->
		<dependency>
			<groupId>co.elastic.clients</groupId>
			<artifactId>elasticsearch-java</artifactId>
			<version>8.16.0</version>
		</dependency>
		<dependency>
			<groupId>jakarta.json</groupId>
			<artifactId>jakarta.json-api</artifactId>
			<version>2.0.1</version>
		</dependency>
		<dependency>
			<groupId>jakarta.annotation</groupId>
			<artifactId>jakarta.annotation-api</artifactId>
			<version>3.0.0</version>
		</dependency>
		<dependency>
			<groupId>com.google.code.gson</groupId>
			<artifactId>gson</artifactId>
			<version>2.10.1</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.datatype</groupId>
			<artifactId>jackson-datatype-jsr310</artifactId>
		</dependency>
		<!-- LEADERSHIP DEPENDENCIES/////////////////////////////////////////////////// -->		
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-kubernetes-fabric8-leader</artifactId>
			<version>3.0.2</version>
		</dependency>
		<dependency>
			<groupId>org.awaitility</groupId>
			<artifactId>awaitility</artifactId>
			<version>4.2.2</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.integration</groupId>
			<artifactId>spring-integration-core</artifactId>
			<version>6.3.5</version>
		</dependency>
		<!-- KAFKA DEPENDENCIES/////////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka</artifactId>
			<version>3.2.4</version>
		</dependency>
		<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka-streams</artifactId>
			<version>3.9.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.kafka</groupId>
			<artifactId>kafka-storage-api</artifactId>
		</dependency>
		<dependency>
			<groupId>org.apache.avro</groupId>
			<artifactId>avro</artifactId>
			<version>${avro.version}</version>
		</dependency>
		<dependency>
			<groupId>org.apache.avro</groupId>
			<artifactId>avro-maven-plugin</artifactId>
			<version>${avro.version}</version>
		</dependency>
		<dependency>
			<groupId>io.confluent</groupId>
			<artifactId>kafka-avro-serializer</artifactId>
			<version>${confluent.version}</version>
		</dependency>
		<dependency>
			<groupId>io.confluent</groupId>
			<artifactId>kafka-streams-avro-serde</artifactId>
			<version>${confluent.version}</version>
		</dependency>
		<dependency>
			<groupId>io.confluent</groupId>
			<artifactId>kafka-schema-registry-client</artifactId>
			<version>${confluent.version}</version>
		</dependency>
		<!-- TEST DEPENDENCIES /////////////////////////////////////////////////// -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka-test</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.junit.jupiter</groupId>
			<artifactId>junit-jupiter-api</artifactId>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.mockito</groupId>
			<artifactId>mockito-core</artifactId>
			<scope>test</scope>
		</dependency>
	</dependencies>
	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<configuration>
					<excludes>
						<exclude>
							<groupId>org.projectlombok</groupId>
							<artifactId>lombok</artifactId>
						</exclude>
					</excludes>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.avro</groupId>
				<artifactId>avro-maven-plugin</artifactId>
				<version>${avro.version}</version>
				<executions>
					<execution>
						<id>idl-protocol</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>idl-protocol</goal>
						</goals>
						<configuration>
							<stringType>String</stringType>
							<sourceDirectory>${avro.idl.directory}</sourceDirectory>
							<outputDirectory>${project.basedir}/src/main/java</outputDirectory>
							<includes>
								<include>**/*.avdl</include>
							</includes>
						</configuration>
					</execution>
				</executions>
				<configuration>
					<javaSourceDirectories>
						<redacted-for-security>
					</javaSourceDirectories>
					<avroOutputDirectory>${avro.schema.directory}</avroOutputDirectory>
				</configuration>
			</plugin>
			<!-- CMD: 'mvn schema-registry:register' -->
			<plugin>
				<groupId>io.confluent</groupId>
				<artifactId>kafka-schema-registry-maven-plugin</artifactId>
				<version>${confluent.version}</version>
				<redacted-for-security>
				<goals>
					<goal>register</goal>
				</goals>
			</plugin>
			<plugin>
				<groupId>com.google.cloud.tools</groupId>
				<artifactId>jib-maven-plugin</artifactId>
				<version>3.3.2</version>
				<redacted-for-security>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-javadoc-plugin</artifactId>
				<version>3.6.0</version>
				<configuration>
					<source>21</source>
					<target>21</target>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.jacoco</groupId>
				<artifactId>jacoco-maven-plugin</artifactId>
				<version>0.8.12</version>
				<configuration>
					<excludes>
						<exclude>META-INF/**</exclude>
						<exclude>${avro.java.shared.directory}</exclude>
						<exclude>${avro.java.events.directory}</exclude>
						<exclude>${avro.base.directory}</exclude>
					</excludes>
				</configuration>
				<executions>
					<execution>
						<goals>
							<goal>prepare-agent</goal>
						</goals>
					</execution>
					<execution>
						<id>report</id>
						<phase>prepare-package</phase>
						<goals>
							<goal>report</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-clean-plugin</artifactId>
				<configuration>
					<filesets>
						<fileset>
							<directory>${avro.java.events.directory}</directory>
							<includes>
								<include>**/*</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
						<fileset>
							<directory>${avro.java.shared.directory}</directory>
							<includes>
								<include>**/*</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
						<fileset>
							<directory>${avro.schema.directory}</directory>
							<includes>
								<include>**/*</include>
							</includes>
							<followSymlinks>false</followSymlinks>
						</fileset>
					</filesets>
				</configuration>
			</plugin>
		</plugins>
	</build>
	<distributionManagement>
		<redacted-for-security>
	</distributionManagement>
</project>

Error that comes up constantly:

2025-01-09 15:24:39.910ERROR --- i.o.exporter.internal.grpc.GrpcExporter  : Failed to export metrics. The request could not be executed. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317

java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:91) ~[opentelemetry-exporter-sender-okhttp-1.44.1.jar:1.44.1]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ~[okhttp-4.10.0.jar:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
        Suppressed: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:4317
                ... 21 common frames omitted
        Caused by: java.net.ConnectException: Connection refused
                at java.base/sun.nio.ch.Net.pollConnect(Native Method)
                at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
                at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
                at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
                at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
                at java.base/java.net.Socket.connect(Socket.java:751)
                at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
                at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
                ... 20 common frames omitted
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[na:na]
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) ~[na:na]
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[na:na]
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[na:na]
        at java.base/java.net.Socket.connect(Socket.java:751) ~[na:na]
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.10.0.jar:na]
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.10.0.jar:na]
        ... 20 common frames omitted

Steps to reproduce
Follow OpenTelemetry documentation for Zero-Code Implementation under the Spring Boot Starter section.

What did you expect to see?
Integration and all telemetry data to show up in my Collector with no errors in the logs

What did you see instead?
All telemetry except a log of an error was shown repeatedly

What version and what artifacts are you using?
Artifacts: opentelemetry-instrumentation-bom, opentelemetry-spring-boot-starter
Version: 2.11.0
How did you reference these artifacts?
pom.xml Maven

Environment
Compiler: openjdk 21.0.4 2024-07-16 LTS
OS: RedHat Linux
Runtime: Corretto-21.0.4.7.1 (build 21.0.4+7-LTS)
OS RedHat Linux

[jeffNedley]

Found additional information on this issue!

I was able to mitigate the issue by instead using environment variables rather than the spring application.yml. I don't count this as a non-issue though since the documentation clearly states that I should be able to use either.

I found that the spring boot starter implementation was not reading ANY of my configurations in the application.yml!

[trask]

transferring to opentelemetry-java-instrumentation repo where the opentelemetry-spring-boot-starter lives

cc @jeanbisutti @zeitlinger

[laurit]

@jeffNedley a minimal sample application that reproduces the issue could help.
Perhaps the issue is that there are 2 OpenTelemetry instances. One that is configured by the spring starter and another one that uses the default configuration?

[jeffNedley]

What would be the other one though?

The only dependencies installed is the spring starter. I mean I followed the documentation exactly as is and still my configurations in my application.yml do not work.

[laurit]

What would be the other one though?

I guess it could happen when otel.java.global-autoconfigure.enabled is set to true and GlobalOpenTelemetry.get() is called since as far as I know spring starter does not register a global instance. For example I believe that the opentelemetry instrumentation in recent versions of elasticsearch-java calls GlobalOpenTelemetry.get(). Though the reason for what you have observed could also be something completely different. If you can use a debugger then it should be easy to verify where the autoconfigured OpenTelemetry instances are created and how many there are by setting a breakpoint in https://github.com/open-telemetry/opentelemetry-java/blob/d13fd38df1ef418dac4bf30be6514d7a56e6ba03/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdk.java#L47 or some other method that is called during the creation of an OpenTelemetry instance.