com.azure...util..DefaultTracerProvider looks for default tracer agents in case any exception occurs while calling any Azure core lib

[narendrasinghdanu]

Type of issue

Other (describe below)

Description

Issue: At the time of springboot application start, whenever there is an exception in any Azure core package (for example, com.azure.security.*), com.azure.core.util.tracing.DefaultTracerProvider looks for the default DefaultTracerProvider, which eventually looks for the Java class io.opentelemetry.javaagent.instrumentation.opentelemetryapi.context.AgentContextStorage.
My use case is to push the traces into azure app insight without using the applicationinsight java agent therefore i am creating my customise Opentelemetry and we don't want to use the applicationinsightjavagent jar file

So When this entire episode happens , it eventually call the AutoConfiguredOpenTelemetrySdk and set the default telemetry and tracer objects and which does not have any use.

Library causing the issue:

com.azure
azure-core-tracing-opentelemetry
1.0.0-beta.50

fyi- My use case is to push the traces into azure app insight without using the application-insight java agent therefore i am creating my customise Opentelemetry.

Page URL

https://learn.microsoft.com/en-us/java/api/overview/azure/core-tracing-opentelemetry-readme?view=azure-java-preview

Content source URL

https://github.com/Azure/azure-docs-sdk-java/blob/master/docs-ref-services/preview/core-tracing-opentelemetry-readme.md

Document Version Independent Id

249ef132-a34b-8c4b-e6e5-ec5dffbe1a30

Article author

@azure-sdk

Metadata

• ID: 0d81d5b8-0d75-0082-cb01-dab0cf98d728
• Service: core

[joshfree]

@trask @lmolkova could you please assist?

[trask]

hi @narendrasinghdanu! the Azure SDK should work with both upstream OpenTelemetry Java agent, and also upstream OpenTelemetry Java SDK

I'm not quite following the issue you are seeing, can you provide more details, or even better a repro that we can run and use to investigate?

[narendrasinghdanu]

Hi @trask ,You are right, and that is my understanding as well. However, it is not working the same way. Below is my code where we expect it to create and push traces to App Insights whenever the application starts, and it is working perfectly. But in case there are any exceptions/issues occurring in any of the Azure library packages, for example, in the "Azure Key Vault Secrets client library" or Azure Blob Storage client library, while starting your Spring Boot project, it is looking for Java agents internally even though they are not defined there, which should not be the behaviour.

My OpenTelemetry bean:(this class should able to send traces to azure insight)

    @Bean
public OpenTelemetry openTelemetry() {
	String connectionString = secret.getConnetionStr()
	AzureMonitorExporterBuilder azureMonitorExporterBuilder = new AzureMonitorExporterBuilder()
		.connectionString(connectionString);
	BatchSpanProcessor batchSpanProcessor = BatchSpanProcessor
		.builder(azureMonitorExporterBuilder.buildTraceExporter())
		.build();
	Resource resource = Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, this.serviceName));
	SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
		.addSpanProcessor(batchSpanProcessor)
		.setResource(resource)
		.build();
	OpenTelemetrySdk openTelemetrySdk = OpenTelemetrySdk.builder()
		.setTracerProvider(sdkTracerProvider)
	
		.buildAndRegisterGlobal();
	log.info("OpenTelemetry initialized");
	return openTelemetrySdk;
}

Springboot Logs where you can see that it is call the AgentContextStorage.getAgentContext

2024-10-16 15:32:06.021 DEBUG [main] [Logging_service - ClientLogger.performDeferredLogging:564] [, ] - Azure Identity => ERROR in EnvironmentCredential: Failed to create a ClientSecretCredential or ClientCertificateCredential. Missing required environment variable either AZURE_CLIENT_SECRET or AZURE_CLIENT_CERTIFICATE_PATH
2024-10-16 15:32:06.022 DEBUG [main] [Logging_service - ClientLogger.performDeferredLogging:564] [, ] - Azure Identity => ERROR in EnvironmentCredential: Failed to determine an authentication scheme based on the available environment variables. Please specify AZURE_TENANT_ID and AZURE_CLIENT_SECRET to authenticate through a ClientSecretCredential; AZURE_TENANT_ID and AZURE_CLIENT_CERTIFICATE_PATH to authenticate through a ClientCertificateCredential; or AZURE_USERNAME and AZURE_PASSWORD to authenticate through a UserPasswordCredential.
2024-10-16 15:32:06.088 DEBUG [main] [Logging_service - Loggers$Slf4JLogger.debug:254] [, ] - Using Slf4j logging framework
2024-10-16 15:32:06.140 DEBUG [main] [Logging_service - ClientLogger.performLogging:513] [, ] - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID
2024-10-16 15:32:06.142 DEBUG [main] [Logging_service - ClientLogger.performLogging:513] [, ] - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID
2024-10-16 15:32:06.176 DEBUG [main] [Logging_service - ClientLogger.performDeferredLogging:564] [, ] - Using com.azure.core.tracing.opentelemetry.OpenTelemetryTracerProvider as the default com.azure.core.util.tracing.TracerProvider.
2024-10-16 15:32:06.185 DEBUG [main] [Logging_service - ClientLogger.performLogging:513] [, ] - Failed to resolve AgentContextStorage.getAgentContext or one of its dependencies java.lang.ClassNotFoundException: io.opentelemetry.javaagent.instrumentation.opentelemetryapi.context.AgentContextStorage at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592) at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:107) at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:413) at java.base/java.lang.Class.forName(Class.java:404) at com.azure.core.tracing.opentelemetry.OpenTelemetryTracer.(OpenTelemetryTracer.java:75) at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:340) at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103) at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:173) at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:785) at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:725) at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1397) at com.azure.core.util.tracing.DefaultTracerProvider.createFallbackTracer(DefaultTracerProvider.java:35) at com.azure.core.util.tracing.DefaultTracerProvider.(DefaultTracerProvider.java:25) at com.azure.core.util.tracing.TracerProvider.getDefaultProvider(TracerProvider.java:43) at com.azure.security.keyvault.secrets.SecretClientBuilder.buildInnerClient(SecretClientBuilder.java:239) at com.azure.security.keyvault.secrets.SecretClientBuilder.buildClient(SecretClientBuilder.java:158) at com.sample.org.secrets.keyvault.AzureKeyVaultSecretClient.(AzureKeyVaultSecretClient.java:46) at com.sample.org.config.KeyVaultConfig.secretService(KeyVaultConfig.java:36) at com.sample.org.config.KeyVaultConfig$$SpringCGLIB$$0.CGLIB$secretService$0() at com.sample.org.config.KeyVaultConfig$$SpringCGLIB$$FastClass$$1.invoke() at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258) at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:348) at com.sample.org.config.KeyVaultConfig$$SpringCGLIB$$0.secretService() at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:785) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:768) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:509) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1439) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:599) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:313) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:954) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:625) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352) at com.sample.org.BootstrapApplication.main(FraudBootstrapApplication.java:21) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102) at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64) at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:40)
2024-10-16 15:32:06.213 INFO [main] [Logging_service - GlobalOpenTelemetry.maybeAutoConfigureAndSetGlobal:234] [, ] - AutoConfiguredOpenTelemetrySdk found on classpath but automatic configuration is disabled. To enable, run your JVM with -Dotel.java.global-autoconfigure.enabled=true

Note :Once the above step happens, we started getting an exception that the default OpenTelemetry is already created and cannot assign a new OpenTelemetry bean.
Exception Details:
024-10-07 17:15:28.124 ERROR [main] [Logging_service - SpringApplication.reportFailure:859] [, ] - Application run failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'openTelemetry' defined in class path resource [com/sample/org/config/OpenTelemetryConfig.class]: Failed to instantiate [io.opentelemetry.api.OpenTelemetry]: Factory method 'openTelemetry' threw exception with message: GlobalOpenTelemetry.set has already been called. GlobalOpenTelemetry.set must be called only once before any calls to GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set to cause of this exception. at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:648) at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485)

------ my observation that it is that----
When you add com.azure.azure-core-tracing-opentelemetry:1.0.0-beta.50 , it automatically looks for java agents incase of exception /or any issue.

[narendrasinghdanu]

@trask @lmolkova , anything else needed from my side

[trask]

hey @narendrasinghdanu, sorry for the delay getting back

this debug log is expected:

2024-10-16 15:32:06.185 DEBUG [main] Failed to resolve AgentContextStorage.getAgentContext or one of its dependencies

check out

azure-sdk-for-java/sdk/core/azure-core-tracing-opentelemetry/src/main/java/com/azure/core/tracing/opentelemetry/OpenTelemetryTracer.java

Lines 85 to 86 in d4f12d3

	// it's expected if we're not running in the agent
	LOGGER.verbose("Failed to resolve AgentContextStorage.getAgentContext or one of its dependencies", t);