Skip to content

One load table API request results in two STS AssumeRole requests #3292

New issue
New issue
Open
Open
One load table API request results in two STS AssumeRole requests#3292
Assignees
dimas-b
Labels
bugSomething isn't working
@dimas-b

Description

@dimas-b
dimas-b
opened on Dec 17, 2025

Describe the bug

First STS call:

java.lang.Throwable: test1
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
	at com.intellij.rt.debugger.MethodInvoker.invokeInternal(MethodInvoker.java:223)
	at com.intellij.rt.debugger.MethodInvoker.invoke1(MethodInvoker.java:35)
	at org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration.getSubscopedCreds(AwsCredentialsStorageIntegration.java:96)
	at org.apache.polaris.core.persistence.AtomicOperationMetaStoreManager.getSubscopedCredsForEntity(AtomicOperationMetaStoreManager.java:1641)
	at org.apache.polaris.core.persistence.ServiceProducers_ProducerMethod_polarisMetaStoreManager_2QgveElh41S2GrgpagbE0iTcmng_ClientProxy.getSubscopedCredsForEntity(Unknown Source)
	at org.apache.polaris.core.storage.StorageCredentialsVendor.getSubscopedCredsForEntity(StorageCredentialsVendor.java:73)
	at org.apache.polaris.core.storage.ServiceProducers_ProducerMethod_storageCredentialsVendor_chZ10WlGqG-XefHi-NQqFKWM7fY_ClientProxy.getSubscopedCredsForEntity(Unknown Source)
	at org.apache.polaris.core.storage.cache.StorageCredentialCache.lambda$getOrGenerateSubScopeCreds$2(StorageCredentialCache.java:141)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$0(BoundedLocalCache.java:2707)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2705)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2686)
	at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
	at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:63)
	at org.apache.polaris.core.storage.cache.StorageCredentialCache.getOrGenerateSubScopeCreds(StorageCredentialCache.java:161)
	at org.apache.polaris.core.storage.cache.ServiceProducers_ProducerMethod_storageCredentialCache_hzAWPa00ffa2II6zBfUMmDXk9AQ_ClientProxy.getOrGenerateSubScopeCreds(Unknown Source)
	at org.apache.polaris.service.catalog.io.StorageAccessConfigProvider.getStorageAccessConfig(StorageAccessConfigProvider.java:120)
	at org.apache.polaris.service.catalog.io.StorageAccessConfigProvider_ClientProxy.getStorageAccessConfig(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.loadFileIOForTableLike(IcebergCatalog.java:2084)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.lambda$doRefresh$0(IcebergCatalog.java:1396)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$PolarisOperationsBase.lambda$refreshFromMetadataLocation$0(IcebergCatalog.java:2035)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$PolarisOperationsBase.refreshFromMetadataLocation(IcebergCatalog.java:2035)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.doRefresh(IcebergCatalog.java:1385)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.refresh(IcebergCatalog.java:1297)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.current(IcebergCatalog.java:1288)
	at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.loadTable(IcebergCatalogHandler.java:809)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$loadTable$8(IcebergCatalogAdapter.java:396)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalogByName(IcebergCatalogAdapter.java:161)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:153)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.loadTable(IcebergCatalogAdapter.java:391)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.loadTable$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator_Gj_WCptqTcdHu-fbZfgVkAwPXCI_Delegate_Subclass.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator.loadTable(IcebergRestCatalogEventServiceDelegator.java:319)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.loadTable(IcebergRestCatalogApi.java:565)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.loadTable$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$12.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
	at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
	at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:47)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor$RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$loadTable_56b49d5a1874f2749e5229d18bb00aeb8fe7fdc8.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:183)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Second STS call:

java.lang.Throwable: test2
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
	at com.intellij.rt.debugger.MethodInvoker.invokeInternal(MethodInvoker.java:223)
	at com.intellij.rt.debugger.MethodInvoker.invoke1(MethodInvoker.java:35)
	at org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration.getSubscopedCreds(AwsCredentialsStorageIntegration.java:96)
	at org.apache.polaris.core.persistence.AtomicOperationMetaStoreManager.getSubscopedCredsForEntity(AtomicOperationMetaStoreManager.java:1641)
	at org.apache.polaris.core.persistence.ServiceProducers_ProducerMethod_polarisMetaStoreManager_2QgveElh41S2GrgpagbE0iTcmng_ClientProxy.getSubscopedCredsForEntity(Unknown Source)
	at org.apache.polaris.core.storage.StorageCredentialsVendor.getSubscopedCredsForEntity(StorageCredentialsVendor.java:73)
	at org.apache.polaris.core.storage.ServiceProducers_ProducerMethod_storageCredentialsVendor_chZ10WlGqG-XefHi-NQqFKWM7fY_ClientProxy.getSubscopedCredsForEntity(Unknown Source)
	at org.apache.polaris.core.storage.cache.StorageCredentialCache.lambda$getOrGenerateSubScopeCreds$2(StorageCredentialCache.java:141)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$0(BoundedLocalCache.java:2707)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2705)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2686)
	at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
	at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:63)
	at org.apache.polaris.core.storage.cache.StorageCredentialCache.getOrGenerateSubScopeCreds(StorageCredentialCache.java:161)
	at org.apache.polaris.core.storage.cache.ServiceProducers_ProducerMethod_storageCredentialCache_hzAWPa00ffa2II6zBfUMmDXk9AQ_ClientProxy.getOrGenerateSubScopeCreds(Unknown Source)
	at org.apache.polaris.service.catalog.io.StorageAccessConfigProvider.getStorageAccessConfig(StorageAccessConfigProvider.java:120)
	at org.apache.polaris.service.catalog.io.StorageAccessConfigProvider_ClientProxy.getStorageAccessConfig(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.buildLoadTableResponseWithDelegationCredentials(IcebergCatalogHandler.java:859)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.loadTable(IcebergCatalogHandler.java:814)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$loadTable$8(IcebergCatalogAdapter.java:396)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalogByName(IcebergCatalogAdapter.java:161)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:153)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.loadTable(IcebergCatalogAdapter.java:391)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.loadTable$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator_Gj_WCptqTcdHu-fbZfgVkAwPXCI_Delegate_Subclass.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergRestCatalogEventServiceDelegator.loadTable(IcebergRestCatalogEventServiceDelegator.java:319)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_Subclass.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.loadTable(IcebergRestCatalogApi.java:565)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.loadTable$$superforward(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$12.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.Future.from(Future.java:85)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364)
	at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20)
	at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)
	at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205)
	at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
	at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)
	at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:47)
	at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor$RolesAllowedInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.loadTable(Unknown Source)
	at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$loadTable_56b49d5a1874f2749e5229d18bb00aeb8fe7fdc8.invoke(Unknown Source)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:183)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

To Reproduce

  1. Restart Polaris (to clean caches)
  2. Put a break point into AwsCredentialsStorageIntegration.getSubscopedCreds()
  3. Make a load table request (e.g. from Spark)
  4. Observe two getSubscopedCreds() calls. Note: the only different in parameters is the refreshCredentialsEndpoint value (empty vs. non-empty).

Actual Behavior

No response

Expected Behavior

Efficient Credentials Cache behaviour leading to at most one STS call for the same set of STS parameters within the STS session validity window.

Additional context

Commit c6ee521

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions