Apache Iceberg version
1.11.0 (latest release)
Query engine
Spark
Please describe the bug 馃悶
We run a single Spark Connect cluster (Spark 4.1.2) serving several tenants. Each tenant has its own Glue catalog in its own AWS account, reached with AssumeRoleAwsClientFactory:
spark.sql.catalog.tenant org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.tenant.type glue
spark.sql.catalog.tenant.io-impl org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.tenant.client.factory org.apache.iceberg.aws.AssumeRoleAwsClientFactory
spark.sql.catalog.tenant.client.assume-role.arn arn:aws:iam::111122223333:role/lakehouse-access
spark.sql.catalog.tenant.client.assume-role.region us-east-2
spark.sql.catalog.tenant.glue.id 111122223333
The cluster itself runs on EKS as an IRSA role in a different account, which deliberately has no access to the tenant buckets:
spark.hadoop.fs.s3.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.aws.credentials.provider com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Nightly maintenance calls expire_snapshots and then remove_orphan_files on each table. The first one works, the second one gets a 403, seconds later, on the same table:
-- ok
CALL tenant.system.expire_snapshots(table => 'db.my_table', older_than => TIMESTAMP '2026-08-21 00:00:00');
-- 403
CALL tenant.system.remove_orphan_files(table => 'db.my_table', older_than => TIMESTAMP '2026-08-25 00:00:00');
java.io.UncheckedIOException: java.nio.file.AccessDeniedException:
s3://my-lakehouse-bucket/iceberg_tables/my_table: listObjects() on s3://my-lakehouse-bucket/iceberg_tables/my_table:
software.amazon.awssdk.services.s3.model.AccessDeniedException:
User: arn:aws:sts::444455556666:assumed-role/spark-cluster-irsa/... is not authorized to perform:
s3:ListBucket on resource: "arn:aws:s3:::my-lakehouse-bucket" (Service: S3, Status Code: 403)
at org.apache.iceberg.util.FileSystemWalker.listDirRecursivelyWithHadoop(FileSystemWalker.java:150)
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.listedFileDS(DeleteOrphanFilesSparkAction.java:440)
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.doExecute(DeleteOrphanFilesSparkAction.java:257)
at org.apache.iceberg.spark.procedures.RemoveOrphanFilesProcedure.call(RemoveOrphanFilesProcedure.java:153)
The principal in that error is the cluster's own IRSA role, which appears nowhere in the catalog config. So the listing is not using the catalog's credentials, while the rest of the procedure is. The cause is that the action takes its Hadoop config from the session and drops the catalog it came from:
// DeleteOrphanFilesSparkAction.java:139
this.hadoopConf = new SerializableConfiguration(spark.sessionState().newHadoopConf());
newHadoopConf() only sees spark.hadoop.*. Everything else in the procedure goes through table.io(), which the catalog built with the assume-role factory, which is why expire_snapshots is fine.
I found #11541, which describes this exact coupling, and PR #12254 which closed it by adding the FileIO listing path. But that path is opt-in (prefix_listing, usePrefixListing defaults to false at DeleteOrphanFilesSparkAction.java:133), so the Hadoop path still behaves this way and it is what you get unless you know to ask for the other one.
prefix_listing => true does fix the credentials for us, but we can't use it: it lists serially on the driver and then does parallelize(matchingFiles, 1) (#16932, #17387). On Spark Connect there's an extra ceiling, since that single partition puts the whole listing under spark.rpc.message.maxSize, which is a server startup setting we can't raise from a client session. So we're stuck with the Hadoop path, and the Hadoop path has the wrong credentials.
What took me longest to work out is that there is no per-catalog knob for this at all. SparkUtil.hadoopConfCatalogOverrides(SparkSession, catalogName) already exists and applies spark.sql.catalog.<name>.hadoop.* on top of the session config, which is exactly what's needed here, but it only has two callers, both in SparkCatalog (SparkCatalog.java:142 and :717). No action or procedure uses it. So setting spark.sql.catalog.tenant.hadoop.fs.s3a.* is accepted, looks right, and is then ignored by the walk. I assumed I had a typo for a while.
Proposal: build the action's Hadoop config with SparkUtil.hadoopConfCatalogOverrides(spark, catalogName) instead of spark.sessionState().newHadoopConf(), and have that method derive the S3A settings from the catalog's own client.assume-role.arn, so the role only has to be declared once:
fs.s3a.aws.credentials.provider becomes AssumedRoleCredentialProviderfs.s3a.assumed.role.arn becomes the catalog's arn- whatever identity the session already resolved stays on as
fs.s3a.assumed.role.credentials.provider, so it is still what signs the AssumeRole call. Without that step S3A falls back to its SimpleAWSCredentialsProvider default and IRSA or instance-profile clusters end up with no credentials for the STS call at all.
Because that config is per catalog there's nothing to scope by bucket, unlike the session-wide workaround below. spark.sql.catalog.<name>.hadoop.* is copied over the derived values afterwards, so anyone who wants different S3A settings for a catalog can still say so, and catalogs that declare no assume-role arn are untouched. BaseProcedure already holds the catalog (tableCatalog()), so the name is available where the action is built.
It is a behaviour change for catalogs that do declare a role: their listing starts going through that role rather than through the cluster identity. That's the point of the fix, and it matches what the rest of the procedure already does, but it is worth calling out.
I have this working against spark/v4.1 with tests, and can open a PR plus the v4.0 and v3.5 backports. If the preference is to leave the Hadoop path alone because the FileIO one is where things are heading, then at minimum the docs should say that the listing uses the session's Hadoop config and not the catalog's, because nothing in the config surface hints at it.
For anyone else hitting the same 403, the workaround is to give S3A the same role, scoped to the buckets:
spark.hadoop.fs.s3a.bucket.my-lakehouse-bucket.aws.credentials.provider org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
spark.hadoop.fs.s3a.bucket.my-lakehouse-bucket.assumed.role.arn arn:aws:iam::111122223333:role/lakehouse-access
spark.hadoop.fs.s3a.assumed.role.credentials.provider com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Per-bucket options key on the bucket name from the URI (fs.s3a.bucket.<bucket>.), independent of the scheme, so this covers s3:// locations too. The role's trust policy has to allow the cluster principal to assume it, which it already does if the catalog is working.
Willingness to contribute
Apache Iceberg version
1.11.0 (latest release)
Query engine
Spark
Please describe the bug 馃悶
We run a single Spark Connect cluster (Spark 4.1.2) serving several tenants. Each tenant has its own Glue catalog in its own AWS account, reached with
AssumeRoleAwsClientFactory:The cluster itself runs on EKS as an IRSA role in a different account, which deliberately has no access to the tenant buckets:
Nightly maintenance calls
expire_snapshotsand thenremove_orphan_fileson each table. The first one works, the second one gets a 403, seconds later, on the same table:The principal in that error is the cluster's own IRSA role, which appears nowhere in the catalog config. So the listing is not using the catalog's credentials, while the rest of the procedure is. The cause is that the action takes its Hadoop config from the session and drops the catalog it came from:
newHadoopConf()only seesspark.hadoop.*. Everything else in the procedure goes throughtable.io(), which the catalog built with the assume-role factory, which is whyexpire_snapshotsis fine.I found #11541, which describes this exact coupling, and PR #12254 which closed it by adding the
FileIOlisting path. But that path is opt-in (prefix_listing,usePrefixListingdefaults tofalseatDeleteOrphanFilesSparkAction.java:133), so the Hadoop path still behaves this way and it is what you get unless you know to ask for the other one.prefix_listing => truedoes fix the credentials for us, but we can't use it: it lists serially on the driver and then doesparallelize(matchingFiles, 1)(#16932, #17387). On Spark Connect there's an extra ceiling, since that single partition puts the whole listing underspark.rpc.message.maxSize, which is a server startup setting we can't raise from a client session. So we're stuck with the Hadoop path, and the Hadoop path has the wrong credentials.What took me longest to work out is that there is no per-catalog knob for this at all.
SparkUtil.hadoopConfCatalogOverrides(SparkSession, catalogName)already exists and appliesspark.sql.catalog.<name>.hadoop.*on top of the session config, which is exactly what's needed here, but it only has two callers, both inSparkCatalog(SparkCatalog.java:142and:717). No action or procedure uses it. So settingspark.sql.catalog.tenant.hadoop.fs.s3a.*is accepted, looks right, and is then ignored by the walk. I assumed I had a typo for a while.Proposal: build the action's Hadoop config with
SparkUtil.hadoopConfCatalogOverrides(spark, catalogName)instead ofspark.sessionState().newHadoopConf(), and have that method derive the S3A settings from the catalog's ownclient.assume-role.arn, so the role only has to be declared once:fs.s3a.aws.credentials.providerbecomesAssumedRoleCredentialProviderfs.s3a.assumed.role.arnbecomes the catalog's arnfs.s3a.assumed.role.credentials.provider, so it is still what signs the AssumeRole call. Without that step S3A falls back to itsSimpleAWSCredentialsProviderdefault and IRSA or instance-profile clusters end up with no credentials for the STS call at all.Because that config is per catalog there's nothing to scope by bucket, unlike the session-wide workaround below.
spark.sql.catalog.<name>.hadoop.*is copied over the derived values afterwards, so anyone who wants different S3A settings for a catalog can still say so, and catalogs that declare no assume-role arn are untouched.BaseProcedurealready holds the catalog (tableCatalog()), so the name is available where the action is built.It is a behaviour change for catalogs that do declare a role: their listing starts going through that role rather than through the cluster identity. That's the point of the fix, and it matches what the rest of the procedure already does, but it is worth calling out.
I have this working against
spark/v4.1with tests, and can open a PR plus the v4.0 and v3.5 backports. If the preference is to leave the Hadoop path alone because the FileIO one is where things are heading, then at minimum the docs should say that the listing uses the session's Hadoop config and not the catalog's, because nothing in the config surface hints at it.For anyone else hitting the same 403, the workaround is to give S3A the same role, scoped to the buckets:
Per-bucket options key on the bucket name from the URI (
fs.s3a.bucket.<bucket>.), independent of the scheme, so this coverss3://locations too. The role's trust policy has to allow the cluster principal to assume it, which it already does if the catalog is working.Willingness to contribute