cacheDriver fails with IRSA: STS AssumeRoleWithWebIdentity returns 404 despite valid token

[tkircsi]

zot version

2.1.2

Describe the bug

Problem

I’m running Zot in EKS with IRSA configured via a properly annotated Kubernetes service account. When using the S3 storageDriver, everything works perfectly via IRSA. However, when enabling the dynamodb cacheDriver, Zot fails to authenticate with AWS STS and crashes with the following error:

operation error DynamoDB: CreateTable, get identity: get credentials:
failed to refresh cached credentials, failed to retrieve credentials,
operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 404,
RequestID: ..., api error UnknownError: UnknownError

This only occurs when using the cacheDriver. With the same config and IRSA role, the S3 driver works without issues.

Verified

• EKS IRSA is correctly set up
• ServiceAccount has eks.amazonaws.com/role-arn and eks.amazonaws.com/audience: sts.amazonaws.com annotations
• Token audience is sts.amazonaws.com
• IAM trust policy allows the sub and aud from the token
• aws sts get-caller-identity works from a debug pod using the same zot-sa ServiceAccount
• The projected token decodes correctly:

"aud": ["sts.amazonaws.com"],
"sub": "system:serviceaccount:phoenix:zot-sa",
"iss": "https://oidc.eks.us-east-2.amazonaws.com/id/..."

To reproduce

1. Create an IAM role with the appropriate trust policy
2. Attach a policy that allows S3 and DynamoDB access (including CreateTable on DynamoDB).
3. Create a Kubernetes ServiceAccount with the appropriate annotations.
4. Deploy Zot with the following configuration:

"storage": {
  "storageDriver": {
    "name": "s3",
    "region": "eu-central-1",
    "bucket": "<bucket-name>",
    "secure": true
  },
  "cacheDriver": {
    "name": "dynamodb",
    "region": "eu-central-1",
    "endpoint": "https://dynamodb.eu-central-1.amazonaws.com",
    "cachetablename": "<cacheTableName>"
  }
}

5. Observe failure at startup. Logs will show:

operation error STS: AssumeRoleWithWebIdentity,
https response error StatusCode: 404, api error UnknownError: UnknownError

6. Remove the cacheDriver section → Zot works perfectly with S3 alone.

Expected behavior

Zot should initialize the DynamoDB client using the same AWS SDK credential chain that works for S3, including support for IRSA via web identity token.

Screenshots

No response

Additional context

No response

[rchincha]

@tkircsi

https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

https://stackoverflow.com/questions/66405794/not-authorized-to-perform-stsassumerolewithwebidentity-403

^ some research about this.

Also, if you are able to root-cause this before we do, pls do give us a PR.

[tkircsi]

Thank you for your reply! I've tested most of the stackoverflow suggestions, but they didn't work, I'll check the ones I haven't tested yet. I have tested the serviceaccount token in the same cluster with an "irsa-debug" pod and the token is fine (OIDC provider, sub, aud, ...) I think the problem is with the AWS SDK's default credential chain handling. Unfortunately, I don't have time now to test and PR, but I would modify here something like this

pkg/storage/cache/dynamodb.go

	cfg, err := config.LoadDefaultConfig(context.Background(),
		config.WithRegion(properParameters.Region),
		config.WithClientLogMode(aws.LogSigning|aws.LogRetries|aws.LogRequest|aws.LogResponse))
	if err != nil {
		log.Error().Err(err).Msg("failed to load AWS SDK config for dynamodb")

		return nil, err
	}

	// Create DynamoDB client with custom endpoint
	client := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
		if properParameters.Endpoint != "" {
			o.EndpointResolver = dynamodb.EndpointResolverFromURL(properParameters.Endpoint)
		}
	})

One thing is that EndpointResolver is depricated. I may test it later and if it works I will make a PR.

[aqemia-aymeric-alixe]

I've got the same issue. I have little knowledge but I'd be happy to help