[BUG] S3 Control ListTagsForResource returns 403 AccessDenied — breaks Terraform AWS provider v6.x S3 bucket reconcile (regression of #341)

[berendt]

Service

S3 (S3 Control API) — bucket tagging.

AWS API Action

ListTagsForResource (S3 Control API — the GET /v20180820/tags/<arn> endpoint on the account-prefixed host <account-id>.s3-control...).

This is the operation #341 added so the Terraform AWS provider v6.x can read S3 bucket tags. It is distinct from the standard S3 GetBucketTagging.

Expected behavior

For an existing bucket, S3 Control ListTagsForResource returns the bucket's tag set with HTTP 200 — an empty <Tags/> set for an untagged bucket — mirroring what the standard S3 GetBucketTagging already returns on the same bucket. This is what #341 set out to provide.

Actual behavior

floci responds with HTTP 403 and an AccessDenied body for every ListTagsForResource call, instead of returning the tags:

<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Error><Code>AccessDenied</Code><Message>Access Denied</Message><AccountId>000000000000</AccountId></Error><RequestId>…</RequestId><HostId>…</HostId></ErrorResponse>

(The <ErrorResponse> wrapper is correct since #557 — the problem is the AccessDenied verdict, not the envelope.)

The standard S3 GetBucketTagging on the same bucket works fine:

$ aws --endpoint-url http://floci:4566 s3api get-bucket-tagging --bucket repro
{ "TagSet": [] }

So floci serves the S3 (data-plane) tagging API but the S3 Control variant denies. Because the Terraform AWS provider v6.x reads aws_s3_bucket tags exclusively through S3 Control ListTagsForResource (AWS ABAC support), every read/refresh of a bucket fails — the bucket is created successfully but never converges.

Reproduction

Note: the AWS CLI / botocore ignores the S3 Control endpoint override (AWS_ENDPOINT_URL_S3CONTROL / --endpoint-url) and resolves S3 Control to the real *.s3-control.<region>.amazonaws.com host, so a naive aws s3control … command does not hit floci. The AWS SDK for Go v2 (which the Terraform AWS provider uses) honours BaseEndpoint, so the repro uses it.

// go.mod: github.com/aws/aws-sdk-go-v2/service/s3control
cfg, _ := config.LoadDefaultConfig(ctx,
    config.WithRegion("us-east-1"),
    config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("test", "test", "")),
)
// First create a bucket via the S3 client (BaseEndpoint http://floci:4566, UsePathStyle true).
// Then:
sc := s3control.NewFromConfig(cfg, func(o *s3control.Options) {
    o.BaseEndpoint = aws.String("http://floci:4566")
})
_, err := sc.ListTagsForResource(ctx, &s3control.ListTagsForResourceInput{
    AccountId:   aws.String("000000000000"),
    ResourceArn: aws.String("arn:aws:s3:::repro"),
})
// err => operation error S3 Control: ListTagsForResource, https response error
//        StatusCode: 403, api error AccessDenied: Access Denied

End-to-end, this surfaces through the Crossplane Upbound provider-aws-s3 v2.6.1 (which wraps Terraform AWS provider v6): an s3.aws.m.upbound.io Bucket is created in floci, but its Observe fails forever with:

observe failed: failed to observe the resource: [{0 listing tags for S3 (Simple Storage) Bucket
(repro-…): operation error S3 Control: ListTagsForResource, https response error StatusCode: 403,
RequestID: …, HostID: …, api error AccessDenied: Access Denied []}]

so the managed resource never reaches Ready even though the bucket exists.

Environment

• Floci version / image tag: floci/floci:latest (build 2026-06-24); also reproduced on 1.5.25 and 1.5.27.
• SDK: AWS SDK for Go v2 via Terraform AWS provider v6.x (Crossplane Upbound provider-aws-s3:v2.6.1).
• How you're running Floci: Docker image in Kubernetes (kind), serving http://floci:4566, account 000000000000, region us-east-1.

Related

• [FEAT] S3: Implement S3 Control API ListTagsForResource (used by Terraform AWS provider v6.x) #341 — [FEAT] S3: Implement S3 Control API ListTagsForResource (used by Terraform AWS provider v6.x) (closed). This report is that the implemented handler returns AccessDenied rather than the tag set, so the feature does not yet unblock the provider-v6 flow it was added for.
• [BUG] S3 Control error responses are missing the '<ErrorResponse xmlns="…">' wrapper — Go SDK v2 collapses all errors to 'UnknownError: UnknownError' #557 — [BUG] S3 Control error responses are missing the '<ErrorResponse …>' wrapper (closed). The wrapper is now present; this is a separate, behavioural issue (the verdict, not the envelope).

[hectorvent]

Thanks for the detailed repro, it made this quick to pin down.

The ListTagsForResource handler from #341 works. Running your exact Go snippet against a live Floci with the final URL logged:

>>> GET http://000000000000.localhost:4566/v20180820/tags/arn:aws:s3:::repro
ListTagsForResource OK: 0 tags

With BaseEndpoint set it returns 200 and an empty tag set. The tell in your 403 is <AccountId> and <HostId>: that is real AWS S3 Control's error shape, which Floci never emits. So the call is reaching real AWS, not Floci.

Here is why. S3 Control gives ListTagsForResource a hostPrefix of {AccountId}., so the SDK rewrites the host to 000000000000.floci:4566. It works on my box because *.localhost resolves to loopback. In k8s, 000000000000.floci is not resolvable by CoreDNS (a Service name has no wildcard subdomain), so the request never lands on Floci.

The fix is on the DNS side: the account-prefixed host has to resolve to Floci. In k8s a CoreDNS rewrite does it:

rewrite name regex (.*)\.floci\.<ns>\.svc\.cluster\.local floci.<ns>.svc.cluster.local

Closing this as not a code bug. I will add a docs note that account-prefixed services like S3 Control need *.<floci-host> to resolve to Floci, so the next person does not hit it. Happy to reopen if a request that actually reaches Floci returns the wrong thing. Thanks again for the careful write up.

[hectorvent]

This will be resolved by #1595