EndpointResolverV2 broken with Ceph

[abdhx]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

After moving to EndpointResolverV2 from EndpointResolver, the s3 client GetObject function returns an error.

Expected Behavior

After migration, the s3 client function GetObject should keep working fine

Current Behavior

the s3 client function GetObject returns the error

api error NoSuchBucket: UnknownError

Reproduction Steps

1. Use an enterprise/local Ceph instead of AWS S3

2. Try with this example inspired by the official documentation

// v1: working fine with getObject
client := svc.NewFromConfig(cfg, func (o *svc.Options) {
    o.EndpointResolver = svc.EndpointResolverFromURL("https://custom.endpoint.api/", func (e *aws.Endpoint) {
        e.HostnameImmutable = true
    })
})
obj, err := s3Client.GetObject(context.Background(), &s3.GetObjectInput{
Bucket: aws.String("<replace_with_bucket_name>"),
Key:    aws.String("<replace_with_object_key>"),
})
if err != nil {
panic(err)
}
fmt.Printf("downloaded %d byes", obj.ContentLength)

// v2: api error NoSuchBucket: UnknownError
import (
    smithyendpoints "github.com/aws/smithy-go/endpoints"
)

type staticResolver struct {}

func (*staticResolver) ResolveEndpoint(ctx context.Context, params svc.EndpointParameters) (
        smithyendpoints.Endpoint, error,
    ) {
    // This value will be used as-is when making the request.
    u, err := url.Parse("https://custom.endpoint.api/")
    if err != nil {
        return smithyendpoints.Endpoint{}, err
    }
    return smithyendpoints.Endpoint{
        URI: *u,
    }, nil
}

client := svc.NewFromConfig(cfg, func (o *svc.Options) {
    o.EndpointResolverV2 = &staticResolver{}
})
obj, err := s3Client.GetObject(context.Background(), &s3.GetObjectInput{
Bucket: aws.String("<replace_with_bucket_name>"),
Key:    aws.String("<replace_with_object_key>"),
})
if err != nil {
panic(err)
}
fmt.Printf("downloaded %d byes", obj.ContentLength)

Possible Solution

No response

Additional Information/Context

Note 1:

My custom Ceph endpoint is working fine with

• aws cli
• EndpointResolver of "all" versions

But not with

• EndpointResolver2

Note 2:

In older versions of this SDK (before adding EndpointResolver2, SDK v2 version v1.18.1), an empty aws region was accepted by the package.
After the latest updates, an empty region gives the error Invalid region: region was not a valid DNS name

Note 3:

My endopoint is immutable

AWS Go SDK V2 Module Versions Used

module xxx

go 1.22.1

require (
	github.com/aws/aws-sdk-go-v2 v1.26.0
	github.com/aws/aws-sdk-go-v2/config v1.27.8
	github.com/aws/aws-sdk-go-v2/credentials v1.17.8
	github.com/aws/aws-sdk-go-v2/service/s3 v1.53.0
)

require (
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.1 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.4 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.4 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.4 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.4 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.6 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.6 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.4 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.20.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.28.5 // indirect
	github.com/aws/smithy-go v1.20.1 // indirect
)

Compiler and Version used

1.22.1

Operating System and version

first try with go on darwin/arm64, second try with docker golang image

[lucix-aws]

The AWS SDKs are designed for and are only guaranteed to work against the public AWS services, this isn't something we can guarantee will work as changes are made.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[abdhx]

@lucix-aws

I think this is not a special case for Ceph, but a problem with the UsePathStyle flag

I just did a little more digging. And it seems like UsePathStyle from s3.Options is not working when it is set to true. The log of the request shows that the bucket name is never added to the request path even with UsePathStyle = true.

[lucix-aws]

That is expected - see https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/endpoints/#v2-endpointresolverv2--baseendpoint. The endpoint returned from resolver V2 is final. S3 has some special behavior around the bucket name and for that reason we don't recommend you override its V2 resolver (that's called out in the link above). You most likely want to customize this in terms of BaseEndpoint instead.