MultipartCopy Fails with the presence of Storage Class Header

[Luka-J9]

An extension of:
#81

When x-amz-storage-class is manually added to the headers of a multipartCopy request the resulting call fails with a

S3Exception: The specified header is not valid in this context

I believe the fix is the same as what was in the original PR, however it was not done for the CopyPart

Changing the code to follow the pattern of the linked PR resolves the issue - however it's worth noting that this does come at the expense of other legitimate fields like x-amz-request-payer

See AWS docs for valid headers

It seems that we would need to filter out the headers that are not in this list, rather than only include the SSE headers.

Happy to contribute - although I'm not exactly sure what would be the pattern to employ to only allow for a subset of headers

[pjfanning]

Any interest in supplying a PR? Pekko is powered by occasional volunteers.

[mdedetrich]

We do have an S3 test account provided by the Apache Foundation linked to Pekko which should help, albeit S3 integration tests only happen on main branch and not on PR's (for security reasons).

[Luka-J9]

Sure - I'll try to have a PR out later today to discuss against.

@mdedetrich I haven't spent too much time in the codebase - can you point me to where the integration tests are located?

[mdedetrich]

AWSS3IntegrationSpec contains the runner which runs against a real S3 account as well as instructions on how to hook it up to your S3 account and how to run the tests.

The actual tests are defined in S3IntegrationSpec which is where you would do the necessary changes.

[mdedetrich]

pekko-connectors/.github/workflows/nightly-builds.yaml

Lines 83 to 90 in ad06c34

	run: |-
	sbt \
	-Dpekko.connectors.s3.aws.credentials.provider=static \
	-Dpekko.connectors.s3.aws.credentials.access-key-id=${{ secrets.AWS_ACCESS_KEY }} \
	-Dpekko.connectors.s3.aws.credentials.secret-access-key=${{ secrets.AWS_SECRET_KEY }} \
	-Dpekko.stream.connectors.s3.scaladsl.AWSS3IntegrationSpec.enableListAllMyBucketsTests=false \
	-Dpekko.stream.connectors.s3.scaladsl.AWSS3IntegrationSpec.enableMFATests=false \
	+ "s3/Test/runMain org.scalatest.tools.Runner -o -s org.apache.pekko.stream.connectors.s3.scaladsl.AWSS3IntegrationSpec"

Is also helpful to show another way on how to run the tests using environment variables instead of having to modify s3/src/test/resources/application.conf

Note that in those tests, enableListAllMyBucketsTests and enableMFATests are disabled because we don't have the necessary permissions in the Apache provided AWS account to run those tests but that might not be the case for your own AWS account