HADOOP-19860. S3A: support custom per-request-type S3 and STS headers by EnricoMi · Pull Request #8417 · apache/hadoop

EnricoMi · 2026-04-10T13:30:50Z

Description of PR

Similar to #7379 (HADOOP-18562), users might want to send custom headers for specific requests only.

With

<property>
    <name>fs.s3a.client.s3.custom.headers.request.DeleteObjectRequest</name>
    <description>List of custom headers to be set on the service client for this command.</description>
    <value>x-amz-delete-contents=true</value>
</property>

Only DeleteObjectRequest requests have a custom header x-amz-delete-contents: true.

Note: DeleteObjectRequest is only an example here. Any request class name could be used here.

See:

How was this patch tested?

There is a unit test verifying the headers are injected into the AWS request objects.

Integration test ITestS3AContractDelete has been run against a company S3 point and headers show up only in the expected (delete) requests.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

No AI tooling has been used.

hadoop-yetus · 2026-04-10T14:45:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 55s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	0m 42s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	0m 39s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	1m 6s		trunk passed
+1 💚	shadedclient	15m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 23s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 24s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	0m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 32 new + 3 unchanged - 0 fixed = 35 total (was 3)
-1 ❌	mvnsite	0m 23s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	javadoc	0m 18s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.10+7-Ubuntu-124.04.txt	hadoop-aws in the patch failed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04.
-1 ❌	javadoc	0m 20s	/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt	hadoop-aws in the patch failed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1.
+1 💚	spotbugs	0m 48s		the patch passed
+1 💚	shadedclient	15m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-aws in the patch passed.
-1 ❌	asflicense	0m 24s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		69m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8417/1/artifact/out/Dockerfile
GITHUB PR	#8417
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 2b7bada2f5bd 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6934987`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8417/1/testReport/
Max. process+thread count	610 (vs. ulimit of 10000)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8417/1/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-04-10T17:39:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 26s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	1m 2s		trunk passed
+1 💚	shadedclient	15m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	0m 23s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 17s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 32 new + 3 unchanged - 0 fixed = 35 total (was 3)
+1 💚	mvnsite	0m 27s		the patch passed
-1 ❌	javadoc	0m 20s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-21.0.10+7-Ubuntu-124.04.txt	hadoop-tools_hadoop-aws-jdkUbuntu-21.0.10+7-Ubuntu-124.04 with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 generated 3 new + 929 unchanged - 0 fixed = 932 total (was 929)
-1 ❌	javadoc	0m 20s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt	hadoop-tools_hadoop-aws-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1 with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 generated 3 new + 851 unchanged - 0 fixed = 854 total (was 851)
+1 💚	spotbugs	0m 50s		the patch passed
+1 💚	shadedclient	15m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 32s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		71m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8417/2/artifact/out/Dockerfile
GITHUB PR	#8417
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 5296065b9338 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b5b309f`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8417/2/testReport/
Max. process+thread count	611 (vs. ulimit of 10000)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8417/2/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2026-04-14T10:19:14Z

Is this a real need? it's adding more complexity to the system

EnricoMi · 2026-04-22T12:55:44Z

It is a generalization of #7379 and required for custom headers that are not allowed by all requests.

github-actions Bot added trunk TOOLS AWS labels Apr 10, 2026

EnricoMi force-pushed the custom-request-headers branch from 29e6332 to 6934987 Compare April 10, 2026 13:34

EnricoMi changed the title ~~HADOOP-19860. S3A: support custom per-request type S3 and STS headers~~ HADOOP-19860. S3A: support custom per-request-type S3 and STS headers Apr 10, 2026

HADOOP-19860. S3A: support custom per-request-type S3 and STS headers

b5b309f

EnricoMi force-pushed the custom-request-headers branch from 6934987 to b5b309f Compare April 10, 2026 16:27

EnricoMi mentioned this pull request Apr 14, 2026

HADOOP-19801. Allow to skip recursive file deletion for non-empty dir… #8306

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19860. S3A: support custom per-request-type S3 and STS headers#8417

HADOOP-19860. S3A: support custom per-request-type S3 and STS headers#8417
EnricoMi wants to merge 1 commit intoapache:trunkfrom
EnricoMi:custom-request-headers

EnricoMi commented Apr 10, 2026

Uh oh!

hadoop-yetus commented Apr 10, 2026

Uh oh!

hadoop-yetus commented Apr 10, 2026

Uh oh!

steveloughran commented Apr 14, 2026

Uh oh!

EnricoMi commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

EnricoMi commented Apr 10, 2026

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Apr 10, 2026

Uh oh!

hadoop-yetus commented Apr 10, 2026

Uh oh!

steveloughran commented Apr 14, 2026

Uh oh!

EnricoMi commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants