HDDS-16151. Support x-amz-expiration header in HEAD object response for lifecycle expiration rules - #11157
Open
henry3260 wants to merge 3 commits into
Open
HDDS-16151. Support x-amz-expiration header in HEAD object response for lifecycle expiration rules#11157henry3260 wants to merge 3 commits into
henry3260 wants to merge 3 commits into
Conversation
S3 returns x-amz-expiration on HeadObject when a lifecycle expiration rule covers the object, naming the date it is scheduled for deletion and the rule that schedules it. S3G never emitted the header. Report the earliest expiry among the enabled expiration rules that cover the key. The header is advisory: any failure to read the bucket's lifecycle configuration only leaves it out, and the HEAD still succeeds.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds S3-compatible lifecycle expiration reporting for HeadObject by emitting the x-amz-expiration response header when an enabled expiration rule covers the requested key, matching AWS’s expected header format and “earliest matching expiry” behavior.
Changes:
- Add
x-amz-expirationconstant and emit the header onHEADwhen lifecycle expiration rules cover the key. - Implement rule/key matching and expiry-date calculation (Days rounded up to next midnight UTC; Date reported as-is).
- Add unit and smoketest coverage to validate header presence/absence and selection of earliest applicable rule.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectHead.java | Adds unit tests covering Days/Date expirations, tag filters, earliest-rule selection, and non-matching/disabled/no-config cases. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Consts.java | Introduces EXPIRATION_HEADER (x-amz-expiration) constant for consistent usage. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java | Computes and injects x-amz-expiration on successful HEAD responses based on lifecycle configuration and key metadata. |
| hadoop-ozone/dist/src/main/smoketest/s3/bucketlifecycle.robot | Adds smoketests verifying the AWS CLI head-object output includes/omits Expiration depending on rule match. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneLifecycleConfiguration.java | Adds rule/filter matching helpers and isEnabled() used by S3G to determine applicability of lifecycle rules. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
utafrali
reviewed
Aug 29, 2026
utafrali
left a comment
utafrali
left a comment
There was a problem hiding this comment.
The core feature is correct and well unit-tested, but there are two real issues: an OM RPC is fired on every HEAD request regardless of whether a lifecycle configuration exists (a per-request cost that worsens under load), and the expiration header uses the wrong modification time when partNumber is non-zero. The robot tests also lack value assertions on the header format.
…sert header format Rule ids are supplied by the client and only validated for length, so a quote or backslash is escaped before it goes into the quoted-string header parameter. An id carrying a control character cannot appear in a header value at all, so the header is dropped instead of being emitted malformed. Note in OzoneLCRule#matches that OM rejects a rule setting both prefix and filter, and one setting neither, so the order there selects whichever is present rather than expressing a precedence. Assert the header format in the robot case instead of only its presence, so a compose run catches format regressions the unit tests cannot see.
Contributor
|
@henry3260 thanks for the patch! |
Most buckets carry no lifecycle configuration, and OM reports that by throwing, so the expiration header added to HEAD paid for an OM call, a bucket read lock and an exception on every request against such a bucket. Remember per bucket, for a configurable TTL, that OM reported no configuration, and skip the lookup while the entry lives. Only the absence is cached: the configuration itself is read under a bucket-level ACL check that HEAD does not repeat, so serving stored rules from the gateway would hand them to callers OM never authorized. ozone.s3g.lifecycle.missing-configuration.cache.ttl (30s default) bounds the staleness; zero disables the cache.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
On
HeadObject, S3 returns anx-amz-expirationheader when a lifecycle expiration rule coversthe object, naming the date it is scheduled for deletion and the rule that schedules it:
Ozone supports lifecycle expiration rules and already deletes the keys they cover, but S3 Gateway
never emitted this header, so clients cannot read a key's expiration metadata.
S3G now reads the bucket's lifecycle configuration while serving
HEADand reports the earliestexpiry among the enabled expiration rules that cover the key. A
Daysrule reports the key'smodification time plus the configured days, rounded up to the next midnight UTC, as S3 does; a
Daterule is reported as is.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16151
How was this patch tested?
TestObjectHeadcases:Daysrule,Daterule, tag-filter rule, earliest of severalmatching rules, non-matching prefix,
Disabledrule,AbortIncompleteMultipartUpload-only rule,and no lifecycle configuration. The
Datecase pins the exact header format against a fixed date.ozone-s3gateway(754 tests) andozone-client(118 tests) pass; checkstyle and RAT are clean.bucketlifecycle.robotcases (header present on a matching key, absent on anon-matching one). These were not run locally -- they need a compose cluster -- so they rely on CI.