HADOOP-19793. S3A: use long for file size in S3A content providers, data blocks#8225
Open
ajfabbri wants to merge 4 commits intoapache:trunkfrom
Open
HADOOP-19793. S3A: use long for file size in S3A content providers, data blocks#8225ajfabbri wants to merge 4 commits intoapache:trunkfrom
ajfabbri wants to merge 4 commits intoapache:trunkfrom
Conversation
|
💔 -1 overall
This message was automatically generated. |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
bf2c793 to
e179121
Compare
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
Majority of CI failures due to HADOOP-19790 |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
e179121 to
d0403b1
Compare
|
💔 -1 overall
This message was automatically generated. |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
d0403b1 to
1e1c09c
Compare
ajfabbri
changed the title
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
changed the title
steveloughran
approved these changes
Feb 19, 2026
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
+1
all looks good, and I reviewed that SingleFilePendingCommit file size too.
Regarding test failures
#2 may mean you aren't set up to create a session for the target user/account. I can help there
#3 things have been playing up with the MR cluster tests since the move to JUnit 5...getting anything working was a challenge enough.
Contributor
|
fix the spotbugs by addressing overrideable methods in verify, or make SinglePendingCommit final. |
|
💔 -1 overall
This message was automatically generated. |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
587e5de to
eed4c14
Compare
Contributor
|
Experimental Warnings Code Warning TEST Unknown bug pattern MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT in o.a.h.fs.s3a.commit.files.PendingSet.readObject(ObjectInputStream) Bug type MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT (click for details) In class o.a.h.fs.s3a.commit.files.PendingSet In method o.a.h.fs.s3a.commit.files.PendingSet.readObject(ObjectInputStream) Called method o.a.h.fs.s3a.commit.files.PendingSet.validate() At PendingSet.java:[line 166] TEST Unknown bug pattern MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT in o.a.h.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) Bug type MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT (click for details) In class o.a.h.fs.s3a.commit.files.SinglePendingCommit In method o.a.h.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) Called method o.a.h.fs.s3a.commit.files.SinglePendingCommit.validate() At SinglePendingCommit.java:[line 201]
Despite changing validate() to final, still get this warning: Unknown bug pattern MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT in org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) At SinglePendingCommit.java:org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) At SinglePendingCommit.java:[line 201] try making the whole class final then.
Contributor
Author
I already fixed it but CI has been stuck for over 24 hours! https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8225/8/ |
|
💔 -1 overall
This message was automatically generated. |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
eed4c14 to
f94a921
Compare
Contributor
Author
|
Force-push: rebase on latest trunk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
From HADOOP-19793:
How was this patch tested?
Tested that large scale test with localstack S3.
Ran all integration tests locally against S3 in us-west-2. All passed except:
ITestS3ACannedACLs>AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:197->AbstractFSContractTestBase.mkdirs:355 » ... S3Exception:The bucket does not allow ACLsITestS3ATemporaryCredentials.testSessionTokenPropagation:202getFileStatus on s3a://fabbri-s3a/job-00-fork-0004/test/testSTS/c20306d0-3eca-4264-b210-3d7beb8c80c7: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden (Service: S3, Status Code: 403, Request ID: 6GV4SMMX9EXVDB0J, Extended Request ID: nB4bQ+CHn+Y2BiR4lLSgLtHyWVd05eerIXrsNDhV9Z0o9I/KZ+N8VD3khlSsiJvh0HkuCjDMtB4=):nullThe first failure seems OK given I haven't enabled bucket ACLs.
I'm not sure what the problem is with 2 & 3.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
Nopewhere is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html