Skip to content

Cleanly close the read channel in the GoogleHadoopFSInputStream so that read to EOF does not happen on calling close() unnecessarily - #1741

Open
Animeshz wants to merge 3 commits into
GoogleCloudDataproc:masterfrom
Animeshz:animeshz/close-gcs-input-stream-cleanly
Open

Cleanly close the read channel in the GoogleHadoopFSInputStream so that read to EOF does not happen on calling close() unnecessarily#1741
Animeshz wants to merge 3 commits into
GoogleCloudDataproc:masterfrom
Animeshz:animeshz/close-gcs-input-stream-cleanly

Conversation

@Animeshz

@Animeshz Animeshz commented Jun 9, 2026

Copy link
Copy Markdown

Fixes #1729.

Calling close() on GoogleHadoopFSInputStream caused the whole stream to get drained, with AUTO/SEQUENTIAL fadvise (default) the stream is opened to the EOF, and hence whole file has to be read once opened even when calling close().

This PR cleanly closes the connection before Apache HTTP library tries to drain the bytes, so that only necessary bytes are read rest is dropped by closure of underlying HTTP response.

…hat read to EOF does not happen on calling close() unnecessarily

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates GoogleCloudStorageReadChannel to track and disconnect the active HTTP media response before closing the content channel, preventing Apache HTTP's ContentLengthInputStream from draining the remaining response body. It also introduces corresponding unit tests. The review feedback suggests removing the event bus exception logging from disconnectHttpResponse to avoid false positives, restoring the original try-catch block in openStream to properly suppress cleanup exceptions, and adding a @Nullable annotation to the overridden test method to match the base class signature.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@Animeshz Animeshz changed the title Cleanly close the reach channel in the GoogleHadoopFSInputStream so that read to EOF does not happen on calling close() unnecessarily Cleanly close the read channel in the GoogleHadoopFSInputStream so that read to EOF does not happen on calling close() unnecessarily Jun 9, 2026
@sonarqubecloud

Copy link
Copy Markdown

@Animeshz

Copy link
Copy Markdown
Author

Also need googleapis/google-http-java-client#1303 fixed before this is merged. The disconnect is broken at google http library level, it forces drain before disconnect.

@dheerajsngh

dheerajsngh commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Refer to comment.
The above fix is for master but the issue is only in branch-2.2.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GoogleHadoopFSInputStream does not close when calling close(), it reads the file fully instead.

2 participants