Skip to content

Conversation

@wendigo
Copy link
Contributor

@wendigo wendigo commented Nov 19, 2025

HttpClient's executeAsync always buffers the entire response before handing it over to the ResponseHandler. This is inefficent as the decoding could be streaming and that doesn't require large amount of memory while reading results.

Description

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Nov 19, 2025
@wendigo wendigo force-pushed the serafin/do-not-buffer branch from bb8dd7d to f8cf9c3 Compare November 19, 2025 15:58
@starburstdata-automation
Copy link

starburstdata-automation commented Nov 19, 2025

Started benchmark workflow for this PR with test type = iceberg/sf1000_parquet_part.

Building Trino finished with status: success
Benchmark finished with status: failure
Comparing results to the static baseline values, follow above workflow link for more details/logs.
Status message: NO Regression found.
Benchmark Comparison to the closest run from Master: Report

@starburstdata-automation
Copy link

starburstdata-automation commented Nov 19, 2025

Started benchmark workflow for this PR with test type = iceberg/sf1000_parquet_unpart.

Building Trino finished with status: success
Benchmark finished with status: success
Comparing results to the static baseline values, follow above workflow link for more details/logs.
Status message: NO Regression found.
Benchmark Comparison to the closest run from Master: Report

@wendigo wendigo force-pushed the serafin/do-not-buffer branch 2 times, most recently from c3d8bdb to 3208109 Compare November 19, 2025 17:50
wendigo and others added 2 commits November 19, 2025 19:25
HttpClient's executeAsync always buffers the entire response before handing it over to the
ResponseHandler. This is inefficent as the decoding could be streaming and that doesn't
require large amount of memory while reading results.
This helps to detect regressions around handling of large requests
by HttpPageBufferClient
E.g. failures like Buffering capacity 67108864 exceeded after a Jetty update

(cherry picked from commit f4d0198)
@wendigo wendigo force-pushed the serafin/do-not-buffer branch from 3208109 to dce8aaa Compare November 19, 2025 18:25
@findepi findepi requested review from electrum and losipiuk November 23, 2025 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants