re-enable HTTP/2 back#259
Open
sepich wants to merge 1 commit intothanos-io:mainfrom
Open
Conversation
Signed-off-by: Alex R <alex@ryabov.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
DefaultTransport() sets TLSClientConfig on the http.Transport, which silently disables HTTP/2 in Go's net/http. From the Go docs:
This means all objstore backends (GCS, S3, Azure, COS) fall back to HTTP/1.1, even when the server supports HTTP/2.
Increasing
max_idle_conns_per_hostto 100 (the default since the GCShttp_configwas added) does not fully resolve the issue because HTTP/1.1 still requires one connection per concurrent request, and connection churn from server-side keepalive limits fills the TIME_WAIT table.Verification
Initial issue: Bucket has 13k blocks, and
thanos-compactfails with default--block-meta-fetch-concurrency=32:Each sync with 13,229 blocks does: list + 13,229 meta.json GETs + 13,229 deletion marks + 13,229 no-compact marks + 13,229 no-downsample marks ≈ 53,000 HTTP requests.
Default ports available:
net.ipv4.ip_local_port_range=32768–60999max_idle_conns_per_hostis already 100 by default, and setting it in--objstore.configdoes not fix the issue.tcpdumpstill shows many established and immediately closed tcp connections.But the change from the PR fixes the issue.