Skip to content

re-enable HTTP/2 back#259

Open
sepich wants to merge 1 commit intothanos-io:mainfrom
sepich:fix-http2
Open

re-enable HTTP/2 back#259
sepich wants to merge 1 commit intothanos-io:mainfrom
sepich:fix-http2

Conversation

@sepich
Copy link
Copy Markdown

@sepich sepich commented Apr 1, 2026

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

DefaultTransport() sets TLSClientConfig on the http.Transport, which silently disables HTTP/2 in Go's net/http. From the Go docs:

// ForceAttemptHTTP2 controls whether HTTP/2 is enabled when a non-zero
// Dial, DialTLS, or DialContext func or TLSClientConfig is provided.
// By default, use of any those fields conservatively disables HTTP/2.
// To use a custom dialer or TLS config and still attempt HTTP/2
// upgrades, set this to true.

This means all objstore backends (GCS, S3, Azure, COS) fall back to HTTP/1.1, even when the server supports HTTP/2.

Increasing max_idle_conns_per_host to 100 (the default since the GCS http_config was added) does not fully resolve the issue because HTTP/1.1 still requires one connection per concurrent request, and connection churn from server-side keepalive limits fills the TIME_WAIT table.

Verification

Initial issue: Bucket has 13k blocks, and thanos-compact fails with default --block-meta-fetch-concurrency=32:

ts=2026-04-01T13:19:18.858633235Z caller=fetcher.go:626 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.8768677s duration_ms=2876 cached=13229 returned=13229 partial=0
ts=2026-04-01T13:20:07.410384621Z caller=compact.go:546 level=error msg="retriable error" err="compaction: sync: filter metas: filter blocks marked for no downsample: get file: 01JNH9J9VHZD6JS4N99MFKEFG7/no-downsample-mark.json: Get \"https://storage.googleapis.com/thanos/01JNH9J9VHZD6JS4N99MFKEFG7%2Fno-downsample-mark.json\": dial tcp 192.178.183.207:443: connect: cannot assign requested address"

Each sync with 13,229 blocks does: list + 13,229 meta.json GETs + 13,229 deletion marks + 13,229 no-compact marks + 13,229 no-downsample marks ≈ 53,000 HTTP requests.

Default ports available: net.ipv4.ip_local_port_range=32768–60999

max_idle_conns_per_host is already 100 by default, and setting it in --objstore.config does not fix the issue. tcpdump still shows many established and immediately closed tcp connections.

But the change from the PR fixes the issue.

Signed-off-by: Alex R <alex@ryabov.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant