fix(io): bound the TLS handshake performed when tunneling HTTPS through a proxy by jszwedko · Pull Request #2309 · DataDog/saluki

jszwedko · 2026-08-11T21:36:17Z

Human Summary

Use the upstream hyper-http-proxy crate, which now supports setting a timeout on TLS handshakes. This started out as a fork while the change was upstream, but that PR has since merged and been released as 1.2.0, so we no longer need it.

AI Summary

When a proxy is configured, HttpClient's TLS handshake timeout only bounds direct connections — hyper_http_proxy::ProxyConnector performs a separate TLS handshake over the CONNECT tunnel to reach HTTPS destinations, and that handshake was previously unbounded, so a stalled or unresponsive proxy could hang a request indefinitely. This adds a set_tls_handshake_timeout API to hyper-http-proxy (contributed upstream via metalbear-co/hyper-http-proxy#9, merged and released as 1.2.0) and wires it up using the same handshake timeout already configured for direct connections.

Test plan

Added tls_handshake_timeout_fires_against_a_stalled_proxy_tunnel, which emulates a proxy that completes the CONNECT tunnel but never speaks TLS, and asserts the client's configured handshake timeout ends the request.

…S through a proxy The TLS handshake timeout configured on HttpClient only covered direct connections; when a proxy is configured, hyper-http-proxy::ProxyConnector performs its own separate TLS handshake over the CONNECT tunnel that wasn't bounded by any timeout, so a stalled proxy could hang a request indefinitely. Switches to a fork of hyper-http-proxy that adds a set_tls_handshake_timeout API and wires it up with the same timeout used for direct connections.

jszwedko · 2026-08-11T21:39:05Z

    }

+    #[tokio::test]
+    async fn tls_handshake_timeout_fires_against_a_stalled_proxy_tunnel() {


I could go either way on this test. It duplicates a test in the upstream crate, but also it checks that we wired the timeout through correctly. I'm inclined to leave it.

Not a bad idea to verify the upstream crate behavior doesn't change.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8467084388

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pr-commenter · 2026-08-11T21:44:01Z

Binary Size Analysis (Agent Data Plane)

Baseline: 4d40f35 · Comparison: 731106d · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.51 MiB (baseline) vs 41.50 MiB (comparison)
Size Change: -552 B (-0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`anon.8c1b48a1363a134b50a48e1aacabbfc0.329.llvm.2532050142918896805`	+17.44 KiB	1
`anon.e12e1bc94a2f8056bdfda22d16c72d6f.329.llvm.2114813585693862172`	-17.35 KiB	1
`anon.44bf66f5bcf6e2ae2aa37756deaffd68.571.llvm.16278901431488619082`	+12.28 KiB	1
`anon.f8eccca6bed15bec3b48551a81cb44a3.571.llvm.6077875107705837680`	-12.20 KiB	1
`anon.8c1b48a1363a134b50a48e1aacabbfc0.243.llvm.2532050142918896805`	+9.71 KiB	1
`anon.e12e1bc94a2f8056bdfda22d16c72d6f.243.llvm.2114813585693862172`	-9.63 KiB	1
`anon.8c1b48a1363a134b50a48e1aacabbfc0.323.llvm.2532050142918896805`	+9.15 KiB	1
`anon.e12e1bc94a2f8056bdfda22d16c72d6f.323.llvm.2114813585693862172`	-9.06 KiB	1
`anon.75f0c43c462b613efe93f4b4732a5bd9.0.llvm.10650251582497753077`	-8.40 KiB	1
`anon.31cdbec94e8e8bc4e8fa8b6ed8cab3b1.0.llvm.11273601663889839289`	+8.39 KiB	1
`anon.f8eccca6bed15bec3b48551a81cb44a3.295.llvm.6077875107705837680`	-7.12 KiB	1
`anon.44bf66f5bcf6e2ae2aa37756deaffd68.295.llvm.16278901431488619082`	+7.12 KiB	1
`anon.75f0c43c462b613efe93f4b4732a5bd9.296.llvm.10650251582497753077`	-6.35 KiB	1
`anon.31cdbec94e8e8bc4e8fa8b6ed8cab3b1.296.llvm.11273601663889839289`	+6.17 KiB	1
`futures_util`	+5.42 KiB	117
`saluki_io::net::client`	+5.25 KiB	85
`core`	-4.68 KiB	12866
`alloc`	+4.42 KiB	2124
`anon.e12e1bc94a2f8056bdfda22d16c72d6f.321.llvm.2114813585693862172`	-4.03 KiB	1
`anon.8c1b48a1363a134b50a48e1aacabbfc0.321.llvm.2532050142918896805`	+4.03 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +60.1Ki  [NEW] +60.0Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::hc7d0835efee94f1d
  [NEW] +46.4Ki  [NEW] +46.3Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hd66f3a656cb7efb8
  [NEW] +38.2Ki  [NEW] +38.0Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hfa9b0182d6516aa7
  [NEW] +37.9Ki  [NEW] +37.8Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h23b0e33a348d8c83
  [NEW] +37.4Ki  [NEW] +37.1Ki    _<saluki_components::common::datadog::config::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::config::ForwarderConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h332cdc3b753247b2
  [NEW] +37.0Ki  [NEW] +36.8Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h7af2dea4b7bc0109
  [NEW] +36.5Ki  [NEW] +36.4Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::hf364db3505dbe4a3
  [NEW] +32.7Ki  [NEW] +32.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::he7a113e40f6ebfc6
  [NEW] +31.1Ki  [NEW] +30.9Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::hd594d6a8deda2114
  [NEW] +30.6Ki  [NEW] +30.5Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::hf7f9cebc12ef577f
  -0.0%    -883  -0.0%    -367    [45580 Others]
  [DEL] -30.2Ki  [DEL] -30.0Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::h4ffdcc7f8fe7225c
  [DEL] -31.1Ki  [DEL] -30.9Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::hf5061c4b35c9609f
  [DEL] -32.7Ki  [DEL] -32.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::hb00f6194ea6b5f50
  [DEL] -36.5Ki  [DEL] -36.4Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h0356f8cbb6e0153d
  [DEL] -36.9Ki  [DEL] -36.7Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h188466136301edcb
  [DEL] -37.3Ki  [DEL] -37.0Ki    _<saluki_components::common::datadog::config::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::config::ForwarderConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h550091c9354a1a2b
  [DEL] -37.9Ki  [DEL] -37.8Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hfc9042dc5d50b63e
  [DEL] -38.3Ki  [DEL] -38.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h256fe1c552d15d03
  [DEL] -46.4Ki  [DEL] -46.3Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::he0c34f1d64f1cb15
  [DEL] -60.3Ki  [DEL] -60.1Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h96edd6670bc21503
  -0.0%    -552  -0.0%     -36    TOTAL

datadog-prod-us1-3

With tls_handshake_timeout: 0 and an HTTPS proxy, the new wiring converts the documented disabled value into an active zero-length proxy deadline, causing new tunneled connections to fail immediately. Preserve zero as None when configuring the proxy connector.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 8467084 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

pr-commenter · 2026-08-11T21:58:53Z

Regression Detector (Agent Data Plane)

Run ID: 9c7af414-d991-4c30-a6be-a4e33122c266
Baseline: 4d40f354 · Comparison: 731106d5 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.30	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.10	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.15	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.33	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.39	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 228 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 50.8 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 90.6 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 387 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 32.1 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

…nels A zero tls_handshake_timeout means "disabled" for direct connections, but it was passed through to the proxy connector as Some(Duration::ZERO), which hyper-http-proxy treats as an active zero-length deadline, failing every tunneled HTTPS connection through a proxy immediately. Map zero to None before handing it to the proxy connector, and add a test.

Points Cargo.lock at the fork commit that addresses upstream review feedback on the TLS handshake timeout PR (metalbear-co/hyper-http-proxy#9).

metalbear-co/hyper-http-proxy#9 merged and released as 1.2.0, so the fork patch is no longer needed.

The git patch this allowance covered was already replaced on this branch, first by a different fork and now by the released upstream crate, so no dependency resolves to that source anymore.

…gh a proxy (#2309) ## Human Summary Use the upstream `hyper-http-proxy` crate, which now [supports setting a timeout on TLS handshakes](metalbear-co/hyper-http-proxy#9). This started out as a fork while the change was upstream, but that PR has since merged and been released as 1.2.0, so we no longer need it. ## AI Summary When a proxy is configured, `HttpClient`'s TLS handshake timeout only bounds direct connections — `hyper_http_proxy::ProxyConnector` performs a separate TLS handshake over the CONNECT tunnel to reach HTTPS destinations, and that handshake was previously unbounded, so a stalled or unresponsive proxy could hang a request indefinitely. This adds a `set_tls_handshake_timeout` API to `hyper-http-proxy` (contributed upstream via [metalbear-co/hyper-http-proxy#9](metalbear-co/hyper-http-proxy#9), merged and released as 1.2.0) and wires it up using the same handshake timeout already configured for direct connections. ## Test plan - [x] Added `tls_handshake_timeout_fires_against_a_stalled_proxy_tunnel`, which emulates a proxy that completes the CONNECT tunnel but never speaks TLS, and asserts the client's configured handshake timeout ends the request. Co-authored-by: jesse.szwedko <jesse.szwedko@datadoghq.com> f76d1fb

webern · 2026-08-13T17:41:56Z

    }

+    #[tokio::test]
+    async fn tls_handshake_timeout_fires_against_a_stalled_proxy_tunnel() {


Not a bad idea to verify the upstream crate behavior doesn't change.

webern · 2026-08-13T17:42:26Z

-[patch.crates-io]
-# Forked version of `hyper-http-proxy` that removes an unused dependency on `rustls-native-certs`, which transitively depends
-# on a version of `rustls-pemfile` that is no longer maintained and triggers a hit when running `cargo deny`.
-hyper-http-proxy = { git = "https://github.com/tobz/hyper-http-proxy.git", branch = "main" }
-


dd-octo-sts Bot added the area/io General I/O and networking. label Aug 11, 2026

jszwedko commented Aug 11, 2026

View reviewed changes

jszwedko changed the title ~~fix(saluki-io): bound the TLS handshake performed when tunneling HTTPS through a proxy~~ fix(io): bound the TLS handshake performed when tunneling HTTPS through a proxy Aug 11, 2026

jszwedko marked this pull request as ready for review August 11, 2026 21:41

jszwedko requested a review from a team as a code owner August 11, 2026 21:41

chatgpt-codex-connector Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread lib/saluki-io/src/net/client/http/client.rs Outdated

datadog-prod-us1-3 Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread lib/saluki-io/src/net/client/http/client.rs Outdated

jszwedko added 3 commits August 11, 2026 15:42

chore(saluki-io): pin hyper-http-proxy fork to latest review fixes

f7fd65c

Points Cargo.lock at the fork commit that addresses upstream review feedback on the TLS handshake timeout PR (metalbear-co/hyper-http-proxy#9).

chore(saluki-io): switch hyper-http-proxy back to upstream crate

7474766

metalbear-co/hyper-http-proxy#9 merged and released as 1.2.0, so the fork patch is no longer needed.

tobz approved these changes Aug 12, 2026

View reviewed changes

chore(deps): remove unused tobz/hyper-http-proxy deny.toml allowance

731106d

The git patch this allowance covered was already replaced on this branch, first by a different fork and now by the released upstream crate, so no dependency resolves to that source anymore.

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: waiting mergequeue-status: queued labels Aug 12, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit f76d1fb into main Aug 12, 2026
93 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Aug 12, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the jszwedko/hyper-http-proxy-tls-timeout branch August 12, 2026 19:23

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Aug 12, 2026

jszwedko mentioned this pull request Aug 12, 2026

Allow configuring the TLS handshake timeout for HTTP clients. #178

Closed

webern approved these changes Aug 13, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(io): bound the TLS handshake performed when tunneling HTTPS through a proxy - #2309

fix(io): bound the TLS handshake performed when tunneling HTTPS through a proxy#2309
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
mainfrom
jszwedko/hyper-http-proxy-tls-timeout

jszwedko commented Aug 11, 2026 •

edited

Loading

Uh oh!

jszwedko Aug 11, 2026

Uh oh!

webern Aug 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-3 Bot left a comment

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

webern Aug 13, 2026

Uh oh!

webern Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jszwedko commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Human Summary

AI Summary

Test plan

Uh oh!

jszwedko Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

webern Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

datadog-prod-us1-3 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

Uh oh!

webern Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

webern Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jszwedko commented Aug 11, 2026 •

edited

Loading

pr-commenter Bot commented Aug 11, 2026 •

edited

Loading

pr-commenter Bot commented Aug 11, 2026 •

edited

Loading