ICAP: no connect/OPTIONS timeout — a black-holed ICAP server hangs every request for the full OS SYN timeout, bypass does not help #66

Summary

When an auditor is configured with icap_reqmod_service / icap_respmod_service, g3proxy has no application-level timeout on establishing the ICAP TCP connection (nor on the subsequent ICAP OPTIONS request). If the ICAP host becomes unreachable at the network level (SYN packets silently dropped / black-holed), every audited request blocks inside TcpStream::connect() for the full OS SYN timeout (~127 s with the default tcp_syn_retries = 6) before the error path runs.

Crucially, bypass: true does not mitigate this: bypass is only evaluated after adapter creation returns an error, and that error only surfaces once connect() has finally given up. So during an ICAP network outage the proxy appears to hang and holds client requests for ~2 minutes each, even with bypass: true.

This differs from a refused ICAP port (TCP RST), where the error is immediate and bypass works as intended.

Environment

• g3proxy 1.13.0 (built from source); the relevant code path is identical on the 1.12 LTS branch.
• Linux, net.ipv4.tcp_syn_retries = 6 (distro default) → ~127 s connect timeout.

Reproduction

Auditor config (both ICAP services point at a black-holed address — a host/port where SYN is dropped, not refused):

auditor:
  - name: default
    protocol_inspection: { }
    task_audit_ratio: 1.0
    icap_reqmod_service:
      url: icap://10.2.221.9:1344/echo   # SYN silently dropped
      bypass: true
    icap_respmod_service:
      url: icap://10.2.221.9:1344/echo
      bypass: true

server:
  - name: http
    type: http_proxy
    escaper: default
    auditor: default
    listen: { address: "127.0.0.1:3199" }
    task_idle_check_interval: 5s          # does NOT help (see below)
    task_idle_max_count: 2

Request through the proxy:

curl -x http://127.0.0.1:3199 http://example.com

Observed vs expected

Expected: with bypass: true, an unreachable ICAP server should fail open within a bounded, configurable time regardless of whether the failure is a RST or a black-hole.

task_idle_check_interval / task_idle_max_count do not help — the stall happens inside connect(), before any adaptation/idle loop is entered (verified: the hang persisted with task_idle_check_interval: 5s).

Root cause

IcapConnector::create() performs a bare connect with no timeout (only the TLS handshake is bounded, by handshake_timeout):

// lib/g3-icap-client/src/service/connection.rs
let peer = self.select_peer_addr().await?;
let socket = g3_socket::tcp::new_socket_to(peer.ip(), /* ... */)?;
let stream = socket.connect(peer).await?;   // <-- no timeout

IcapServiceClient::fetch_connection() then issues the ICAP OPTIONS request, also without a timeout:

// lib/g3-icap-client/src/service/client.rs
let mut conn = self.conn_creator.create().await /* ... */;   // blocks here on black-hole
let options = options_req.get_options(&mut conn, /* ... */).await /* ... */;  // also unbounded

The ICAP service config parser (lib/g3-icap-client/src/service/config/yaml.rs) exposes no connect or OPTIONS timeout — the only timeouts available are preview_data_read_timeout (RESPMOD preview only) and the TLS handshake_timeout.

Because bypass is only checked on the Err branch of adapter creation (e.g. g3proxy/src/inspect/http/v1/forward/mod.rs), it cannot fire until connect() returns — i.e. after the full OS SYN timeout.

Proposed fix

Add configurable timeouts to the ICAP service, applied in IcapConnector::create() / fetch_connection():

• connect_timeout — wrap socket.connect(peer) with tokio::time::timeout(...).
• options_timeout (or reuse a general icap_read_timeout) — bound the OPTIONS request.

On timeout, return the same error used for a failed connection so that the existing bypass logic fails open promptly. Sensible defaults (e.g. a few seconds) would make bypass: true behave correctly for black-holed endpoints out of the box.

Optionally, a transaction-level timeout for REQMOD/RESPMOD response reads would also help the related case where an ICAP server accepts the connection but then stops responding mid-transaction (currently only caught by the task idle checker after task_idle_check_interval × task_idle_max_count, and it errors the task rather than bypassing).

Workaround (for reference)

Until a connect timeout exists, the only way to keep bypass effective during a network-level ICAP outage is to make the failure a RST instead of a silent drop — e.g. firewall the ICAP path with REJECT rather than DROP, or lower net.ipv4.tcp_syn_retries system-wide (which affects all outbound connections).