Add HTTP/2 ping-based health checks to `auth.Manager`'s `http.Client` to avoid long-lived half-dead JWKS/enrollment connections

[etgraylog]

Expected Behavior

The auth manager's HTTP client (used for GET /.well-known/jwks.json and enrollment-related requests) could detect a half-dead HTTP/2 connection via proactive PING frames and evict it promptly, so the JWKS fetch retry loop would recover within seconds rather than being stalled by a silent-dead peer.

HTTP/2 PING (RFC 9113 §6.7) is an optional keepalive mechanism; neither Go's stdlib nor x/net/http2 enables it by default. Configuring it is a policy choice rather than a specified behavior — valid alternatives include tighter per-request timeouts, application-layer heartbeats, or relying on TCP keepalive (with its ~11-minute detection window at Linux defaults).

Current Behavior

A sibling issue in open-telemetry/opamp-go covers the same fix pattern for the library's HTTPSender HTTP client.

superv/auth/manager.go constructs its own http.Client when cfg.HTTPClient is not provided, with a bare http.Transport and no HTTP/2 ping configuration:

httpClient = &http.Client{Timeout: 30 * time.Second, Transport: &http.Transport{
    Proxy:                 http.ProxyFromEnvironment,
    ForceAttemptHTTP2:     true,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    TLSClientConfig:       &tls.Config{InsecureSkipVerify: cfg.InsecureTLS},
    ExpectContinueTimeout: 1 * time.Second,
}}

The auth.Manager's HTTP/2 client maintains long-lived connections for periodic certificate-signing-request operations after enrollment is complete. When the upstream silently drops one of these connections (reverse-proxy worker recycle, transient conntrack eviction, brief path loss), the client is left blocked on a read with no fast-detection mechanism. Without HTTP/2 PING configured, detection relies on either ResponseHeaderTimeout (not set here) or kernel TCP keepalive (~11 minutes).

Under a service manager like SystemD or Kubernetes the resulting long stalls can cascade into process restarts via the caller's 401-handling logic; without a process manager, the collector remains blocked on the stale connection for up to ~11 minutes.

Possible Solution

Configure the HTTP/2 ping via http.Transport.HTTP2 (Go 1.24+):

httpClient = &http.Client{Timeout: 30 * time.Second, Transport: &http.Transport{
    Proxy:                 http.ProxyFromEnvironment,
    ForceAttemptHTTP2:     true,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    TLSClientConfig:       &tls.Config{InsecureSkipVerify: cfg.InsecureTLS},
    ExpectContinueTimeout: 1 * time.Second,
    HTTP2: &http.HTTP2Config{
        SendPingTimeout: 30 * time.Second,  // PING after 30s of inactivity
        PingTimeout:     10 * time.Second,  // close conn if no PONG in 10s
    },
}}

This configuration applies to the *http2.Transport created by Go's auto-upgrade path. On a healthy connection, SendPingTimeout is always reset by inbound frames (server responses, server-initiated frames) so healthy traffic is unaffected. On a half-dead connection, PINGs go unanswered, the connection is evicted within SendPingTimeout + PingTimeout seconds (~40s), and the next request dials a fresh connection — restoring the retry-loop's ability to make progress.

Note: consider also reducing http.Client.Timeout from 30s to ~10s or switching to per-request context deadlines, so individual JWKS fetch attempts fail faster once dead-connection detection is active. The current 30s-per-attempt × 5-retry budget of ~2.5 minutes was sized before HTTP/2 PING-based detection was readily configurable; with PING active a tighter per-request budget is safe.

Tested on a live 2000-collector fleet

Fix applied across a 2000-collector fleet. GODEBUG=http2debug=2 on a sample collector process shows, every SendPingTimeout interval:

http2: Transport sending health check
http2: Framer ...: wrote PING len=8 ping=...
http2: Framer ...: read PING flags=ACK len=8 ping=...
http2: Transport health check success

PING fires at SendPingTimeout intervals during idle windows, and detection of a silently-dropped connection completes within SendPingTimeout + PingTimeout (~40 s), at which point the transport evicts the dead connection so the next request dials fresh.

Mechanism-equivalence to the opamp-go HTTP/2 keepalive fix

This is the same http.Transport.HTTP2 config surface, applied to the separate auth.Manager HTTP client. A single-pod A/B isolation test was run for the opamp-go sibling fix: the patched build detected a half-dead connection in ~40 s via SendPingTimeout + PingTimeout; the unpatched build had no idle-liveness mechanism. The runtime behavior here follows the same path — the config travels through the same stdlib HTTP/2 code — applied to the separate auth.Manager connection lifecycle (enrollment + periodic CSR renewal + JWKS refresh). The sibling's A/B result is mechanism-equivalent evidence for this patch. A fleet-scale in-isolation test of this specific auth.Manager path was not performed because its failure-mode window is narrower (active only during enrollment/renewal/JWKS fetches) and the PING mechanism is provided by Go's stdlib once the config is set.

Unit tests TestNewManager_DefaultHTTPClientHasHTTP2Keepalive and TestNewManager_CallerSuppliedHTTPClientUntouched cover the config plumbing at the auth.Manager boundary (default transport gets SendPingTimeout=30s, PingTimeout=10s; a caller-supplied transport is not overridden).

Steps to Reproduce

1. Deploy 1500+ collector-sidecar processes against a load-balanced graylog-server fronted by an HTTP/2 reverse proxy (in the observation environment: nginx-ingress behind kube-proxy).
2. Trigger a simultaneous cold start (e.g. a Kubernetes Deployment scaled from 0 → 1500; on a SystemD-hosted fleet, systemctl start across the hosts in a tight window).
3. Watch process restart counts. A meaningful fraction (typically 0.5-1% of the fleet) of processes will hit fetch JWKS failed after 5 attempts in their previous-instance log.
4. Per-process log analysis: in affected processes, each of the 5 JWKS attempts shows http2: Transport failed to get client conn for <graylog-host>:443 followed by a long silence ending in context deadline exceeded (Client.Timeout exceeded while awaiting headers).

Context

A sibling fix for the same pattern is proposed against open-telemetry/opamp-go for that library's HTTPSender client. Both clients — opamp-go's HTTPSender and collector-sidecar's auth.Manager — run HTTP/2 against the same upstream, but they are distinct clients with independent transports. Applying the fix in one place alone leaves the other client vulnerable to the same failure mode; both benefit from the same configuration.

The two clients are not trivially shareable: auth.Manager runs during enrollment to validate the server's TLS chain via JWKS, before the opamp-go client is constructed, and continues independently afterwards for CSR renewal and JWKS refresh. Reusing a single transport would couple two layers that today have independent lifecycles.

The underlying change is the same in both places (http.Transport.HTTP2 = &http.HTTP2Config{...}); only the file and caller differ.

Your Environment

• Graylog Version: 7.1.0-beta.1
• Go Version: 1.25.9 (collector build)
• OpenSearch Version: 2.x (not relevant)
• MongoDB Version: 8.0.20 (not relevant)
• Operating System: Debian 12 on K3s 1.35

Checklist

[x] This issue fix need to be backported — applies to all versions of collector-sidecar that use auth.Manager with a default (non-user-supplied) HTTPClient.
[ ] Does this issue have security implications? No.