Bug: maxLatencyWriter panics on hijacked connections with OpenTelemetry enabled

[probelabs]

Root Cause Analysis

The crash is caused by a nil pointer dereference when the Gateway attempts to asynchronously flush an HTTP response on a hijacked connection.

1. Hijacked Connection: When an API uses WebSockets, Server-Sent Events (SSE), or custom streaming, the connection is hijacked. Once hijacked, the standard Go HTTP server stops managing the connection buffer.
2. Asynchronous Flush: Tyk Gateway uses a maxLatencyWriter which spawns a background goroutine (delayedFlush) to periodically flush the response writer. For WebSockets and SSE, the flush interval is hardcoded to -1 (flush immediately).
3. The Crash: When OpenTelemetry is enabled, the response writer is wrapped by otelhttp and httpsnoop. The delayedFlush goroutine triggers a flush that propagates through this middleware chain down to the standard net/http and bufio packages. Because the connection has already been hijacked, the underlying buffer is invalid (nil), resulting in a segmentation violation (SIGSEGV) when bufio.Flush is called.

Steps to Reproduce

1. Enable OpenTelemetry in tyk.conf (opentelemetry.traces.enabled = true).
2. Create an API that proxies to a WebSocket backend.
3. Send a WebSocket upgrade request to the Gateway.
4. The Gateway will panic with a nil pointer dereference in maxLatencyWriter.delayedFlush.

A test case has been added in gateway/reproduce_panic_test.go that reproduces this exact scenario.

Impact

Gateway crashes in production when OpenTelemetry is enabled and WebSocket/SSE traffic is processed.

Workaround

Disable OpenTelemetry globally until the bug is fixed. There is no API-level override to disable OpenTelemetry for specific APIs.

[probelabs]

Thank You for This Detailed Bug Report

Hello @probelabs[bot], and thank you for opening this issue. Your root cause analysis is exceptionally clear and detailed, and we appreciate you providing steps to reproduce and a dedicated test case. This level of detail is incredibly helpful for our team.

Triage and Analysis

Based on your report, we understand the issue to be a critical bug where the Gateway panics on hijacked connections (like WebSockets or SSE) when OpenTelemetry is enabled. The core of the problem appears to be a race condition where the maxLatencyWriter's delayedFlush goroutine attempts to flush a response writer whose underlying buffer has been set to nil post-hijack, leading to a nil pointer dereference.

• Category: Bug
• Priority: High (due to production crashes)
• Complexity: Moderate
• Impact: Service unavailability for APIs using WebSockets/SSE with OpenTelemetry enabled.

Your analysis seems spot-on. The interaction between the otelhttp middleware, our maxLatencyWriter, and the standard library's handling of hijacked connections is the key area to investigate. The workaround to disable OpenTelemetry globally is a reasonable temporary measure for affected users.

Next Steps

The engineering team will use your report and the provided test case as a starting point for developing a fix. The solution will likely involve preventing the delayedFlush from executing on connections that have been successfully hijacked.

We will track the progress of this issue here. Thank you again for the excellent report.

References

• gateway/middleware_otel.go – OpenTelemetry middleware wrapper.
• gateway/proxy_max_latency_writer.go – Implementation of maxLatencyWriter and the delayedFlush goroutine.
• gateway/reproduce_panic_test.go – The test case that reproduces the panic.

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>