Description
If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
[email protected] where the issue will be triaged appropriately.
Title: WebSocket upgrades only work for the first stream when allow_connect is enabled in http2_protocol_options
Description:
It looks like the following configuration breaks WebSocket support:
http2_protocol_options:
allow_connect: trueThe first stream of a http/2 connection can upgrade to a WebSocket, but subsequent streams cannot. I don't think this is always true, though. If the WebSocket is closed, then a new WebSocket can upgrade immediately after. This seems to be easily reproducible when lots of streams are opened, like when loading a web page.
The behaviour is pretty strange, as envoy does end up sending requests to the upstream, but they're broken and the upstream (in this case, self hosted GitLab) gets really confused and responds with both a 400 and 404 depending on where you look.
Repro steps:
- Have a plain HTTP/1.1 upstream which handles WebSockets.
- Configure envoy as a HTTP/2 server, with support for WebSockets.
- Enable http2_protocol_options.allow_connect in the HTTP connection manager.
- Load a web page which connects to a WebSocket. The WebSocket will never upgrade.
- Restart envoy, and let the client create a new WebSocket without making any other requests. It will succeed.
- Disable http2_protocol_options.allow_connect and restart envoy. WebSockets will work as normal.
Admin and Stats Output:
Unfortunately the trace logs gave literally no information.
Config:
admin:
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
address:
socket_address:
address: "::"
port_value: 9901
ipv4_compat: true
static_resources:
listeners:
- name: listener_http
address:
socket_address:
address: "::"
port_value: 8080
ipv4_compat: true
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http_redirect
route_config:
name: redirect_route
virtual_hosts:
- name: redirect_to_https
domains: ["*"]
routes:
- match:
prefix: /
redirect:
https_redirect: true
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
- name: listener_h2
address:
socket_address:
address: "::"
port_value: 8443
ipv4_compat: true
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match:
prefix: /
route:
cluster: http_backend
timeout: 360s
retry_policy:
retry_on: 5xx
num_retries: 3
per_try_timeout: 120s
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
http2_protocol_options:
allow_connect: true
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
upgrade_configs:
- upgrade_type: websocket
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/envoy/tls.crt
private_key:
filename: /etc/envoy/tls.key
alpn_protocols:
- h2
- http/1.1
clusters:
- name: http_backend
connect_timeout: 0.25s
type: logical_dns
load_assignment:
cluster_name: http_backend
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: "gitlab"
port_value: 8181Logs:
Envoy:
[2025-03-05T00:14:32.325Z] "GET /-/cable HTTP/2" 400 - 0 0 38 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" "76c9abda-3905-48fd-a9ec-acc4d77184cd" "<redacted>" "10.32.151.111:8181"From the upstream (GitLab):
{"backend_id":"rails","content_type":"text/plain","correlation_id":"01JNHTDEYZDDQ8XM0F62R6501H","duration_ms":20,"host":"<redacted>","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.18.181.141:45314","remote_ip":"172.18.181.141","route":"^/-/cable\\z","route_id":"action_cable","status":400,"system":"http","time":"2025-03-05T00:13:46Z","ttfb_ms":20,"uri":"/-/cable","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36","written_bytes":51}Call Stack:
N/A