Open
Description
Title: Misbehaviour with External Processing Filter in Buffered mode since v1.32.0
Description:
Since v1.32.0, Envoy no longer reliably returns the status “413 Payload too large” if the External Processing filter is used in request body BUFFERED mode and if clients send payloads that are larger than the buffer limit per connection (per_connection_buffer_limit_bytes). Instead, requests occasionally get stuck until a timeout occurs.
- start external processing (gRPC) server
> git clone https://github.com/wrossmorrow/envoy-extproc-sdk-go
> cd envoy-extproc-sdk-go/examples
> go run main.go noop.go echo.go trivial.go timer.go data.go digest.go dedup.go masker.go echo --port 12345
- start HTTP server
python3 -m http.server 54321
- run Envoy
> func-e use 1.32.0
> func-e run -c envoy.yaml --base-id 1
with attached configuration below.
- send payload bigger than per_connection_buffer_limit_bytes
> yes "testdata" | head -c 10M > data
> while [ 1 ]; do echo && date && sleep 1 && curl -X POST -v [127.0.0.1:10000](http://127.0.0.1:10000/) -d @data; done
- see curl waiting for the response from Envoy
* Trying 127.0.0.1:10000...
* Connected to 127.0.0.1 (127.0.0.1) port 10000 (#0)
> POST / HTTP/1.1
> Host: [127.0.0.1:10000](http://127.0.0.1:10000/)
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 7864320
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
<no response recevied => timeout>
NOTE: Several attempts are required to reproduce the behaviour that envoy does not return the status 413 but stops.
NOTE: The misbehaviour can be intensified by deactivating (not starting) the upstream server.
NOTE: Earlier versions (e.g. v.1.31.7) do not seem to show this behaviour.
Config:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": [type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager](http://type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager)
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: service_cluster
http_filters:
- name: envoy.filters.http.ext_proc
typed_config:
"@type": [type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor](http://type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor)
failure_mode_allow: false
allow_mode_override: true
processing_mode:
request_header_mode: "SEND"
response_header_mode: "SKIP"
request_body_mode: "BUFFERED"
response_body_mode: "NONE"
request_trailer_mode: "SKIP"
response_trailer_mode: "SKIP"
grpc_service:
envoy_grpc:
cluster_name: ext_proc_cluster
message_timeout: "100s"
- name: envoy.filters.http.router
typed_config:
"@type": [type.googleapis.com/envoy.extensions.filters.http.router.v3.Router](http://type.googleapis.com/envoy.extensions.filters.http.router.v3.Router)
clusters:
- name: service_cluster
connect_timeout: 2s
type: STRICT_DNS
load_assignment:
cluster_name: service_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 54321
- name: ext_proc_cluster
type: STATIC
connect_timeout: 0.25s
http2_protocol_options: {
}
load_assignment:
cluster_name: ext_proc_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 12345