gateway-api tests and benchmarks

[vixns]

a nice benchmark of gateway-api implementations, some envoy gateway issues highlighted

https://github.com/howardjohn/gateway-api-bench?tab=readme-ov-file

[arkodg]

hey @vixns we're digging into the claims and seeing how they stack up against real-world Envoy Gateway use cases, will share that when ready.
In the meantime, if you've got a specific scenario or any burning questions, please feel free to raise them.

[arkodg]

hey @vixns we had a chance to look into the repo, and the claims made, sharing our perspective below

Enterprise Vendors
A quick look at the logos on https://gateway.envoyproxy.io/ indicates multiple vendors and end-user companies have adopted Envoy Gateway. The CNCF stats dashboard highlights the contributions across multiple companies.

Route Change
The test randomly adds and removes a backendRef filter (different from a top-level per-rule filter) while sending continuous traffic and evaluating the response.

Envoy Gateway’s implementation differs from other Envoy Proxy-based controllers in this regard; it selectively uses weighted clusters only when backendRef filters are configured, due to the limitations highlighted here, and defaults to non-weighted xDS clusters. The test exploits this design decision, resulting in clusters being continuously created, destroyed, and swapped. This results in connection drops to the backend.

If this specific case were to become a valid use case, the traffic drop could be circumvented by configuring retries. This is another difference with other implementations, which may configure retries by default, affecting non-idempotent APIs. The more typical use cases, such as rolling updates of a Deployment or adding, removing, or updating weights for a Canary Service using backendRefs do not result in downtime.

Architecture
Envoy Gateway’s default architecture involves deploying a unique Envoy Proxy per Gateway resource in the Controller Namespace (usually envoy-gateway-system). Each Envoy Proxy holds xDS specific to the corresponding Gateway and cannot be modified by non-admins.
This deployment model limits the permissions given to Envoy Gateway to create or delete Deployments, Services, and Service Accounts in other namespaces, which is a requirement imposed by many Security teams.

Envoy Gateway also supports a Gateway Namespace Mode, which deploys the Envoy Proxy fleet in the Gateway resource namespace. This mode does require the user to give more permissions to Envoy Gateway.

Memory
Members of the community have tested Envoy Gateway and have been unable to detect any memory leaks.

container_memory_working_set_bytes is not a reliable way to measure an application’s memory leak, as it represents RAM (pages) that the operating system cannot reclaim. If Golang’s GC releases heap memory, it may not result in the pages being reclaimed by the OS.
Envoy Gateway intentionally uses memory to cache resources, which has two benefits.

• Reduces CPU - Resource processing and translation are broken into several steps, and if the resource has not changed, subsequent translations are skipped to save CPU
• Reduces I/O - Allows Envoy Gateway to perform delta xDS with Envoy Proxy and only push incremental updates, instead of the entire xDS configuration.

Performance Metrics
This highlights room for improvement within the project to enhance the benchmark tests and better present the results, which are run on every PR and for every release. The work is tracked in #6182, which we will prioritize; contributions to these efforts are welcome from the community.

[howardjohn]

(Benchmark author here)

Thanks for the response!

Re vendors -- sorry for the confusion here, the intent of this row was to indicate who you can pay to support the product, not who is using it. I've pushed an update to clarify this. WRT to namespace mode, I had previously pushed a note about the new Namespace Mode (unfortunately, it came out after I had written that section so I initially missed it!). If/when I do a part 2 I will make sure to use that mode.

Re the route change -- this test case is derived from real world experience with multiple users that have have outages during route updates. For instance, Knative maintainers have given talks about this specific issue. Also I don't think retries would fix this given the errors are NC errors in Envoy which are not retriable (I didn't verify this).

Regarding memory. container_memory_working_set_bytes is both the correct metric to use (you can learn more at https://onairotich.medium.com/understand-container-metrics-and-why-they-matter-9e88434ca62a or others) but also irrelevant to this test. The Go heap is legitimately using that memory, it is not page caching. Envoy Gateway may use RAM to tradeoff on IO and CPU, but fwiw it also uses the highest amount of Data Plane CPU (suggesting it may not effectively reduce IO) and 2nd highest Control Plane CPU which suggests it may not be making an effective tradeoff. Its easy to reproduce the issue following the steps in the repo (and other scenarios).

[zirain]

Re the route change -- this test case is derived from real world experience with multiple users that have have outages during route updates.

This's a fundamental issue around envoy (delta) update, detail can be found here.
You cannot reproduce it in istio or other implement because istio push all services(1:1 mapping to cluster) by default.
You can reproduce this if you enable PILOT_FILTER_GATEWAY_CLUSTER_CONFIG, or create an new service with the networking policy at the same time.

[howardjohn]

It is clearly not a fundamental issue of Envoy that you must have outages during route updates, as the benchmark tests 4 different Envoy implementations and only 1 of them has this issue. Even if it was, the purpose of the benchmark is to show how well an implementation behaves -- so whether it is a data plane or control plane issue is irrelevant, the end result is outages during route updates.

PILOT_FILTER_GATEWAY_CLUSTER_CONFIG is off, and will be off forever, in Istio specifically because it causes this issue which is deemed unsafe.

create an new service with the networking policy at the same time.

This is tested in the test as well -- it is not expected for this to be zero downtime since this is an eventually consistent operation, switching from Invalid state --> valid state. However, switching config from 1 valid state to another valid state is something that should remain in a valid state throughout.

[zirain]

What I want to say that, it's a trade off.
EG choose not to send all servcies/clusters by default, we will try to improve the updating flow.

[zhuoyang]

There's a part 2 of the benchmark
https://github.com/howardjohn/gateway-api-bench/blob/main/README-v2.md

[zhuoyang]

I am particularly interested and concerned about the route propagation test, as it is unexpected that it returns 503 errors
I tried to run the test myself and figure out what is causing the issue, it seems like under load there is some sort of race condition between creation of cluster and routes. I guess this is also a side effect of EG creating a cluster per httproute.

On the bright side, putting a delay between the creation of each httproute does eliminate all the 503 errors. For my use case it is probably good enough as I don't create httproute as often.

[zirain]

please checkout #5307

[cbf123]

Did anyone ever take another look at the memory consumption from the "Route Scale" test? https://github.com/howardjohn/gateway-api-bench?tab=readme-ov-file#route-scale

There was an initial objection about which metric was being used, but the benchmark author responded trying to justify his choice. I haven't seen any response to that explanation.

Does anyone know what the cause of the memory consumption is? Would I see it with a less-frequently-changing configuration or is it caused by the changes?