Maximum number of practical routes?

[wandbjake]

Describe the bug
I have an HTTPRoute with 6 rule matches that I'm using to create ~200 environments for ephemeral branch environments. Each lives in a vcluster and is idle most of the time, so traffic volume is not an issue. At a certain point though, the gateway fails to start up.

To Reproduce

jacob.pusateri@Mac scratch % kubectl get HTTPRoute -A | wc -l
     179

By varying the matches, I can try and determine the limit. Note, I had to kill the pod after each iteration, as deploying 200 new HTTPRoutes sent things into a bit of a tailspin.

Matches	nginx container memory	nginx.conf lines	nginx.conf file size
2	15 GB	65k	1.9MB
3	20 GB	79k	2.4MB

And already at the 3 case we're in trouble as the nginx-gateway process gets stuck in a loop of

│ {"level":"info","ts":"2025-03-26T12:55:45Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":506}                                                                                                     │
│ {"level":"info","ts":"2025-03-26T12:55:45Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":507}                                                                                                     │
│ {"level":"info","ts":"2025-03-26T12:55:46Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}                                                                                                                                         │
│ {"level":"info","ts":"2025-03-26T12:55:46Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":508}                                                                                                     │
│ {"level":"info","ts":"2025-03-26T12:55:46Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":509}                                                                                                     │

and never comes up

Expected behavior
NGINX Gateway Fabric scales to, say 5000 routes. I can easily imagine a future with double the number of environments so 400 and double the number of services, so 12. Rounds out to 5000. Maybe I need multiple gateways though?

Your environment

• Version of the NGINX Gateway Fabric - release version or a specific commit. The first line of the nginx-gateway container logs includes the commit info.
• {"level":"info","ts":"2025-03-26T12:46:15Z","msg":"Starting NGINX Gateway Fabric in static mode","version":"1.5.1","commit":"7b609828f77a05dd9ca3d27a41e08cd7f454fdaf","date":"2024-12-16T20:41:56Z","dirty":"false"} │
• Version of Kubernetes
• v1.31.6-gke.1020000
• Kubernetes platform (e.g. Mini-kube or GCP)
• GCP
• Details on how you expose the NGINX Gateway Fabric Pod (e.g. Service of type LoadBalancer or port-forward)
• LoadBalancer
• Logs of NGINX container: kubectl -n nginx-gateway logs -l app=nginx-gateway -c nginx

(wandb-local) jacob.pusateri@Mac integration % kubectl -n gateway-system logs nginx-gateway-nginx-gateway-fabric-64cc74f998-qjn9c
Defaulted container "nginx-gateway" out of: nginx-gateway, nginx, init (init)
{"level":"info","ts":"2025-03-26T12:46:15Z","msg":"Starting NGINX Gateway Fabric in static mode","version":"1.5.1","commit":"7b609828f77a05dd9ca3d27a41e08cd7f454fdaf","date":"2024-12-16T20:41:56Z","dirty":"false"}
{"level":"info","ts":"2025-03-26T12:46:16Z","msg":"Starting manager"}
{"level":"info","ts":"2025-03-26T12:46:16Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-03-26T12:46:16Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-03-26T12:46:16Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":9113","secure":false}
{"level":"info","ts":"2025-03-26T12:46:16Z","msg":"attempting to acquire leader lease gateway-system/nginx-gateway-nginx-gateway-fabric-leader-election..."}
{"level":"info","ts":"2025-03-26T12:46:18Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:19Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:20Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:21Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:22Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:23Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:24Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:25Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:26Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:27Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:28Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:29Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:30Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:31Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:32Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:33Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:34Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:34Z","msg":"successfully acquired lease gateway-system/nginx-gateway-nginx-gateway-fabric-leader-election"}
{"level":"info","ts":"2025-03-26T12:46:35Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:36Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:37Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:38Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:39Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:40Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:41Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:42Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:43Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:44Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:45Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:46Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:47Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:48Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:49Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:50Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:51Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:52Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:53Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:54Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:55Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:56Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:57Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:58Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:46:59Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:00Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:01Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:02Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:03Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:04Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:05Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:06Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:07Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:08Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:09Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:10Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:11Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:12Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:13Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:14Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:15Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:16Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:17Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:18Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"error","ts":"2025-03-26T12:47:18Z","logger":"eventLoop.eventHandler","msg":"Failed to update NGINX configuration","batchID":1,"error":"failed to reload NGINX: reload unsuccessful: no new NGINX worker processes started for config version 1. Please check the NGINX container logs for possible configuration issues: context deadline exceeded","stacktrace":"github.com/nginxinc/nginx-gateway-fabric/internal/mode/static.(*eventHandlerImpl).HandleEventBatch\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/mode/static/handler.go:209\ngithub.com/nginxinc/nginx-gateway-fabric/internal/framework/events.(*EventLoop).Start.func1.1\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/framework/events/loop.go:74"}
{"level":"info","ts":"2025-03-26T12:47:18Z","logger":"eventLoop.eventHandler","msg":"Reconfigured control plane.","batchID":2}
{"level":"info","ts":"2025-03-26T12:47:19Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:20Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:21Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:22Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:23Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:24Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:25Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:26Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:27Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:28Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:29Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:30Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:31Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:32Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:33Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:34Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:35Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:35Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:36Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:37Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:38Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:39Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:40Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:41Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:42Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:43Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:44Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:45Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:46Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:47Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:48Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:49Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:50Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:51Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:52Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:53Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:54Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:55Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:56Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:57Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:58Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:47:59Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:00Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:01Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:02Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:03Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:04Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:05Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:06Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:07Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:08Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:09Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:10Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:11Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:12Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:13Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:14Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:15Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:16Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:17Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:18Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"error","ts":"2025-03-26T12:48:18Z","logger":"eventLoop.eventHandler","msg":"Failed to update NGINX configuration","batchID":2,"error":"failed to reload NGINX: could not get expected config version 2: config version check didn't return expected version 2 within the deadline","stacktrace":"github.com/nginxinc/nginx-gateway-fabric/internal/mode/static.(*eventHandlerImpl).HandleEventBatch\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/mode/static/handler.go:209\ngithub.com/nginxinc/nginx-gateway-fabric/internal/framework/events.(*EventLoop).Start.func1.1\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/framework/events/loop.go:74"}
{"level":"info","ts":"2025-03-26T12:48:18Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":3}
{"level":"info","ts":"2025-03-26T12:48:19Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:20Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:20Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":4}
{"level":"info","ts":"2025-03-26T12:48:20Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":5}
{"level":"info","ts":"2025-03-26T12:48:21Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:21Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":6}
{"level":"info","ts":"2025-03-26T12:48:21Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":7}
{"level":"info","ts":"2025-03-26T12:48:22Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":8}
{"level":"info","ts":"2025-03-26T12:48:22Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":9}
{"level":"info","ts":"2025-03-26T12:48:22Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:22Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":10}
{"level":"info","ts":"2025-03-26T12:48:22Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":11}
{"level":"info","ts":"2025-03-26T12:48:23Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:23Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":12}
{"level":"info","ts":"2025-03-26T12:48:23Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":13}
{"level":"info","ts":"2025-03-26T12:48:23Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":14}
{"level":"info","ts":"2025-03-26T12:48:23Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":15}
{"level":"info","ts":"2025-03-26T12:48:24Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":16}
{"level":"info","ts":"2025-03-26T12:48:24Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":17}
{"level":"info","ts":"2025-03-26T12:48:24Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:25Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:26Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:27Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:28Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:29Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:30Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:31Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
{"level":"info","ts":"2025-03-26T12:48:32Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":18}
{"level":"info","ts":"2025-03-26T12:48:32Z","logger":"eventLoop.eventHandler","msg":"Handling events didn't result into NGINX configuration changes","batchID":19}

(wandb-local) jacob.pusateri@Mac integration % kubectl -n gateway-system logs nginx-gateway-nginx-gateway-fabric-64cc74f998-qjn9c -c nginx
2025/03/26 12:46:16 [notice] 45#45: using the "epoll" event method
2025/03/26 12:46:16 [notice] 45#45: nginx/1.27.2
2025/03/26 12:46:16 [notice] 45#45: built by gcc 13.2.1 20240309 (Alpine 13.2.1_git20240309)
2025/03/26 12:46:16 [notice] 45#45: OS: Linux 6.6.72+
2025/03/26 12:46:16 [notice] 45#45: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2025/03/26 12:46:16 [notice] 45#45: start worker processes
2025/03/26 12:46:16 [notice] 45#45: start worker process 52
2025/03/26 12:46:16 [notice] 45#45: start worker process 53
2025/03/26 12:46:16 [notice] 45#45: start worker process 54
2025/03/26 12:46:16 [notice] 45#45: start worker process 55
2025/03/26 12:46:16 [notice] 45#45: start worker process 56
2025/03/26 12:46:16 [notice] 45#45: start worker process 57
2025/03/26 12:46:16 [notice] 45#45: start worker process 58
2025/03/26 12:46:16 [notice] 45#45: start worker process 59
2025/03/26 12:46:16 [notice] 45#45: start worker process 60
2025/03/26 12:46:16 [notice] 45#45: start worker process 61
2025/03/26 12:46:16 [notice] 45#45: start worker process 62
2025/03/26 12:46:16 [notice] 45#45: start worker process 63
2025/03/26 12:46:16 [notice] 45#45: start worker process 64
2025/03/26 12:46:16 [notice] 45#45: start worker process 65
2025/03/26 12:46:16 [notice] 45#45: start worker process 66
2025/03/26 12:46:16 [notice] 45#45: start worker process 67
2025/03/26 12:46:16 [notice] 45#45: start worker process 68
2025/03/26 12:46:16 [notice] 45#45: start worker process 69
2025/03/26 12:46:16 [notice] 45#45: start worker process 70
2025/03/26 12:46:16 [notice] 45#45: start worker process 71
2025/03/26 12:46:16 [notice] 45#45: start worker process 72
2025/03/26 12:46:16 [notice] 45#45: start worker process 73
2025/03/26 12:46:16 [notice] 45#45: start worker process 74
2025/03/26 12:46:16 [notice] 45#45: start worker process 75
2025/03/26 12:46:16 [notice] 45#45: start worker process 76
2025/03/26 12:46:16 [notice] 45#45: start worker process 77
2025/03/26 12:46:16 [notice] 45#45: start worker process 78
2025/03/26 12:46:16 [notice] 45#45: start worker process 79
2025/03/26 12:46:16 [notice] 45#45: start worker process 80
2025/03/26 12:46:16 [notice] 45#45: start worker process 81
2025/03/26 12:46:16 [notice] 45#45: start worker process 82
2025/03/26 12:46:16 [notice] 45#45: start worker process 83
2025/03/26 12:46:18 [notice] 45#45: signal 1 (SIGHUP) received from 24, reconfiguring
2025/03/26 12:46:18 [notice] 45#45: reconfiguring
2025/03/26 12:46:18 [notice] 45#45: using the "epoll" event method
2025/03/26 12:46:18 [notice] 45#45: start worker processes
2025/03/26 12:46:18 [notice] 45#45: start worker process 105
2025/03/26 12:46:18 [notice] 45#45: start worker process 106
2025/03/26 12:46:18 [notice] 45#45: start worker process 107
2025/03/26 12:46:18 [notice] 45#45: start worker process 108
2025/03/26 12:46:18 [notice] 45#45: start worker process 109
2025/03/26 12:46:18 [notice] 45#45: start worker process 110
2025/03/26 12:46:18 [notice] 45#45: start worker process 111
2025/03/26 12:46:18 [notice] 45#45: start worker process 112
2025/03/26 12:46:18 [notice] 45#45: start worker process 113
2025/03/26 12:46:18 [notice] 45#45: start worker process 114
2025/03/26 12:46:18 [notice] 45#45: start worker process 115
2025/03/26 12:46:18 [notice] 45#45: start worker process 116
2025/03/26 12:46:18 [notice] 45#45: start worker process 117
2025/03/26 12:46:18 [notice] 45#45: start worker process 118
2025/03/26 12:46:18 [notice] 45#45: start worker process 119
2025/03/26 12:46:18 [notice] 45#45: start worker process 120
2025/03/26 12:46:18 [notice] 45#45: start worker process 121
2025/03/26 12:46:18 [notice] 45#45: start worker process 122
2025/03/26 12:46:18 [notice] 45#45: start worker process 123
2025/03/26 12:46:18 [notice] 45#45: start worker process 124
2025/03/26 12:46:18 [notice] 45#45: start worker process 125
2025/03/26 12:46:18 [notice] 45#45: start worker process 126
2025/03/26 12:46:18 [notice] 45#45: start worker process 127
2025/03/26 12:46:18 [notice] 45#45: start worker process 128
2025/03/26 12:46:18 [notice] 45#45: start worker process 129
2025/03/26 12:46:18 [notice] 45#45: start worker process 130
2025/03/26 12:46:18 [notice] 45#45: start worker process 131
2025/03/26 12:46:18 [notice] 45#45: start worker process 132
2025/03/26 12:46:18 [notice] 45#45: start worker process 133
2025/03/26 12:46:18 [notice] 45#45: start worker process 134
2025/03/26 12:46:18 [notice] 45#45: start worker process 135
2025/03/26 12:46:18 [notice] 45#45: start worker process 136
2025/03/26 12:47:38 [notice] 52#52: gracefully shutting down
2025/03/26 12:47:38 [notice] 54#54: gracefully shutting down
2025/03/26 12:47:38 [notice] 56#56: gracefully shutting down
2025/03/26 12:47:38 [notice] 55#55: gracefully shutting down
2025/03/26 12:47:38 [notice] 53#53: gracefully shutting down
2025/03/26 12:47:38 [notice] 56#56: exiting
2025/03/26 12:47:38 [notice] 52#52: exiting
2025/03/26 12:47:38 [notice] 61#61: gracefully shutting down
2025/03/26 12:47:38 [notice] 54#54: exiting
2025/03/26 12:47:38 [notice] 59#59: gracefully shutting down
2025/03/26 12:47:38 [notice] 53#53: exiting
2025/03/26 12:47:38 [notice] 60#60: gracefully shutting down
2025/03/26 12:47:38 [notice] 62#62: gracefully shutting down
2025/03/26 12:47:38 [notice] 55#55: exiting
2025/03/26 12:47:38 [notice] 61#61: exiting
2025/03/26 12:47:38 [notice] 59#59: exiting
2025/03/26 12:47:38 [notice] 58#58: gracefully shutting down
2025/03/26 12:47:38 [notice] 62#62: exiting
2025/03/26 12:47:38 [notice] 57#57: gracefully shutting down
2025/03/26 12:47:38 [notice] 58#58: exiting
2025/03/26 12:47:38 [notice] 67#67: gracefully shutting down
2025/03/26 12:47:38 [notice] 64#64: gracefully shutting down
2025/03/26 12:47:38 [notice] 60#60: exiting
2025/03/26 12:47:38 [notice] 66#66: gracefully shutting down
2025/03/26 12:47:38 [notice] 67#67: exiting
2025/03/26 12:47:38 [notice] 64#64: exiting
2025/03/26 12:47:38 [notice] 57#57: exiting
2025/03/26 12:47:38 [notice] 66#66: exiting
2025/03/26 12:47:38 [notice] 63#63: gracefully shutting down
2025/03/26 12:47:38 [notice] 63#63: exiting
2025/03/26 12:47:38 [notice] 65#65: gracefully shutting down
2025/03/26 12:47:38 [notice] 65#65: exiting
2025/03/26 12:47:38 [notice] 45#45: signal 1 (SIGHUP) received from 24, reconfiguring
2025/03/26 12:47:38 [notice] 45#45: reconfiguring
2025/03/26 12:47:38 [notice] 69#69: gracefully shutting down
2025/03/26 12:47:38 [notice] 69#69: exiting
2025/03/26 12:47:38 [notice] 75#75: gracefully shutting down
2025/03/26 12:47:38 [notice] 70#70: gracefully shutting down
2025/03/26 12:47:38 [notice] 75#75: exiting
2025/03/26 12:47:38 [notice] 70#70: exiting
2025/03/26 12:47:38 [notice] 74#74: gracefully shutting down
2025/03/26 12:47:38 [notice] 74#74: exiting
2025/03/26 12:47:38 [notice] 71#71: gracefully shutting down
2025/03/26 12:47:38 [notice] 72#72: gracefully shutting down
2025/03/26 12:47:38 [notice] 71#71: exiting
2025/03/26 12:47:38 [notice] 72#72: exiting
2025/03/26 12:47:38 [notice] 76#76: gracefully shutting down
2025/03/26 12:47:38 [notice] 76#76: exiting
2025/03/26 12:47:38 [notice] 79#79: gracefully shutting down
2025/03/26 12:47:38 [notice] 80#80: gracefully shutting down
2025/03/26 12:47:38 [notice] 79#79: exiting
2025/03/26 12:47:38 [notice] 80#80: exiting
2025/03/26 12:47:38 [notice] 82#82: gracefully shutting down
2025/03/26 12:47:38 [notice] 82#82: exiting
2025/03/26 12:47:38 [notice] 83#83: gracefully shutting down
2025/03/26 12:47:38 [notice] 83#83: exiting
2025/03/26 12:47:38 [notice] 68#68: gracefully shutting down
2025/03/26 12:47:38 [notice] 68#68: exiting
2025/03/26 12:47:38 [notice] 81#81: gracefully shutting down
2025/03/26 12:47:38 [notice] 81#81: exiting
2025/03/26 12:47:38 [notice] 78#78: gracefully shutting down
2025/03/26 12:47:38 [notice] 78#78: exiting
2025/03/26 12:47:38 [notice] 77#77: gracefully shutting down
2025/03/26 12:47:38 [notice] 77#77: exiting
2025/03/26 12:47:38 [notice] 73#73: gracefully shutting down
2025/03/26 12:47:38 [notice] 73#73: exiting
2025/03/26 12:47:38 [notice] 60#60: exit
2025/03/26 12:47:38 [notice] 53#53: exit
2025/03/26 12:47:38 [notice] 65#65: exit
2025/03/26 12:47:38 [notice] 76#76: exit
2025/03/26 12:47:38 [notice] 80#80: exit
2025/03/26 12:47:38 [notice] 70#70: exit
2025/03/26 12:47:38 [notice] 66#66: exit
2025/03/26 12:47:38 [notice] 81#81: exit
2025/03/26 12:47:38 [notice] 77#77: exit
2025/03/26 12:47:38 [notice] 74#74: exit
2025/03/26 12:47:38 [notice] 73#73: exit
2025/03/26 12:47:38 [notice] 67#67: exit
2025/03/26 12:47:38 [notice] 68#68: exit
2025/03/26 12:47:38 [notice] 83#83: exit
2025/03/26 12:47:38 [notice] 55#55: exit
2025/03/26 12:47:38 [notice] 58#58: exit
2025/03/26 12:47:38 [notice] 59#59: exit
2025/03/26 12:47:38 [notice] 69#69: exit
2025/03/26 12:47:38 [notice] 72#72: exit
2025/03/26 12:47:38 [notice] 82#82: exit
2025/03/26 12:47:38 [notice] 57#57: exit
2025/03/26 12:47:38 [notice] 56#56: exit
2025/03/26 12:47:38 [notice] 52#52: exit
2025/03/26 12:47:38 [notice] 71#71: exit
2025/03/26 12:47:38 [notice] 62#62: exit
2025/03/26 12:47:38 [notice] 78#78: exit
2025/03/26 12:47:38 [notice] 64#64: exit
2025/03/26 12:47:38 [notice] 61#61: exit
2025/03/26 12:47:38 [notice] 54#54: exit
2025/03/26 12:47:38 [notice] 79#79: exit
2025/03/26 12:47:38 [notice] 63#63: exit
2025/03/26 12:47:38 [notice] 75#75: exit
2025/03/26 12:47:38 [notice] 45#45: using the "epoll" event method
2025/03/26 12:47:38 [notice] 45#45: start worker processes
2025/03/26 12:47:38 [notice] 45#45: start worker process 137
2025/03/26 12:47:38 [notice] 45#45: start worker process 138
2025/03/26 12:47:38 [notice] 45#45: start worker process 139
2025/03/26 12:47:38 [notice] 45#45: start worker process 140
2025/03/26 12:47:38 [notice] 45#45: start worker process 141
2025/03/26 12:47:38 [notice] 45#45: start worker process 142
2025/03/26 12:47:38 [notice] 45#45: start worker process 143
2025/03/26 12:47:38 [notice] 45#45: start worker process 144
2025/03/26 12:47:38 [notice] 45#45: start worker process 145
2025/03/26 12:47:38 [notice] 45#45: start worker process 146
2025/03/26 12:47:38 [notice] 45#45: start worker process 147
2025/03/26 12:47:38 [notice] 45#45: start worker process 148
2025/03/26 12:47:38 [notice] 45#45: start worker process 149
2025/03/26 12:47:38 [notice] 45#45: start worker process 150
2025/03/26 12:47:38 [notice] 45#45: start worker process 151
2025/03/26 12:47:38 [notice] 45#45: start worker process 152
2025/03/26 12:47:38 [notice] 45#45: start worker process 153
2025/03/26 12:47:38 [notice] 45#45: start worker process 154
2025/03/26 12:47:38 [notice] 45#45: start worker process 155
2025/03/26 12:47:38 [notice] 45#45: start worker process 156
2025/03/26 12:47:38 [notice] 45#45: start worker process 157
2025/03/26 12:47:38 [notice] 45#45: start worker process 158
2025/03/26 12:47:38 [notice] 45#45: start worker process 159
2025/03/26 12:47:38 [notice] 45#45: start worker process 160
2025/03/26 12:47:38 [notice] 45#45: start worker process 161
2025/03/26 12:47:38 [notice] 45#45: start worker process 162
2025/03/26 12:47:38 [notice] 45#45: start worker process 163
2025/03/26 12:47:38 [notice] 45#45: start worker process 164
2025/03/26 12:47:38 [notice] 45#45: start worker process 165
2025/03/26 12:47:38 [notice] 45#45: start worker process 166
2025/03/26 12:47:38 [notice] 45#45: start worker process 167
2025/03/26 12:47:38 [notice] 45#45: start worker process 168
2025/03/26 12:49:03 [notice] 105#105: gracefully shutting down
2025/03/26 12:49:03 [notice] 106#106: gracefully shutting down
2025/03/26 12:49:03 [notice] 108#108: gracefully shutting down
2025/03/26 12:49:03 [notice] 107#107: gracefully shutting down
2025/03/26 12:49:03 [notice] 110#110: gracefully shutting down
2025/03/26 12:49:03 [notice] 109#109: gracefully shutting down
2025/03/26 12:49:03 [notice] 111#111: gracefully shutting down
2025/03/26 12:49:03 [notice] 112#112: gracefully shutting down
2025/03/26 12:49:03 [notice] 113#113: gracefully shutting down
2025/03/26 12:49:03 [notice] 107#107: exiting
2025/03/26 12:49:03 [notice] 106#106: exiting
2025/03/26 12:49:03 [notice] 108#108: exiting
2025/03/26 12:49:03 [notice] 114#114: gracefully shutting down
2025/03/26 12:49:03 [notice] 117#117: gracefully shutting down
2025/03/26 12:49:03 [notice] 115#115: gracefully shutting down
2025/03/26 12:49:03 [notice] 105#105: exiting
2025/03/26 12:49:03 [notice] 109#109: exiting
2025/03/26 12:49:03 [notice] 110#110: exiting
2025/03/26 12:49:03 [notice] 112#112: exiting
2025/03/26 12:49:03 [notice] 113#113: exiting
2025/03/26 12:49:03 [notice] 111#111: exiting
2025/03/26 12:49:03 [notice] 116#116: gracefully shutting down
2025/03/26 12:49:03 [notice] 114#114: exiting
2025/03/26 12:49:03 [notice] 115#115: exiting
2025/03/26 12:49:03 [notice] 117#117: exiting
2025/03/26 12:49:03 [notice] 116#116: exiting
2025/03/26 12:49:03 [notice] 118#118: gracefully shutting down
2025/03/26 12:49:03 [notice] 120#120: gracefully shutting down
2025/03/26 12:49:03 [notice] 119#119: gracefully shutting down
2025/03/26 12:49:03 [notice] 118#118: exiting
2025/03/26 12:49:03 [notice] 120#120: exiting
2025/03/26 12:49:03 [notice] 123#123: gracefully shutting down
2025/03/26 12:49:03 [notice] 122#122: gracefully shutting down
2025/03/26 12:49:03 [notice] 121#121: gracefully shutting down
2025/03/26 12:49:03 [notice] 119#119: exiting
2025/03/26 12:49:03 [notice] 123#123: exiting
2025/03/26 12:49:03 [notice] 126#126: gracefully shutting down
2025/03/26 12:49:03 [notice] 122#122: exiting
2025/03/26 12:49:03 [notice] 127#127: gracefully shutting down
2025/03/26 12:49:03 [notice] 121#121: exiting
2025/03/26 12:49:03 [notice] 125#125: gracefully shutting down
2025/03/26 12:49:03 [notice] 128#128: gracefully shutting down
2025/03/26 12:49:03 [notice] 126#126: exiting
2025/03/26 12:49:03 [notice] 129#129: gracefully shutting down
2025/03/26 12:49:03 [notice] 132#132: gracefully shutting down
2025/03/26 12:49:03 [notice] 127#127: exiting
2025/03/26 12:49:03 [notice] 124#124: gracefully shutting down
2025/03/26 12:49:03 [notice] 125#125: exiting
2025/03/26 12:49:03 [notice] 45#45: signal 17 (SIGCHLD) received from 52
2025/03/26 12:49:03 [notice] 129#129: exiting
2025/03/26 12:49:03 [notice] 132#132: exiting
2025/03/26 12:49:03 [notice] 128#128: exiting
2025/03/26 12:49:03 [notice] 124#124: exiting
2025/03/26 12:49:03 [notice] 134#134: gracefully shutting down
2025/03/26 12:49:03 [notice] 45#45: worker process 52 exited with code 0
2025/03/26 12:49:03 [notice] 134#134: exiting
2025/03/26 12:49:03 [notice] 136#136: gracefully shutting down
2025/03/26 12:49:03 [notice] 136#136: exiting
2025/03/26 12:49:03 [notice] 45#45: worker process 53 exited with code 0
2025/03/26 12:49:03 [notice] 131#131: gracefully shutting down
2025/03/26 12:49:03 [notice] 131#131: exiting
2025/03/26 12:49:03 [notice] 45#45: worker process 54 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 55 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 56 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 57 exited with code 0
2025/03/26 12:49:03 [notice] 130#130: gracefully shutting down
2025/03/26 12:49:03 [notice] 45#45: worker process 58 exited with code 0
2025/03/26 12:49:03 [notice] 130#130: exiting
2025/03/26 12:49:03 [notice] 45#45: worker process 59 exited with code 0
2025/03/26 12:49:03 [notice] 133#133: gracefully shutting down
2025/03/26 12:49:03 [notice] 45#45: worker process 60 exited with code 0
2025/03/26 12:49:03 [notice] 133#133: exiting
2025/03/26 12:49:03 [notice] 45#45: worker process 61 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 62 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 63 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 64 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 65 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 66 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 67 exited with code 0
2025/03/26 12:49:03 [notice] 135#135: gracefully shutting down
2025/03/26 12:49:03 [notice] 135#135: exiting
2025/03/26 12:49:03 [notice] 45#45: worker process 68 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 69 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 70 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 71 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 72 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 73 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 74 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 75 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 76 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 77 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 78 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 79 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 80 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 81 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 82 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: worker process 83 exited with code 0
2025/03/26 12:49:03 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 134#134: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 134
2025/03/26 12:49:30 [notice] 45#45: worker process 134 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 122#122: exit
2025/03/26 12:49:03 [notice] 120#120: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 122
2025/03/26 12:49:30 [notice] 45#45: worker process 122 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 120
2025/03/26 12:49:30 [notice] 45#45: worker process 120 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 115#115: exit
2025/03/26 12:49:03 [notice] 117#117: exit
2025/03/26 12:49:03 [notice] 116#116: exit
2025/03/26 12:49:03 [notice] 109#109: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 115
2025/03/26 12:49:30 [notice] 45#45: worker process 115 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 110#110: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 116
2025/03/26 12:49:30 [notice] 45#45: worker process 116 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 117
2025/03/26 12:49:30 [notice] 45#45: worker process 117 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 109
2025/03/26 12:49:30 [notice] 45#45: worker process 109 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 126#126: exit
2025/03/26 12:49:03 [notice] 131#131: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 110
2025/03/26 12:49:30 [notice] 45#45: worker process 110 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 111#111: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 126
2025/03/26 12:49:30 [notice] 45#45: worker process 126 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 112#112: exit
2025/03/26 12:49:03 [notice] 114#114: exit
2025/03/26 12:49:03 [notice] 119#119: exit
2025/03/26 12:49:03 [notice] 108#108: exit
2025/03/26 12:49:03 [notice] 107#107: exit
2025/03/26 12:49:03 [notice] 124#124: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 131
2025/03/26 12:49:30 [notice] 45#45: worker process 131 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 111
2025/03/26 12:49:30 [notice] 45#45: worker process 111 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 123#123: exit
2025/03/26 12:49:03 [notice] 128#128: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 112
2025/03/26 12:49:30 [notice] 45#45: worker process 112 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 114
2025/03/26 12:49:30 [notice] 45#45: worker process 114 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 119
2025/03/26 12:49:30 [notice] 45#45: worker process 119 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 125#125: exit
2025/03/26 12:49:03 [notice] 136#136: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 108
2025/03/26 12:49:30 [notice] 45#45: worker process 108 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 113#113: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 107
2025/03/26 12:49:30 [notice] 45#45: worker process 107 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: worker process 123 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 123
2025/03/26 12:49:03 [notice] 135#135: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 124
2025/03/26 12:49:30 [notice] 45#45: worker process 124 exited with code 0
2025/03/26 12:49:03 [notice] 133#133: exit
2025/03/26 12:49:03 [notice] 127#127: exit
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 128
2025/03/26 12:49:30 [notice] 45#45: worker process 128 exited with code 0
2025/03/26 12:49:03 [notice] 132#132: exit
2025/03/26 12:49:03 [notice] 105#105: exit
2025/03/26 12:49:03 [notice] 129#129: exit
2025/03/26 12:49:03 [notice] 118#118: exit
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 125
2025/03/26 12:49:30 [notice] 45#45: worker process 125 exited with code 0
2025/03/26 12:49:03 [notice] 121#121: exit
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:03 [notice] 106#106: exit
2025/03/26 12:49:03 [notice] 130#130: exit
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 136
2025/03/26 12:49:30 [notice] 45#45: worker process 136 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 113
2025/03/26 12:49:30 [notice] 45#45: worker process 113 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 105
2025/03/26 12:49:30 [notice] 45#45: worker process 105 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: worker process 133 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 133
2025/03/26 12:49:30 [notice] 45#45: worker process 135 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 106
2025/03/26 12:49:30 [notice] 45#45: worker process 106 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 129
2025/03/26 12:49:30 [notice] 45#45: worker process 129 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: worker process 118 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: worker process 127 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: worker process 132 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 127
2025/03/26 12:49:30 [notice] 45#45: worker process 121 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received
2025/03/26 12:49:30 [notice] 45#45: signal 17 (SIGCHLD) received from 130
2025/03/26 12:49:30 [notice] 45#45: worker process 130 exited with code 0
2025/03/26 12:49:30 [notice] 45#45: signal 29 (SIGIO) received

• NGINX Configuration: kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T
  nginx.conf.txt

Additional context
HTTPRoute.yaml

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: ${NAMESPACE}-route
  namespace: ${NAMESPACE}
spec:
  parentRefs:
    - name: wandb-gateway
      namespace: gateway-system
  hostnames:
    - "${NAMESPACE}.wandb.dev"
    - "${NAMESPACE}-api.wandb.dev"
    - "${NAMESPACE}-fixtures.wandb.dev"
    - "${NAMESPACE}-minio.wandb.dev"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
          headers:
            - name: "Host"
              value: "${NAMESPACE}.wandb.dev"
      backendRefs:
        - group: ""
          kind: Service
          name: frontend-x-frontend-x-vcluster
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /
          headers:
            - name: "Host"
              value: "${NAMESPACE}-api.wandb.dev"
      backendRefs:
        - group: ""
          kind: Service
          name: gorilla-api-x-gorilla-x-vcluster
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /
          headers:
            - name: "Host"
              value: "${NAMESPACE}-fixtures.wandb.dev"
      backendRefs:
        - group: ""
          kind: Service
          name: fixtures-x-gorilla-x-vcluster
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /
          headers:
            - name: "Host"
              value: "${NAMESPACE}-minio.wandb.dev"
      backendRefs:
        - group: ""
          kind: Service
          name: minio-x-minio-x-vcluster
          port: 9000

[nginx-bot]

Hi @wandbjake! Welcome to the project! 🎉

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[sjberman]

Hi @wandbjake, we do have scale testing that verifies that everything works properly up to 1000 HTTPRoute objects. We also verify scaling up to about 650 Services. Although I don't think we verify both of those things at the same time, which it looks like the situation you are in. Looks like you end up with about 700 server blocks and upstreams in the provided nginx.conf.

Looking at the memory usage, that's very high and I suspect that's why it fails to continue at a certain point. Though we haven't seen the memory get that high in any of our scale tests (never over 50 MiB). (If you're curious, we store those results here).

Splitting up into multiple Gateways would definitely alleviate the pressure on the single nginx instance. Right now that means installing NGINX Gateway Fabric multiple times, but in our upcoming 2.0 release we will support multiple Gateways with a single installation of the NGINX Gateway Fabric control plane.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.