NGINX Plus fails to route traffic when docker container abruptly restarts

[bjee19]

Describe the bug
When I have NGF deployed with NGINX Plus and a simple cafe application, if I abruptly restart the docker container through docker restart <name of node> on Kind, all the applications successfully restart and are ready, but NGINX fails to route my traffic.

To Reproduce
Steps to reproduce the behavior:

1. Deploy NGF with NGINX Plus and cafe applications -- verify traffic flows when port-forwarding
2. Run docker restart <name of node>
3. After applications are all ready, restart port-forward, and try to curl cafe applications.

Expected behavior
NGINX successfully routes traffic to my applications.

Your environment

• Version of the NGINX Gateway Fabric - release version or a specific commit. The first line of the nginx-gateway container logs includes the commit info. - edge on change/control-data-plane-split
• Version of Kubernetes: v1.32.2
• Kubernetes platform (e.g. Mini-kube or GCP): Kind
• Details on how you expose the NGINX Gateway Fabric Pod (e.g. Service of type LoadBalancer or port-forward): port-forward
• Logs of NGINX container: kubectl -n nginx-gateway logs -l app=nginx-gateway -c nginx

Defaulted container "nginx" out of: nginx, init (init)
+ trap handle_term TERM
+ rm -rf /var/run/nginx/connection-closed-server.sock /var/run/nginx/nginx-500-server.sock /var/run/nginx/nginx-503-server.sock /var/run/nginx/nginx-plus-api.sock
+ echo 'starting nginx ...'
+ '[' false = debug ']'
starting nginx ...
+ nginx_pid=12
+ SECONDS=0
+ /usr/sbin/nginx -g 'daemon off;'
+ ps -ef
+ grep -v grep
+ grep 'nginx: master process'
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ grep 'nginx: master process'
+ ps -ef
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
2025/03/19 23:02:24 [notice] 12#12: js vm init njs: 0000FFFFA1BC3A00
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ + grep grep -v 'nginx: master process'grep

+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
+ (( SECONDS > 5 ))
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
2025/03/19 23:02:24 [notice] 12#12: using the "epoll" event method
2025/03/19 23:02:24 [notice] 12#12: nginx/1.27.2 (nginx-plus-r33-p2)
2025/03/19 23:02:24 [notice] 12#12: built by gcc 14.2.0 (Alpine 14.2.0) 
2025/03/19 23:02:24 [notice] 12#12: OS: Linux 6.10.14-linuxkit
2025/03/19 23:02:24 [notice] 12#12: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2025/03/19 23:02:24 [notice] 12#12: start worker processes
2025/03/19 23:02:24 [notice] 12#12: start worker process 61
2025/03/19 23:02:24 [notice] 12#12: start worker process 62
2025/03/19 23:02:24 [notice] 12#12: start worker process 63
+ (( SECONDS > 5 ))
2025/03/19 23:02:24 [notice] 12#12: start worker process 64
+ ps -ef
+ grep 'nginx: master process'
+ grep -v grep
2025/03/19 23:02:24 [notice] 12#12: start worker process 68
2025/03/19 23:02:24 [notice] 12#12: start worker process 69
2025/03/19 23:02:24 [notice] 12#12: start worker process 70
2025/03/19 23:02:24 [notice] 12#12: start worker process 71
2025/03/19 23:02:24 [notice] 12#12: start worker process 72
2025/03/19 23:02:24 [notice] 12#12: start worker process 73
   12 nginx     0:00 nginx: master process /usr/sbin/nginx -g daemon off;
   70 nginx     0:00 nginx: master process /usr/sbin/nginx -g daemon off;
starting nginx-agent ...
2025/03/19 23:02:24 [notice] 12#12: start worker process 74
2025/03/19 23:02:24 [notice] 12#12: start worker process 75
+ echo 'starting nginx-agent ...'
+ agent_pid=76
+ '[' 0 '!=' 0 ']'
+ wait_term
+ nginx-agent
+ wait 76
2025/03/19 23:02:24 INFO Configured labels labels=map[]
2025/03/19 23:02:24 INFO Enabled features features="[connection configuration certificates metrics api-action]"
2025/03/19 23:02:24 INFO Excluded files from being watched for file changes exclude_files=[^.*(\.log|.swx|~|.swp)$]
time=2025-03-19T23:02:24.222Z level=INFO msg="Starting NGINX Agent" version=v3.0.0 commit=e79d42d6
time=2025-03-19T23:02:24.222Z level=INFO msg="Dialing grpc server" server_addr=my-release-nginx-gateway-fabric.nginx-gateway.svc:443
time=2025-03-19T23:02:24.223Z level=INFO msg="Finished registering plugins" plugins="[resource command file collector watcher]"
time=2025-03-19T23:02:24.223Z level=INFO msg="Starting OTel Collector plugin"
time=2025-03-19T23:02:24.223Z level=INFO msg="No receivers configured for OTel Collector. Waiting to discover a receiver before starting OTel collector."
time=2025-03-19T23:02:29.237Z level=WARN msg="Currently error log outputs to stderr. Log monitoring is disabled while applying a config; log errors to file to enable error monitoring" error_log=stderr correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:29.244Z level=INFO msg="Reloading OTel collector config" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:29.245Z level=INFO msg="Closing OTel Collector plugin" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:29.245Z level=INFO msg="Starting OTel collector" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.228Z level=ERROR msg="Unable to update data plane health" error="command service client not connected yet" correlation_id=3e5e41e5-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.251Z level=INFO msg="Reloading OTel collector config" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.251Z level=INFO msg="Closing OTel Collector plugin" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.251Z level=INFO msg="Shutting down OTel Collector" state=Running correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.252Z level=INFO msg="OTel collector run finished" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.792Z level=INFO msg="OTel Collector shutdown" state=Closed correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:34.792Z level=INFO msg="Starting OTel collector" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:39.257Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup my-release-nginx-gateway-fabric.nginx-gateway.svc on 10.96.0.10:53: read udp 10.244.0.4:54387->10.96.0.10:53: read: connection refused" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:39.540Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup my-release-nginx-gateway-fabric.nginx-gateway.svc on 10.96.0.10:53: read udp 10.244.0.4:54387->10.96.0.10:53: read: connection refused" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:39.949Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup my-release-nginx-gateway-fabric.nginx-gateway.svc on 10.96.0.10:53: read udp 10.244.0.4:54387->10.96.0.10:53: read: connection refused" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:40.707Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup my-release-nginx-gateway-fabric.nginx-gateway.svc on 10.96.0.10:53: read udp 10.244.0.4:54387->10.96.0.10:53: read: connection refused" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:41.965Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup my-release-nginx-gateway-fabric.nginx-gateway.svc on 10.96.0.10:53: read udp 10.244.0.4:54387->10.96.0.10:53: read: connection refused" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:43.709Z level=ERROR msg="Failed to create connection" error="rpc error: code = Unavailable desc = dns: A record lookup error: lookup my-release-nginx-gateway-fabric.nginx-gateway.svc on 10.96.0.10:53: read udp 10.244.0.4:54387->10.96.0.10:53: read: connection refused" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
2025/03/19 23:02:45 [error] 61#61: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.2:8080/coffee", host: "cafe.example.com:8080"
127.0.0.1 - - [19/Mar/2025:23:02:45 +0000] "GET /coffee HTTP/1.1" 502 150 "-" "curl/8.7.1"
2025/03/19 23:02:45 [info] 73#73: *5 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 72#72: *4 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 73#73: *10 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 73#73: *11 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 63#63: *17 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 64#64: *12 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 62#62: *16 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 61#61: *13 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 61#61: *14 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 70#70: *15 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 72#72: *6 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 72#72: *7 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 68#68: *18 client unix: closed keepalive connection
2025/03/19 23:02:45 [info] 73#73: *9 client unix: closed keepalive connection
2025/03/19 23:02:46 [info] 61#61: *2 client 127.0.0.1 closed keepalive connection
2025/03/19 23:02:46 [error] 62#62: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.2:8080/coffee", host: "cafe.example.com:8080"
127.0.0.1 - - [19/Mar/2025:23:02:46 +0000] "GET /coffee HTTP/1.1" 502 150 "-" "curl/8.7.1"
time=2025-03-19T23:02:47.556Z level=INFO msg="Connection created" response=response:{status:COMMAND_STATUS_OK} correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:47.556Z level=INFO msg="Agent connected" correlation_id=3b6443ed-0516-11f0-874d-9eae44764613
2025/03/19 23:02:47 [info] 62#62: *20 client 127.0.0.1 closed keepalive connection
time=2025-03-19T23:02:48.115Z level=INFO msg="Updating file overview" instance_id=e8d1bda6-397e-3b98-a179-e500ff99fbc7 parent_correlation_id=3b6443ed-0516-11f0-874d-9eae44764613 correlation_id=3b660ba2-0516-11f0-874d-9eae44764613
time=2025-03-19T23:02:48.582Z level=INFO msg="NGINX config tested" output="2025/03/19 23:02:48 [notice] 99#99: js vm init njs: 0000FFFFA071AA00\nnginx: the configuration file /etc/nginx/nginx.conf syntax is ok\nnginx: configuration file /etc/nginx/nginx.conf test is successful\n" correlation_id=39261d88-2438-46ec-9ed4-fd727108018a
time=2025-03-19T23:02:48.582Z level=INFO msg="Reloading NGINX PID" pid=12 correlation_id=39261d88-2438-46ec-9ed4-fd727108018a
time=2025-03-19T23:02:48.582Z level=INFO msg="NGINX reloaded" processid=12 correlation_id=39261d88-2438-46ec-9ed4-fd727108018a
time=2025-03-19T23:02:48.582Z level=INFO msg="Finished monitoring post reload" correlation_id=39261d88-2438-46ec-9ed4-fd727108018a
2025/03/19 23:02:48 [notice] 12#12: signal 1 (SIGHUP) received from 76, reconfiguring
2025/03/19 23:02:48 [notice] 12#12: reconfiguring
time=2025-03-19T23:02:48.582Z level=WARN msg="Currently error log outputs to stderr. Log monitoring is disabled while applying a config; log errors to file to enable error monitoring" error_log=stderr correlation_id=39261d88-2438-46ec-9ed4-fd727108018a
time=2025-03-19T23:02:48.583Z level=INFO msg="No NGINX error logs found to monitor" correlation_id=39261d88-2438-46ec-9ed4-fd727108018a
time=2025-03-19T23:02:48.586Z level=INFO msg=location ""=unix:/var/run/nginx/nginx-plus-api.sock
2025/03/19 23:02:48 [notice] 12#12: js vm init njs: 0000FFFFA133D800
2025/03/19 23:02:48 [notice] 12#12: using the "epoll" event method
2025/03/19 23:02:48 [notice] 12#12: start worker processes
2025/03/19 23:02:48 [notice] 12#12: start worker process 102
2025/03/19 23:02:48 [notice] 12#12: start worker process 103
2025/03/19 23:02:48 [notice] 12#12: start worker process 104
2025/03/19 23:02:48 [notice] 12#12: start worker process 105
2025/03/19 23:02:48 [notice] 12#12: start worker process 106
2025/03/19 23:02:48 [info] 71#71: *22 client unix: closed keepalive connection
2025/03/19 23:02:48 [notice] 12#12: start worker process 107
2025/03/19 23:02:48 [notice] 12#12: start worker process 108
time=2025-03-19T23:02:48.593Z level=INFO msg=location ""=unix:/var/run/nginx/nginx-plus-api.sock
2025/03/19 23:02:48 [notice] 12#12: start worker process 109
2025/03/19 23:02:48 [notice] 12#12: start worker process 110
2025/03/19 23:02:48 [notice] 12#12: start worker process 111
2025/03/19 23:02:48 [notice] 12#12: start worker process 112
2025/03/19 23:02:48 [notice] 12#12: start worker process 113
2025/03/19 23:02:48 [info] 74#74: *23 client unix: closed keepalive connection
time=2025-03-19T23:02:48.595Z level=INFO msg="Updating file overview" instance_id=e8d1bda6-397e-3b98-a179-e500ff99fbc7 parent_correlation_id=39261d88-2438-46ec-9ed4-fd727108018a correlation_id=46eeab35-0516-11f0-874d-9eae44764613
2025/03/19 23:02:48 [info] 61#61: *25 client unix: closed keepalive connection
2025/03/19 23:02:48 [info] 61#61: *26 client unix: closed keepalive connection
2025/03/19 23:02:48 [notice] 63#63: gracefully shutting down
2025/03/19 23:02:48 [notice] 61#61: gracefully shutting down
2025/03/19 23:02:48 [notice] 69#69: gracefully shutting down
2025/03/19 23:02:48 [notice] 74#74: gracefully shutting down
2025/03/19 23:02:48 [notice] 68#68: gracefully shutting down
2025/03/19 23:02:48 [notice] 64#64: gracefully shutting down
2025/03/19 23:02:48 [notice] 71#71: gracefully shutting down
2025/03/19 23:02:48 [notice] 63#63: exiting
2025/03/19 23:02:48 [notice] 62#62: gracefully shutting down
2025/03/19 23:02:48 [notice] 74#74: exiting
2025/03/19 23:02:48 [notice] 64#64: exiting
2025/03/19 23:02:48 [notice] 69#69: exiting
2025/03/19 23:02:48 [notice] 61#61: exiting
2025/03/19 23:02:48 [notice] 71#71: exiting
2025/03/19 23:02:48 [notice] 68#68: exiting
2025/03/19 23:02:48 [notice] 70#70: gracefully shutting down
2025/03/19 23:02:48 [notice] 70#70: exiting
2025/03/19 23:02:48 [notice] 62#62: exiting
2025/03/19 23:02:48 [notice] 72#72: gracefully shutting down
2025/03/19 23:02:48 [notice] 72#72: exiting
2025/03/19 23:02:48 [notice] 73#73: gracefully shutting down
2025/03/19 23:02:48 [notice] 73#73: exiting
2025/03/19 23:02:48 [notice] 75#75: gracefully shutting down
2025/03/19 23:02:48 [notice] 74#74: exit
2025/03/19 23:02:48 [notice] 61#61: exit
2025/03/19 23:02:48 [notice] 68#68: exit
2025/03/19 23:02:48 [notice] 69#69: exit
2025/03/19 23:02:48 [notice] 64#64: exit
2025/03/19 23:02:48 [notice] 71#71: exit
2025/03/19 23:02:48 [notice] 75#75: exiting
2025/03/19 23:02:48 [notice] 72#72: exit
2025/03/19 23:02:48 [notice] 63#63: exit
2025/03/19 23:02:48 [notice] 70#70: exit
2025/03/19 23:02:48 [notice] 62#62: exit
2025/03/19 23:02:48 [notice] 75#75: exit
2025/03/19 23:02:48 [notice] 73#73: exit
2025/03/19 23:02:48 [notice] 12#12: signal 17 (SIGCHLD) received from 74
2025/03/19 23:02:48 [notice] 12#12: worker process 70 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 72 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 74 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 75 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: signal 29 (SIGIO) received
2025/03/19 23:02:48 [notice] 12#12: signal 17 (SIGCHLD) received from 75
2025/03/19 23:02:48 [notice] 12#12: signal 17 (SIGCHLD) received from 71
2025/03/19 23:02:48 [notice] 12#12: worker process 71 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 73 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: signal 29 (SIGIO) received
2025/03/19 23:02:48 [notice] 12#12: signal 17 (SIGCHLD) received from 73
2025/03/19 23:02:48 [notice] 12#12: worker process 63 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: signal 29 (SIGIO) received
2025/03/19 23:02:48 [notice] 12#12: signal 17 (SIGCHLD) received from 68
2025/03/19 23:02:48 [notice] 12#12: worker process 61 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 62 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 64 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: worker process 68 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: signal 29 (SIGIO) received
2025/03/19 23:02:48 [notice] 12#12: signal 17 (SIGCHLD) received from 69
2025/03/19 23:02:48 [notice] 12#12: worker process 69 exited with code 0
2025/03/19 23:02:48 [notice] 12#12: signal 29 (SIGIO) received
time=2025-03-19T23:02:49.229Z level=WARN msg="Currently error log outputs to stderr. Log monitoring is disabled while applying a config; log errors to file to enable error monitoring" error_log=stderr correlation_id=474ed4a8-0516-11f0-874d-9eae44764613
2025/03/19 23:02:49 [error] 102#102: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.2:8080/coffee", host: "cafe.example.com:8080"
127.0.0.1 - - [19/Mar/2025:23:02:49 +0000] "GET /coffee HTTP/1.1" 502 150 "-" "curl/8.7.1"
2025/03/19 23:02:50 [info] 102#102: *28 client 127.0.0.1 closed keepalive connection
2025/03/19 23:02:55 [info] 112#112: *42 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 112#112: *41 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 103#103: *43 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *33 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *35 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *39 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *30 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *37 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *36 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *34 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *32 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 107#107: *31 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 105#105: *44 client unix: closed keepalive connection
2025/03/19 23:02:55 [info] 105#105: *45 client unix: closed keepalive connection
127.0.0.1 - - [19/Mar/2025:23:03:01 +0000] "GET /tea HTTP/2.0" 404 19 "-" "curl/8.7.1"
127.0.0.1 - - [19/Mar/2025:23:03:03 +0000] "GET /tea HTTP/2.0" 404 19 "-" "curl/8.7.1"
2025/03/19 23:03:05 [error] 103#103: *50 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.2:8080/coffee", host: "cafe.example.com:8080"
127.0.0.1 - - [19/Mar/2025:23:03:05 +0000] "GET /coffee HTTP/1.1" 502 150 "-" "curl/8.7.1"
2025/03/19 23:03:05 [info] 110#110: *58 client unix: closed keepalive connection

...

2025/03/19 23:04:15 [info] 112#112: *158 client unix: closed keepalive connection
127.0.0.1 - - [19/Mar/2025:23:04:17 +0000] "GET /tea HTTP/2.0" 404 19 "-" "curl/8.7.1"
127.0.0.1 - - [19/Mar/2025:23:04:24 +0000] "GET /coffee HTTP/1.1" 502 150 "-" "curl/8.7.1"
2025/03/19 23:04:24 [error] 104#104: *166 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.2:8080/coffee", host: "cafe.example.com:8080"
...
2025/03/19 23:06:45 [info] 110#110: *368 client unix: closed keepalive connection
2025/03/19 23:06:45 [info] 110#110: *359 client unix: closed keepalive connection
2025/03/19 23:06:48 [error] 105#105: *369 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: cafe.example.com, request: "GET /coffee HTTP/1.1", upstream: "http://10.244.0.2:8080/coffee", host: "cafe.example.com:8080"
127.0.0.1 - - [19/Mar/2025:23:06:48 +0000] "GET /coffee HTTP/1.1" 502 150 "-" "curl/8.7.1"
2025/03/19 23:06:49 [info] 105#105: *369 client 127.0.0.1 closed keepalive connection
127.0.0.1 - - [19/Mar/2025:23:06:50 +0000] "GET /tea HTTP/2.0" 404 19 "-" "curl/8.7.1"
...

• NGINX Configuration: kubectl -n nginx-gateway exec <gateway-pod> -c nginx -- nginx -T

2025/03/18 23:15:50 [notice] 95#95: js vm init njs: 0000FFFF8E977980
	nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
	nginx: configuration file /etc/nginx/nginx.conf test is successful
	# configuration file /etc/nginx/nginx.conf:
	load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
	include /etc/nginx/main-includes/*.conf;
	
	worker_processes auto;
	
	pid /var/run/nginx/nginx.pid;
	
	events {
	  worker_connections 1024;
	}
	
	http {
	  include /etc/nginx/conf.d/*.conf;
	  include /etc/nginx/mime.types;
	  js_import /usr/lib/nginx/modules/njs/httpmatches.js;
	
	  default_type application/octet-stream;
	
	  proxy_headers_hash_bucket_size 512;
	  proxy_headers_hash_max_size 1024;
	  server_names_hash_bucket_size 256;
	  server_names_hash_max_size 1024;
	  variables_hash_bucket_size 512;
	  variables_hash_max_size 1024;
	
	  sendfile on;
	  tcp_nopush on;
	
	  server_tokens off;
	}
	
	stream {
	  variables_hash_bucket_size 512;
	  variables_hash_max_size 1024;
	
	  map_hash_max_size 2048;
	  map_hash_bucket_size 256;
	
	  log_format stream-main '$remote_addr [$time_local] '
	                         '$protocol $status $bytes_sent $bytes_received '
	                         '$session_time "$ssl_preread_server_name"';
	  access_log /dev/stdout stream-main;
	  include /etc/nginx/stream-conf.d/*.conf;
	}
	
	# configuration file /etc/nginx/main-includes/main.conf:
	
	error_log stderr info;
	
	
	# configuration file /etc/nginx/main-includes/mgmt.conf:
	
	mgmt {
	        license_token /etc/nginx/secrets/license.jwt;
	        deployment_context /etc/nginx/main-includes/deployment_ctx.json;
	}
	
	# configuration file /etc/nginx/conf.d/http.conf:
	http2 on;
	
	# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
	# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
	# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
	# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
	map $http_host $gw_api_compliant_host {
	    '' $host;
	    default $http_host;
	}
	
	# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
	# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
	map $http_upgrade $connection_upgrade {
	    default upgrade;
	    '' close;
	}
	
	## Returns just the path from the original request URI.
	map $request_uri $request_uri_path {
	  "~^(?P<path>[^?]*)(\?.*)?$"  $path;
	}
	
	
	js_preload_object matches from /etc/nginx/conf.d/matches.json;
	server {
	    listen 80 default_server;
	    listen [::]:80 default_server;
	    default_type text/html;
	    return 404;
	}
	
	server {
	    listen 80;
	    listen [::]:80;
	
	    server_name cafe.example.com;
	    status_zone cafe.example.com;
	
	        
	    location /coffee/ {
	        
	
	        
	
	        proxy_http_version 1.1;
	        proxy_set_header Host "$gw_api_compliant_host";
	        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
	        proxy_set_header X-Real-IP "$remote_addr";
	        proxy_set_header X-Forwarded-Proto "$scheme";
	        proxy_set_header X-Forwarded-Host "$host";
	        proxy_set_header X-Forwarded-Port "$server_port";
	        proxy_set_header Upgrade "$http_upgrade";
	        proxy_set_header Connection "$connection_upgrade";
	        proxy_pass http://default_coffee_80$request_uri/;
	            
	            
	            
	    }
	    location = /coffee {
	        
	
	        
	
	        proxy_http_version 1.1;
	        proxy_set_header Host "$gw_api_compliant_host";
	        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
	        proxy_set_header X-Real-IP "$remote_addr";
	        proxy_set_header X-Forwarded-Proto "$scheme";
	        proxy_set_header X-Forwarded-Host "$host";
	        proxy_set_header X-Forwarded-Port "$server_port";
	        proxy_set_header Upgrade "$http_upgrade";
	        proxy_set_header Connection "$connection_upgrade";
	        proxy_pass http://default_coffee_80$request_uri/;
	            
	            
	            
	    }
	    location / {
	        
	        return 404 "";
	
	        
	
	        proxy_http_version 1.1;
	    }
	}
	server {
	    listen 443 ssl default_server;
	    listen [::]:443 ssl default_server;
	    ssl_reject_handshake on;
	}
	
	server {
	    listen 443 ssl;
	    listen [::]:443 ssl;
	    ssl_certificate /etc/nginx/secrets/ssl_keypair_default_cafe-secret.pem;
	    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_default_cafe-secret.pem;
	
	    if ($ssl_server_name != $host) {
	        return 421;
	    }
	
	    server_name cafe.example.com;
	    status_zone cafe.example.com;
	
	        
	    location /tea/ {
	        
	
	        
	
	        proxy_http_version 1.1;
	        proxy_set_header Host "$gw_api_compliant_host";
	        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
	        proxy_set_header X-Real-IP "$remote_addr";
	        proxy_set_header X-Forwarded-Proto "$scheme";
	        proxy_set_header X-Forwarded-Host "$host";
	        proxy_set_header X-Forwarded-Port "$server_port";
	        proxy_set_header Upgrade "$http_upgrade";
	        proxy_set_header Connection "$connection_upgrade";
	        proxy_pass http://default_tea_80$request_uri/;
	            
	            
	            
	    }
	    location = /tea {
	        
	
	        
	
	        proxy_http_version 1.1;
	        proxy_set_header Host "$gw_api_compliant_host";
	        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
	        proxy_set_header X-Real-IP "$remote_addr";
	        proxy_set_header X-Forwarded-Proto "$scheme";
	        proxy_set_header X-Forwarded-Host "$host";
	        proxy_set_header X-Forwarded-Port "$server_port";
	        proxy_set_header Upgrade "$http_upgrade";
	        proxy_set_header Connection "$connection_upgrade";
	        proxy_pass http://default_tea_80$request_uri/;
	            
	            
	            
	    }
	    location / {
	        
	        return 404 "";
	
	        
	
	        proxy_http_version 1.1;
	    }
	}
	
	server {
	    listen unix:/var/run/nginx/nginx-503-server.sock;
	    access_log off;
	
	    return 503;
	}
	
	server {
	    listen unix:/var/run/nginx/nginx-500-server.sock;
	    access_log off;
	
	    return 500;
	}
	
	
	upstream default_coffee_80 {
	    random two least_conn;
	    zone default_coffee_80 1m;
	    
	    state /var/lib/nginx/state/default_coffee_80.conf;
	    
	    
	    
	    
	}
	
	upstream default_tea_80 {
	    random two least_conn;
	    zone default_tea_80 1m;
	    
	    state /var/lib/nginx/state/default_tea_80.conf;
	    
	    
	    
	    
	}
	
	upstream invalid-backend-ref {
	    random two least_conn;
	    
	        
	    server unix:/var/run/nginx/nginx-500-server.sock;
	    
	    
	    
	    
	}
	
	
	
	
	
	# configuration file /var/lib/nginx/state/default_coffee_80.conf:
	server 10.244.0.6:8080;
	
	# configuration file /var/lib/nginx/state/default_tea_80.conf:
	server 10.244.0.5:8080;
	
	# configuration file /etc/nginx/conf.d/plus-api.conf:
	
	server {
	    listen unix:/var/run/nginx/nginx-plus-api.sock;
	    access_log off;
	
	    location /api {
	      api write=on;
	    }
	}
	
	server {
	    listen 8765;
	    root /usr/share/nginx/html;
	    access_log off;
	    
	    allow 127.0.0.1;
	    deny all;
	
	    location = /dashboard.html {}
	
	    location /api {
	      api write=off;
	    }
	}
	
	# configuration file /etc/nginx/mime.types:
	
	types {
	    text/html                                        html htm shtml;
	    text/css                                         css;
	    text/xml                                         xml;
	    image/gif                                        gif;
	    image/jpeg                                       jpeg jpg;
	    application/javascript                           js;
	    application/atom+xml                             atom;
	    application/rss+xml                              rss;
	
	    text/mathml                                      mml;
	    text/plain                                       txt;
	    text/vnd.sun.j2me.app-descriptor                 jad;
	    text/vnd.wap.wml                                 wml;
	    text/x-component                                 htc;
	
	    image/avif                                       avif;
	    image/png                                        png;
	    image/svg+xml                                    svg svgz;
	    image/tiff                                       tif tiff;
	    image/vnd.wap.wbmp                               wbmp;
	    image/webp                                       webp;
	    image/x-icon                                     ico;
	    image/x-jng                                      jng;
	    image/x-ms-bmp                                   bmp;
	
	    font/woff                                        woff;
	    font/woff2                                       woff2;
	
	    application/java-archive                         jar war ear;
	    application/json                                 json;
	    application/mac-binhex40                         hqx;
	    application/msword                               doc;
	    application/pdf                                  pdf;
	    application/postscript                           ps eps ai;
	    application/rtf                                  rtf;
	    application/vnd.apple.mpegurl                    m3u8;
	    application/vnd.google-earth.kml+xml             kml;
	    application/vnd.google-earth.kmz                 kmz;
	    application/vnd.ms-excel                         xls;
	    application/vnd.ms-fontobject                    eot;
	    application/vnd.ms-powerpoint                    ppt;
	    application/vnd.oasis.opendocument.graphics      odg;
	    application/vnd.oasis.opendocument.presentation  odp;
	    application/vnd.oasis.opendocument.spreadsheet   ods;
	    application/vnd.oasis.opendocument.text          odt;
	    application/vnd.openxmlformats-officedocument.presentationml.presentation
	                                                     pptx;
	    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
	                                                     xlsx;
	    application/vnd.openxmlformats-officedocument.wordprocessingml.document
	                                                     docx;
	    application/vnd.wap.wmlc                         wmlc;
	    application/wasm                                 wasm;
	    application/x-7z-compressed                      7z;
	    application/x-cocoa                              cco;
	    application/x-java-archive-diff                  jardiff;
	    application/x-java-jnlp-file                     jnlp;
	    application/x-makeself                           run;
	    application/x-perl                               pl pm;
	    application/x-pilot                              prc pdb;
	    application/x-rar-compressed                     rar;
	    application/x-redhat-package-manager             rpm;
	    application/x-sea                                sea;
	    application/x-shockwave-flash                    swf;
	    application/x-stuffit                            sit;
	    application/x-tcl                                tcl tk;
	    application/x-x509-ca-cert                       der pem crt;
	    application/x-xpinstall                          xpi;
	    application/xhtml+xml                            xhtml;
	    application/xspf+xml                             xspf;
	    application/zip                                  zip;
	
	    application/octet-stream                         bin exe dll;
	    application/octet-stream                         deb;
	    application/octet-stream                         dmg;
	    application/octet-stream                         iso img;
	    application/octet-stream                         msi msp msm;
	
	    audio/midi                                       mid midi kar;
	    audio/mpeg                                       mp3;
	    audio/ogg                                        ogg;
	    audio/x-m4a                                      m4a;
	    audio/x-realaudio                                ra;
	
	    video/3gpp                                       3gpp 3gp;
	    video/mp2t                                       ts;
	    video/mp4                                        mp4;
	    video/mpeg                                       mpeg mpg;
	    video/quicktime                                  mov;
	    video/webm                                       webm;
	    video/x-flv                                      flv;
	    video/x-m4v                                      m4v;
	    video/x-mng                                      mng;
	    video/x-ms-asf                                   asx asf;
	    video/x-ms-wmv                                   wmv;
	    video/x-msvideo                                  avi;
	}
	
	# configuration file /etc/nginx/stream-conf.d/stream.conf:
	
	
	server {
	    listen unix:/var/run/nginx/connection-closed-server.sock;
	    return "";
	}

Additional context

When I built the nginx image with curl installed, I could successfully kubectl exec into the nginx container and curl to the service endpoints of my cafe applications. But when I tried to curl from my machine I was unable to.

The problem also seems to shift from 404 errors to 502 errors to 504 errors. I also once replicated this error but one of the applications was working, the tea application failed with 502 errors, but the coffee application successfully sent back 200 response codes.

Results can be seen when running the graceful-recovery test.

[mpstefan]

Putting this on hold until someone runs into this and has a valid use case. Comment here if you run into this!