[Python] Graceful shutdown immediatly close client connection

[RomainMou]

Hi,

I'm still trying to replace some Gunicorn with Unit, but I have issues when I try to shut down Unit gracefully. I expect it to properly handle the current requests before shutting down.

So, I have Unit running into a container, I made a request with curl, and during the process I stop the container with: sudo docker stop -t 60 -s 9 django_unit.

The process continues, but the socket is closed. So the curl gets an error with an empty reply:

romain@grace-hopper:~ curl http://localhost:8000/test/timeout
curl: (52) Empty reply from serve

In the container output, I see that the process continues, but the server can't respond to the client at the end:

0
1
2
2023/05/11 14:29:09 [notice] 1#1 process 44 exited with code 0
2023/05/11 14:29:09 [notice] 1#1 process 45 exited with code 0
3
4
2023/05/11 14:29:23 [warn] 47#47 [unit] sendmsg(11, 311) failed: Broken pipe (32)
2023/05/11 14:29:23 [notice] 46#46 app process 47 exited with code 0
2023/05/11 14:29:23 [alert] 46#46 sendmsg(13, -1, -1, 2) failed (32: Broken pipe)
2023/05/11 14:29:23 [notice] 1#1 process 46 exited with code 0

Did I miss some configuration?
I put all the necessary stuff to reproduce this here https://github.com/RomainMou/unit-django-issue-headers#timeout.

[TokuiNico]

I also encountered the same issue. Does anyone have any ideas?

[TokuiNico]

I wrote a script that checks the current number of connections every second and ends when the connection count is zero.
I set it as a preStop probe for k8s, solving the connection problem when the server shuts down.

#!/bin/bash
while true; do
    connection_count=$(curl --unix-socket /var/run/control.unit.sock http://localhost/status/connections/active 2>/dev/null | grep -oP '\d+')
    echo "ACTIVE: $connection_count"
    if [ "$connection_count" -eq 0 ]; then
        echo "CLOSE"
        break
    fi
    sleep 1
done

[tippexs]

Thanks @TokuiNico for sharing this!
The root cause reported by @RomainMou needs still some investigation on our side. I would like to test this insight and outside the container. I will give this issue another push to see whats the general shutdown behaviour for Unit.

[RomainMou]

I've tried to understand what happens, I'm not really sure if I understand how all processes communicate, but I think something like this happens:

• The main process receives the termination signal.
• It calls nxt_main_process_sigterm_handler, which calls nxt_process_quit, which calls nxt_runtime_quit, which calls nxt_runtime_stop_all_processes, which sends NXT_PORT_MSG_QUIT to all processes.
• One of the processes receiving NXT_PORT_MSG_QUIT is the nxt_router. It will do some tasks, like calling nxt_runtime_quit and exiting. This breaks all connections with all clients.

I tried to confirm that by asking the nxt_router to wait for the apps to close before exiting with this dirty hack:

diff --git a/src/nxt_runtime.c b/src/nxt_runtime.c
index 9bfabc75..9c2fde9b 100644
--- a/src/nxt_runtime.c
+++ b/src/nxt_runtime.c
@@ -11,7 +11,8 @@
 #include <nxt_main_process.h>
 #include <nxt_router.h>
 #include <nxt_regex.h>
-
+#include <sys/types.h>
+#include <signal.h>
 
 static nxt_int_t nxt_runtime_inherited_listen_sockets(nxt_task_t *task,
     nxt_runtime_t *rt);
@@ -454,6 +455,28 @@ nxt_runtime_quit(nxt_task_t *task, nxt_uint_t status)
             nxt_runtime_stop_all_processes(task, rt);
             done = 0;
         }
+        else if (rt->type == NXT_PROCESS_ROUTER) {
+            nxt_process_t        *process;
+            int                   app_count;
+
+            // Wait for apps to stopped before stopping the router
+            for (;;) {
+                app_count = 0;
+                nxt_runtime_process_each(rt, process) {
+                    if (nxt_process_type(process) == NXT_PROCESS_APP && process->use_count > 0) {
+                        // check if process->pid still running
+                        if (kill(process->pid, 0) == 0) {
+                            app_count++;
+                        }
+                    }
+                } nxt_runtime_process_loop;
+                if (app_count == 0) {
+                    break;
+                }
+                nxt_trace(task, "waiting for %d app processes", app_count);
+                sleep(1);
+            }
+        }
     }
 
     nxt_runtime_close_idle_connections(engine);

With this change, the app is able to finish processing and send the answer to the client:

0
2023/11/27 14:12:00 [notice] 1#1 process 51 exited with code 0
2023/11/27 14:12:00 [notice] 53#53 app process 54 exited with code 0
2023/11/27 14:12:00 [notice] 53#53 app process 56 exited with code 0
2023/11/27 14:12:00 [notice] 53#53 app process 55 exited with code 0
1
2023/11/27 14:12:00 [notice] 52#52 waiting for 1 app processes
[...]
2023/11/27 14:12:00 [notice] 52#52 waiting for 1 app processes
172.17.0.1 - - [27/Nov/2023:14:12:21 +0000] "GET /test/timeout HTTP/1.1" 200 19 "-" "curl/8.4.0"
2023/11/27 14:12:21 [notice] 53#53 app process 57 exited with code 0
2023/11/27 14:12:21 [notice] 1#1 process 53 exited with code 0
2023/11/27 14:12:21 [notice] 1#1 process 52 exited with code 0

One downside I noticed is that new connections are still accepted during the time it waits for the end of the processing. All new connections will receive an empty reply when the current request finishes processing and everything quits.

[RomainMou]

Hello, just to let you know that the issue is still present in the last version (1.33.0) of Unit. Is there anything I can do to help further on this issue?