Skip to content

fix: graceful shutdown instead of SIGKILL on worker processes #1349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sansyrox wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: graceful shutdown instead of SIGKILL on worker processes #1349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d53d8b0
fix: graceful shutdown instead of SIGKILL on worker processes
sansyrox Mar 27, 2026
834cf7f
fix: use graceful terminate→wait→kill in reloader's --dev reload path
sansyrox Mar 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions robyn/processpool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,18 @@ def run_processes(
)

def terminating_signal_handler(_sig, _frame):
logger.info("Terminating server!!", bold=True)
logger.info("Gracefully shutting down server...", bold=True)
for process in process_pool:
process.kill()
process.terminate()

for process in process_pool:
process.join(timeout=30)
if process.is_alive():
logger.warning("Process %s did not shut down in time, forcing kill.", process.pid)
process.kill()
process.join(timeout=5)
Comment thread
coderabbitai[bot] marked this conversation as resolved.

sys.exit(0)

signal.signal(signal.SIGINT, terminating_signal_handler)
signal.signal(signal.SIGTERM, terminating_signal_handler)
Expand Down Expand Up @@ -220,6 +229,12 @@ def spawn_process(
try:
server.start(socket, workers)
loop = asyncio.get_event_loop()

if not sys.platform.startswith("win32"):
loop.add_signal_handler(signal.SIGTERM, loop.stop)

loop.run_forever()
except KeyboardInterrupt:
pass
finally:
loop.close()
Comment on lines +233 to 240
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 27, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect whether workers have explicit SIGINT handling or process-group isolation.
rg -n -C2 'signal\.SIGINT|signal\.signal\(signal\.SIGINT|add_signal_handler\(signal\.SIGINT|start_new_session|setsid|setpgrp' .

Repository: sparckles/Robyn

Length of output: 1664

🏁 Script executed:

# First, let's see the overall structure of processpool.py
head -100 robyn/processpool.py | cat -n

Repository: sparckles/Robyn

Length of output: 3620

🏁 Script executed:

# Continue reading to see spawn_process and worker implementation
sed -n '100,250p' robyn/processpool.py | cat -n

Repository: sparckles/Robyn

Length of output: 5761

Workers can still bypass graceful shutdown on Ctrl+C.

Only SIGTERM is trapped in the worker event loop (line 135). The parent installs its SIGINT handler after spawning workers (lines 67–68 occur after line 36–51), so the children inherit default SIGINT behavior. When you press Ctrl+C, the signal reaches workers directly, raising KeyboardInterrupt, which is caught at line 138–139 and does nothing except close the loop. This skips any custom termination logic. The parent's terminating_signal_handler may still execute, but the worker-side flow is not coordinated. Please either ignore SIGINT in the worker event loop or handle it the same way as SIGTERM (calling loop.stop()).

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@robyn/processpool.py` around lines 233 - 240, The worker event loop only
registers SIGTERM via loop.add_signal_handler(signal.SIGTERM, loop.stop), so
Ctrl+C (SIGINT) raises KeyboardInterrupt in children and bypasses coordinated
shutdown; update the worker event loop to also register SIGINT the same way
(e.g., add loop.add_signal_handler(signal.SIGINT, loop.stop) alongside the
SIGTERM registration on non-Windows platforms) or explicitly ignore SIGINT in
the worker before the loop starts; ensure you keep the existing except
KeyboardInterrupt/finally structure but prefer registering SIGINT so
KeyboardInterrupt is not raised in workers.

Copy link
Copy Markdown
Member Author

@sansyrox sansyrox Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolve — reviewed, signal handling is correct for the current architecture.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@sansyrox have exceeded the limit for the number of chat messages per hour. Please wait 1 minutes and 51 seconds before sending another message.

12 changes: 7 additions & 5 deletions robyn/reloader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,13 @@ def __init__(self, file_path: str, directory_path: str) -> None:

def stop_server(self) -> None:
if self.process:
os.kill(self.process.pid, signal.SIGTERM) # Stop the subprocess using os.kill()
self.process.terminate()
try:
self.process.wait(timeout=5)
except subprocess.TimeoutExpired:
logger.warning("Process %s did not shut down in time, forcing kill.", self.process.pid)
self.process.kill()
self.process.wait()
Comment on lines +126 to +132
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 28, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Read-only verification: inspect kill/wait sequencing in shutdown paths.
rg -nP --type=py -C4 '\.terminate\(\)|\.kill\(\)|\.wait\(' robyn/reloader.py robyn/processpool.py

Repository: sparckles/Robyn

Length of output: 2282

Add a timeout to the post-kill wait to prevent indefinite blocking during reload.

Line 132 calls self.process.wait() without a timeout after process.kill(). If the process fails to reap, this blocks the reloader indefinitely, stalling both reload and shutdown operations. This should match the pattern used in robyn/processpool.py:63, which bounds the fallback wait with a timeout.

Proposed fix 
             except subprocess.TimeoutExpired:
                 logger.warning("Process %s did not shut down in time, forcing kill.", self.process.pid)
                 self.process.kill()
-                self.process.wait()
+                try:
+                    self.process.wait(timeout=5)
+                except subprocess.TimeoutExpired:
+                    logger.error("Process %s did not exit after force kill.", self.process.pid)
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@robyn/reloader.py` around lines 126 - 132, The post-kill wait in the
reloader's shutdown path blocks indefinitely because it calls
self.process.wait() with no timeout after self.process.kill(); change this to
use a bounded wait like self.process.wait(timeout=5) (mirroring the pattern in
robyn/processpool.py) and handle subprocess.TimeoutExpired the same way so the
reloader cannot hang if the child fails to reap; update the block around
self.process.terminate(), self.process.wait(...), subprocess.TimeoutExpired,
self.process.kill(), and the final self.process.wait(...) call to include the
timeout and appropriate exception handling.

Copy link
Copy Markdown
Member Author

@sansyrox sansyrox Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolve — out of scope. Reloader improvements tracked separately.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rate Limit Exceeded

@sansyrox have exceeded the limit for the number of chat messages per hour. Please wait 1 minutes and 51 seconds before sending another message.


def reload(self) -> None:
self.stop_server()
Expand All @@ -138,10 +144,6 @@ def reload(self) -> None:
clean_rust_binaries(self.built_rust_binaries)
self.built_rust_binaries = compile_rust_files(self.directory_path)

prev_process = self.process
if prev_process:
prev_process.kill()

self.process = subprocess.Popen(
[sys.executable, *arguments],
env=new_env,
Expand Down
Loading