-
Notifications
You must be signed in to change notification settings - Fork 14
Fix/clear zombie captures on startup #59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,6 +25,17 @@ def __init__(self, loglevel: int | None=None) -> None: | |
| self.captures: set[Task[None]] = set() | ||
| self.lacus = Lacus() | ||
|
|
||
| def _clear_ongoing_on_startup(self) -> None: | ||
| # At process start, self.captures is empty — no task can be running. | ||
| # Any UUID left in lacus:ongoing is a zombie from a previous crash. | ||
| # Use clear_capture() per UUID so each gets a proper error result | ||
| # and capture_settings are cleaned up. | ||
| ongoing = self.lacus.monitoring.get_ongoing_captures() | ||
| if ongoing: | ||
| self.logger.warning(f'Startup cleanup: clearing {len(ongoing)} zombie capture(s) from lacus:ongoing') | ||
| for uuid, _ in ongoing: | ||
| self.lacus.core.clear_capture(uuid, 'Cleared on startup: previous process died.') | ||
|
|
||
| async def clear_dead_captures(self) -> None: | ||
| ongoing = {capture.get_name(): capture for capture in self.captures} | ||
| max_capture_time = get_config('generic', 'max_capture_time') | ||
|
|
@@ -47,6 +58,12 @@ async def clear_dead_captures(self) -> None: | |
| if not capture.done(): | ||
| self.logger.error(f'{expected_uuid} is not done after canceling, trying {max_cancel} more times.') | ||
| await asyncio.sleep(1) | ||
| # All cancel attempts exhausted but the task is still stuck. | ||
| # Free the Redis slot so new captures aren't blocked. | ||
| if not capture.done(): | ||
| self.logger.error(f'{expected_uuid} could not be canceled after 5 attempts, force-clearing from Redis.') | ||
| self.lacus.core.clear_capture(expected_uuid, 'Force-cleared: task could not be canceled.') | ||
cln-io marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| self.captures.discard(capture) | ||
|
Comment on lines
+63
to
+66
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This path frees the in-memory slot, but it leaves the Redis Useful? React with 👍 / 👎. |
||
|
|
||
| async def _to_run_forever_async(self) -> None: | ||
|
|
||
|
|
@@ -82,6 +99,10 @@ def main() -> None: | |
| loop.add_signal_handler(signal.SIGTERM, lambda: loop.create_task(p.stop_async())) | ||
|
|
||
| try: | ||
| # Flush stale captures before the event loop starts. | ||
| # Valkey persists across container restarts, so lacus:ongoing | ||
| # may contain UUIDs from a process that was killed mid-capture. | ||
| p._clear_ongoing_on_startup() | ||
| loop.run_until_complete(p.run_async(sleep_in_sec=1)) | ||
| finally: | ||
| loop.close() | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling
self.lacus.core.clear_capture()here does not actually clear most crash leftovers.lacuscore.LacusCore.clear_capture()returns early for any UUID that is still inlacus:ongoingand started less thanmax_capture_time * 1.1ago, so after a normal crash/restart these orphaned captures are usually still considered “probably still going.” The manager logs that it cleared them, but/capture_status,/lacus_status, and/is_busywill keep reporting them as ongoing until they age past the timeout window instead of immediately getting the failure result this startup cleanup is meant to write.Useful? React with 👍 / 👎.