-
Notifications
You must be signed in to change notification settings - Fork 304
Feat improved py cancellation #2965
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+1,531
−68
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
ae66e99
lint
grutt a02a30a
lint
grutt 2007e3f
fix lockfile
grutt 633864a
py test
grutt dfdda69
improved cancellation signaling
grutt 800bb88
feedback
grutt 2d659aa
exit early
grutt 5b0f188
cleanup
grutt a9edb6b
cleanup
grutt 93d5aaa
rm unused
grutt 5253bd7
chore: lint py
grutt 8179fe8
Chore: Fix Python linters, regenerate (#2966)
mrkaye97 0e9fcd2
Merge branch 'main' into feat--improved-py-cancellation
grutt 4e43803
lint
grutt ebf3cbc
revert: go whitespace
grutt 8892da1
Merge branch 'main' into feat--improved-py-cancellation
grutt 3d317d6
chore: feedback round i
grutt 9260500
chore: feedback ii
grutt 281bb0a
Merge branch 'main' into feat--improved-py-cancellation
grutt 08e7eb2
chore: feedback
grutt 388c299
chore: lint
grutt c06145f
release: 1.25.0
grutt 3366b7e
revert: worker to main
grutt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,197 @@ | ||
| """Cancellation token for coordinating cancellation across async and sync operations.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| import threading | ||
| from collections.abc import Callable | ||
| from typing import TYPE_CHECKING | ||
|
|
||
| from hatchet_sdk.exceptions import CancellationReason | ||
| from hatchet_sdk.logger import logger | ||
|
|
||
| if TYPE_CHECKING: | ||
| pass | ||
|
|
||
|
|
||
| class CancellationToken: | ||
| """ | ||
| A token that can be used to signal cancellation across async and sync operations. | ||
|
|
||
| The token provides both asyncio and threading event primitives, allowing it to work | ||
| seamlessly in both async and sync code paths. Child workflow run IDs can be registered | ||
| with the token so they can be cancelled when the parent is cancelled. | ||
|
|
||
| Example: | ||
| ```python | ||
| token = CancellationToken() | ||
|
|
||
| # In async code | ||
| await token.aio_wait() # Blocks until cancelled | ||
|
|
||
| # In sync code | ||
| token.wait(timeout=1.0) # Returns True if cancelled within timeout | ||
|
|
||
| # Check if cancelled | ||
| if token.is_cancelled: | ||
| raise CancelledError("Operation was cancelled") | ||
|
|
||
| # Trigger cancellation | ||
| token.cancel() | ||
| ``` | ||
| """ | ||
|
|
||
| def __init__(self) -> None: | ||
| self._cancelled = False | ||
| self._reason: CancellationReason | None = None | ||
| self._async_event: asyncio.Event | None = None | ||
| self._sync_event = threading.Event() | ||
| self._child_run_ids: list[str] = [] | ||
| self._callbacks: list[Callable[[], None]] = [] | ||
| self._lock = threading.Lock() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. how does the async code play with
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this should be fine afict i refactored some of the callback stuff to minimize any contention |
||
|
|
||
| def _get_async_event(self) -> asyncio.Event: | ||
| """Lazily create the asyncio event to avoid requiring an event loop at init time.""" | ||
| if self._async_event is None: | ||
| self._async_event = asyncio.Event() | ||
| # If already cancelled, set the event | ||
| if self._cancelled: | ||
| self._async_event.set() | ||
| return self._async_event | ||
|
|
||
| def cancel( | ||
| self, reason: CancellationReason = CancellationReason.TOKEN_CANCELLED | ||
| ) -> None: | ||
| """ | ||
| Trigger cancellation. | ||
|
|
||
| This will: | ||
| - Set the cancelled flag and reason | ||
| - Signal both async and sync events | ||
| - Invoke all registered callbacks | ||
|
|
||
| Args: | ||
| reason: The reason for cancellation. | ||
| """ | ||
| with self._lock: | ||
| if self._cancelled: | ||
| logger.debug( | ||
| f"CancellationToken: cancel() called but already cancelled, " | ||
| f"reason={self._reason.value if self._reason else 'none'}" | ||
| ) | ||
| return | ||
|
|
||
| logger.debug( | ||
| f"CancellationToken: cancel() called, reason={reason.value}, " | ||
| f"{len(self._child_run_ids)} children registered" | ||
| ) | ||
|
|
||
| self._cancelled = True | ||
| self._reason = reason | ||
|
|
||
| # Signal both event types | ||
| if self._async_event is not None: | ||
| self._async_event.set() | ||
| self._sync_event.set() | ||
grutt marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # Snapshot callbacks under the lock, invoke outside to avoid deadlocks | ||
| callbacks = list(self._callbacks) | ||
|
|
||
| for callback in callbacks: | ||
| try: | ||
| logger.debug(f"CancellationToken: invoking callback {callback}") | ||
| callback() | ||
| except Exception as e: # noqa: PERF203 | ||
| logger.warning(f"CancellationToken: callback raised exception: {e}") | ||
|
|
||
| logger.debug(f"CancellationToken: cancel() complete, reason={reason.value}") | ||
|
|
||
| @property | ||
| def is_cancelled(self) -> bool: | ||
| """Check if cancellation has been triggered.""" | ||
| return self._cancelled | ||
|
|
||
| @property | ||
| def reason(self) -> CancellationReason | None: | ||
| """Get the reason for cancellation, or None if not cancelled.""" | ||
| return self._reason | ||
|
|
||
| async def aio_wait(self) -> None: | ||
| """ | ||
| Await until cancelled (for use in asyncio). | ||
|
|
||
| This will block until cancel() is called. | ||
| """ | ||
| await self._get_async_event().wait() | ||
| logger.debug( | ||
| f"CancellationToken: async wait completed (cancelled), " | ||
| f"reason={self._reason.value if self._reason else 'none'}" | ||
| ) | ||
|
|
||
| def wait(self, timeout: float | None = None) -> bool: | ||
| """ | ||
| Block until cancelled (for use in sync code). | ||
|
|
||
| Args: | ||
| timeout: Maximum time to wait in seconds. None means wait forever. | ||
|
|
||
| Returns: | ||
| True if the token was cancelled (event was set), False if timeout expired. | ||
| """ | ||
| result = self._sync_event.wait(timeout) | ||
| if result: | ||
| logger.debug( | ||
| f"CancellationToken: sync wait interrupted by cancellation, " | ||
| f"reason={self._reason.value if self._reason else 'none'}" | ||
| ) | ||
| return result | ||
|
|
||
| def register_child(self, run_id: str) -> None: | ||
| """ | ||
| Register a child workflow run ID with this token. | ||
|
|
||
| When the parent is cancelled, these child run IDs can be used to cancel | ||
| the child workflows as well. | ||
|
|
||
| Args: | ||
| run_id: The workflow run ID of the child workflow. | ||
| """ | ||
| with self._lock: | ||
| logger.debug(f"CancellationToken: registering child workflow {run_id}") | ||
| self._child_run_ids.append(run_id) | ||
|
|
||
| @property | ||
| def child_run_ids(self) -> list[str]: | ||
| """The registered child workflow run IDs.""" | ||
| return self._child_run_ids | ||
|
|
||
| def add_callback(self, callback: Callable[[], None]) -> None: | ||
| """ | ||
| Register a callback to be invoked when cancellation is triggered. | ||
|
|
||
| If the token is already cancelled, the callback will be invoked immediately. | ||
|
|
||
| Args: | ||
| callback: A callable that takes no arguments. | ||
| """ | ||
| with self._lock: | ||
| if self._cancelled: | ||
| invoke_now = True | ||
| else: | ||
| invoke_now = False | ||
| self._callbacks.append(callback) | ||
|
|
||
| if invoke_now: | ||
| logger.debug( | ||
| f"CancellationToken: invoking callback immediately (already cancelled): {callback}" | ||
| ) | ||
| try: | ||
| callback() | ||
| except Exception as e: | ||
| logger.warning(f"CancellationToken: callback raised exception: {e}") | ||
|
|
||
| def __repr__(self) -> str: | ||
| return ( | ||
| f"CancellationToken(cancelled={self._cancelled}, " | ||
| f"children={len(self._child_run_ids)}, callbacks={len(self._callbacks)})" | ||
| ) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.