Add execution state tracking and progress monitoring by desertaxle · Pull Request #181 · chrisguidry/docket

desertaxle · 2025-10-31T20:53:24Z

Summary

Adds comprehensive execution state tracking and progress monitoring capabilities to Docket, enabling real-time observability of task execution through Redis pub/sub.

Implements ExecutionProgress class with instance attributes for tracking task progress
Adds execution state machine: SCHEDULED → QUEUED → RUNNING → COMPLETED/FAILED
Provides pub/sub event publishing for state transitions and progress updates
Moves scheduling logic from Docket to Execution class for better encapsulation
Add docket watch CLI to view progress for a single task

Demo

Screen.Recording.2025-11-03.at.10.17.22.AM.mov

Key Changes

Execution State Management

Renamed PENDING state to QUEUED throughout codebase
Immediate tasks now transition directly to QUEUED state
Future tasks start as SCHEDULED, then move to QUEUED when due
Worker scheduler now publishes pub/sub events when moving tasks from scheduled to queued

ExecutionProgress Class

Added instance attributes: current, total, message, updated_at
All Redis-modifying methods now update both Redis and instance attributes atomically
Added create() classmethod for initialization
Progress data automatically deleted on task completion/failure

Scheduling Refactor

Moved scheduling logic from Docket._schedule() to Execution.schedule()
Atomic operation combining stream insertion and state record writing via Lua script
Removed redundant set_scheduled() and set_queued() methods
State transitions now handled automatically by scheduling logic

Pub/Sub Event Publishing

State changes publish to {docket}:state:{key} channel
Progress updates publish to {docket}:progress:{key} channel
Both Execution.subscribe() and ExecutionProgress.subscribe() methods available
Worker scheduler Lua script now publishes events for scheduled→queued transitions

Closes #88

🤖 Generated with Claude Code

Ensures the worker scheduler emits pub/sub events when moving tasks from scheduled to queued state, providing consistent event monitoring across all state transitions. Removes the redundant set_queued() method since state transitions are now automated in the scheduling Lua scripts. Also improves type annotations to achieve 100% type completeness score in pyright verification by adding explicit return types to subscribe() methods and list type annotations in tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

claude · 2025-10-31T20:53:35Z

Claude finished @desertaxle's task —— View job

PR Review: Add execution state tracking and progress monitoring

I'm reviewing this comprehensive PR that adds execution state tracking and progress monitoring capabilities to Docket.

Review Checklist

Review ExecutionProgress implementation
Examine state transition logic
Check pub/sub event system
Review scheduling refactor (Docket → Execution)
Analyze test coverage quality
Performance considerations
Security review
Review existing feedback and suggestions

Summary

This is an excellent, well-architected implementation that adds comprehensive execution state tracking and real-time progress monitoring to Docket. The code quality is high, test coverage remains at 100%, and the design patterns are solid.

Strengths ✅

Architecture & Design

Clean separation of concerns: Moving scheduling logic from Docket to Execution class improves encapsulation
State machine implementation: Clear transitions through SCHEDULED → QUEUED → RUNNING → COMPLETED/FAILED
Atomic operations: Excellent use of Lua scripts for atomic Redis operations
Progress tracking: Well-designed ExecutionProgress class with intuitive API

Code Quality

Type safety: Comprehensive type hints throughout
Documentation: Excellent docstrings and inline comments
Error handling: Robust validation (e.g., total >= 1 check)
Test coverage: Maintained at 100% with thorough test scenarios

Performance

Atomic Lua scripts: Efficient Redis operations minimize round trips
TTL management: Smart automatic cleanup of execution state data
Pub/sub efficiency: Events only published when needed

Review Comments & Suggestions 🔍

1. Default Progress Total Value

In ExecutionProgress.__init__ (execution.py:127), the default total is 1, but in other places it's 100. Consider standardizing:

# execution.py:127 - Currently 1
self.total: int = 1

# execution.py:236, 249 - Uses 100
self.total = 100

# worker.py:596 - Sets to 100
redis.call('HSET', progress_key, 'current', '0', 'total', '100')

Recommendation: Set default to 1 consistently as suggested by reviewer, since it's more intuitive for simple use cases.

2. Progress Total Validation

The validation in set_total() (execution.py:152) is good. Consider adding similar validation in the constructor to prevent direct attribute manipulation:

def __init__(self, docket: "Docket", key: str, initial_total: int = 1) -> None:
    if initial_total < 1:
        raise ValueError("Total must be at least 1")
    self.total = initial_total

3. Context Manager Support

The reviewer asked about context managers. While not implemented, this could be a nice future enhancement for automatic progress cleanup:

@asynccontextmanager
async def track_progress(execution: Execution):
    """Context manager for automatic progress tracking."""
    try:
        yield execution.progress
    finally:
        # Progress is already auto-deleted on completion/failure
        pass

4. Event Type Definitions

Good use of TypedDict for ProgressEvent and StateEvent. Consider adding a discriminated union type for better type narrowing:

ExecutionEvent = Union[StateEvent, ProgressEvent]

5. Legacy Key Cleanup

Excellent handling of backwards compatibility with TODO comments marking legacy code for removal. The migration path is clear and safe.

Minor Issues 🔬

1. Redundant Comments

Some AI-generated breadcrumb comments should be removed (docket.py:439, as noted by reviewer).

2. Progress Subscribe Coverage

The subscribe() methods have pragma: no cover on the async iteration loops. While understandable for testing async generators, consider adding integration tests for these paths.

3. TTL Configuration

Good addition of configurable execution_ttl. Consider documenting recommended values based on use cases in the docstring.

Security Considerations 🔒

✅ No credential exposure in logs or events
✅ Proper input validation for progress values
✅ Safe Redis key construction preventing injection
✅ Atomic operations prevent race conditions
✅ TTL ensures automatic cleanup of sensitive data

Performance Considerations ⚡

✅ Lua scripts minimize Redis round trips
✅ Efficient pub/sub with targeted channels
✅ Smart TTL management prevents memory bloat
✅ Atomic state transitions avoid race conditions
⚠️ Consider adding a maximum message size check for progress messages to prevent memory issues with very large status messages

Test Coverage Analysis 🧪

Test coverage is excellent at 100%. Tests comprehensively cover:

State transitions through full lifecycle
Progress tracking with increments and messages
Edge cases (invalid values, missing data)
TTL verification
Pub/sub event publishing
CLI integration with watch command

Recommendations 📝

Immediate: Fix the default total value inconsistency (use 1 throughout)
Immediate: Add total validation as suggested by reviewer
Consider: Add maximum length validation for progress messages
Future: Consider adding context manager support for progress tracking
Future: Add metrics for tracking average task duration by state

Conclusion

This is a well-executed feature addition that significantly enhances Docket's observability capabilities. The code is production-ready with only minor adjustments needed. The architecture is solid, the implementation is clean, and the test coverage is comprehensive.

Verdict: ✅ Approved with minor suggestions

Excellent work on this significant enhancement to Docket! The execution state tracking and progress monitoring features are well-designed and thoroughly tested.

github-actions · 2025-10-31T20:53:50Z

📚 Documentation has been built for this PR!

You can download the documentation directly here:
https://github.com/chrisguidry/docket/actions/runs/19083904930/artifacts/4467514519

chrisguidry

Love where this is heading!!!

src/docket/execution.py

chrisguidry · 2025-11-03T14:58:00Z

src/docket/execution.py

        initiating_context = initiating_span.get_span_context()
        return [trace.Link(initiating_context)] if initiating_context.is_valid else []

+    async def schedule(self, replace: bool = False) -> None:


Oh wow this is a cool refactoring, didn't think of this!

src/docket/cli.py

Adds module-level docstring explaining progress tracking concepts, detailed function docstrings for long_task and main, and inline comments throughout to help users understand and adapt the example to their own use cases. Documentation covers: - Progress dependency injection pattern - ExecutionProgress API (increment, set_message) - Real-time monitoring with docket watch command - Task scheduling and state lifecycle - Complete setup from Redis to Worker to execution 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Reduces Redis key count from 4 to 2-3 per task by storing known marker and stream_id as fields in the execution state hash instead of separate keys. Changes: - Immediate tasks: 4→2 keys (50% reduction) - Future tasks: 4→3 keys (25% reduction) Implementation: - Store 'known' field in {docket}:runs:{key} hash for duplicate prevention - Store 'stream_id' field in {docket}:runs:{key} hash for cancellation - Delete known/stream_id fields from runs hash when task completes to allow rescheduling - Update retry logic to use replace=True to handle existing runs hash Backwards compatibility: - Check both new (HGET runs hash) and legacy (GET separate key) locations - Clean up legacy keys during replacement and cancellation - All backwards compat code marked with TODO for removal in v0.14.0 Benefits: - Fewer Redis keys to manage and clean up - Reduced memory overhead (~200-300 bytes per task) - Simpler cleanup logic (fewer DEL operations) - Better data locality (related fields in same hash) Testing: - All 349 tests passing including legacy compatibility test - Fixed test_redis_key_cleanup_cancelled_task to use correct assertion method 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Consolidates worker operations when claiming a task into a single atomic Lua script, reducing Redis round trips from 3-4 to 1 operation. Changes: - Add claim_and_run() method that atomically: - Sets state to RUNNING with worker name and timestamp - Initializes progress tracking (current=0, total=100) - Deletes known/stream_id fields to allow task rescheduling - Cleans up legacy keys for backwards compatibility - Remove redundant set_running() method - Update worker to use claim_and_run() instead of set_running() - Update all tests to use claim_and_run() Benefits: - Single Redis round trip (75% reduction) - Atomic operation prevents partial state updates - Consistent with schedule() Lua script pattern - Better performance and reliability Testing: - All 349 tests passing - All state transitions verified - Progress initialization working correctly 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Renames state transition methods to be more explicit and self-documenting: - set_completed() → mark_as_completed() - set_failed() → mark_as_failed() This improves API clarity and consistency with the new claim_and_run() method. Changes: - Rename method definitions in Execution class - Update 2 call sites in Worker - Update 2 call sites in tests All 349 tests passing with 98% coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Renames the method to better reflect its actual behavior - it claims the task and transitions to RUNNING state, but does not execute the function. Changes: - claim_and_run() → claim() in Execution class - Update Worker call site - Update 6 test call sites The new name is more accurate since the method only claims the task for execution; the actual function execution happens later in Worker._execute(). All 349 tests passing with 98% coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

codecov-commenter · 2025-11-04T14:58:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (eb8269f) to head (8d0029f).

Additional details and impacted files

@@            Coverage Diff             @@
##              main      #181    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           34        36     +2     
  Lines         5216      6100   +884     
  Branches       265       295    +30     
==========================================
+ Hits          5216      6100   +884

Flag	Coverage Δ
python-3.10	`100.00% <100.00%> (ø)`
python-3.11	`98.96% <99.89%> (+0.17%)`	⬆️
python-3.12	`100.00% <100.00%> (ø)`
python-3.13	`100.00% <100.00%> (ø)`
python-3.14	`100.00% <100.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/docket/__init__.py	`100.00% <100.00%> (ø)`
src/docket/agenda.py	`100.00% <ø> (ø)`
src/docket/cli.py	`100.00% <100.00%> (ø)`
src/docket/dependencies.py	`100.00% <100.00%> (ø)`
src/docket/docket.py	`100.00% <100.00%> (ø)`
src/docket/execution.py	`100.00% <100.00%> (ø)`
src/docket/worker.py	`100.00% <100.00%> (ø)`
tests/cli/test_watch.py	`100.00% <100.00%> (ø)`
tests/test_dependencies.py	`100.00% <ø> (ø)`
tests/test_docket.py	`100.00% <100.00%> (ø)`
... and 5 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The watch command was failing in CI when tasks completed quickly because: - subscribe() would sync() and find task already completed - Progress data would be deleted on completion - Watch would never see progress events Solution: subscribe() now emits initial progress event along with initial state event, ensuring subscribers capture progress state before deletion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The conditional check was preventing progress events from being emitted when tasks hadn't set progress (current=None, total=100 defaults). Now always emits initial progress event to ensure watch command and other subscribers can display progress state consistently. Also fixes test expecting 3 progress events (initial + 2 updates). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The watch command was checking worker_name to determine if a task had started, but worker_name timing varied across Python versions in CI. Now checks execution.started_at which is set atomically when task starts, providing a more reliable indicator that's available in initial state event. Fixes Python 3.12-specific test failures in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The redis context was incorrectly dedented outside the Worker context. This caused branch coverage issues in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

This reverts commit 4fc3780.

chrisguidry

It's a beaut!

chrisguidry · 2025-11-04T19:08:00Z

src/docket/cli.py

 from rich.console import Console
+from rich.layout import Layout
+from rich.live import Live
+from rich.progress import (


❤️ what a great toolkit

chrisguidry · 2025-11-04T19:09:09Z

src/docket/docket.py

    TaskFunction,
 )
+
+# Run class has been consolidated into Execution


Probably AI breadcrumb comments

chrisguidry · 2025-11-04T19:10:56Z

src/docket/docket.py

                    # ARGV: task_key
                    """
                    local stream_key = KEYS[1]
+                    -- TODO: Remove in next breaking release (v0.14.0) - legacy key locations


chrisguidry · 2025-11-04T19:12:57Z

src/docket/execution.py

+        self.key = key
+        self._redis_key = f"{docket.name}:progress:{key}"
+        self.current: int | None = None
+        self.total: int = 100


Let's make the default for total 1 instead.

chrisguidry · 2025-11-04T19:13:47Z

src/docket/execution.py

+        await instance.sync()
+        return instance
+
+    async def set_total(self, total: int) -> None:


Let's validate that total >= 1

chrisguidry · 2025-11-04T19:15:46Z

src/docket/execution.py

+            }
+            await redis.publish(channel, json.dumps(payload))
+
+    async def subscribe(self) -> AsyncGenerator[dict[str, Any], None]:


I think we could define a ProgressEvent here if we wanted to

… persistence Documents features added in PRs #181, #184, and #189: - Add comprehensive "Task State and Progress Monitoring" section to advanced-patterns.md covering execution state machine, Redis data model, pub/sub events, progress reporting, result retrieval, and CLI monitoring - Add Progress() dependency documentation to dependencies.md with usage examples and links to detailed patterns - Add "State and Result Storage" section to production.md documenting execution_ttl configuration, fire-and-forget mode (execution_ttl=0), and custom result storage backends - Add "Task Observability" section to getting-started.md introducing state tracking, progress reporting, and result retrieval with links to detailed documentation All documentation includes working code examples and follows the existing tone and style. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

… persistence (#191) ## Summary Adds comprehensive documentation for the execution state tracking, progress monitoring, and result persistence features introduced in PRs #181, #184, and #189. **Related PRs:** - #181 - Add execution state tracking and progress monitoring - #184 - Add result persistence - #189 - Add support for execution_ttl=0 and timeout/deadline parameters ## Documentation Changes ### docs/advanced-patterns.md Added a new major section **"Task State and Progress Monitoring"** covering: - **High-Level Design** - Architecture overview explaining the execution state machine, Redis data model, pub/sub event system, and result storage - **Tracking Execution State** - How to query and monitor task states - **Reporting Task Progress** - Using the `Progress()` dependency with `ExecutionProgress` API - **Monitoring Progress in Real-Time** - Subscribing to progress and state events programmatically - **Advanced Progress Patterns** - Incremental progress, batch updates, and nested tracking patterns - **Retrieving Task Results** - Using `get_result()` with timeout/deadline parameters and exception handling - **CLI Monitoring with Watch** - Using the `docket watch` command for real-time monitoring - **Fire-and-Forget Mode** - Documenting `execution_ttl=0` for high-throughput scenarios ### docs/dependencies.md Added **"Reporting Task Progress"** subsection to Built-in Context Dependencies: - Documents the `Progress()` dependency injection pattern - Explains `ExecutionProgress` methods (`set_total`, `increment`, `set_message`, `sync`) - Highlights key characteristics (atomic, real-time, observable, ephemeral) - Links to detailed documentation in advanced-patterns.md ### docs/production.md Added **"State and Result Storage"** section: - Execution state tracking configuration with `execution_ttl` - Fire-and-forget mode using `execution_ttl=0` for maximum throughput - Result storage configuration with custom backends (RedisStore, MemoryStore) - Best practices for TTL management, separate databases, and large results - Monitoring state and result storage in Redis ### docs/getting-started.md Added **"Task Observability"** section: - Introduction to execution state tracking - Basic progress reporting examples - Task result retrieval with `get_result()` - Links to detailed documentation in advanced-patterns.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>

desertaxle and others added 2 commits October 31, 2025 14:20

Initial implementation

72a0ded

desertaxle mentioned this pull request Oct 31, 2025

Add task completion tracking with progress state management #177

Closed

chrisguidry reviewed Nov 3, 2025

View reviewed changes

src/docket/cli.py Outdated Show resolved Hide resolved

desertaxle and others added 10 commits November 3, 2025 10:20

Add progress example

4a08a0a

Get to 100% test coverage

665a804

Add fundamental tests

282cfce

Merge branch 'main' into exectuion-expansion

7754f0d

Fix test failures after updates from main

0519481

desertaxle added 3 commits November 4, 2025 09:06

Add ability to specify custom ttl

64ad325

Fixes test failure

18b91fc

Fix memory test failures

5aea763

chrisguidry mentioned this pull request Nov 4, 2025

Best-effort cancellation of running tasks #183

Closed

desertaxle and others added 3 commits November 4, 2025 11:22

Add verbose logging for Python 3.12

bdabee2

desertaxle marked this pull request as ready for review November 4, 2025 17:39

desertaxle requested a review from chrisguidry November 4, 2025 17:39

desertaxle and others added 3 commits November 4, 2025 11:42

Remove checks for progress percentage in CLI tests

fd2ed28

Increase timeout

b6732ea

desertaxle and others added 7 commits November 4, 2025 12:02

Fix indentation in test_custom_execution_ttl

4fc3780

The redis context was incorrectly dedented outside the Worker context. This caused branch coverage issues in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Revert "Fix indentation in test_custom_execution_ttl"

18697d3

This reverts commit 4fc3780.

Improve test timing

0c79581

Adjust test timing

6abe56a

Fix test

1a6e8aa

Fix assertion

a8ba66b

Merge branch 'main' into exectuion-expansion

c7854f4

chrisguidry approved these changes Nov 4, 2025

View reviewed changes

desertaxle added 3 commits November 4, 2025 15:41

Address review comments

aabacc9

Add coverage

d668e4b

Remove verbose pytest flag

8d0029f

desertaxle merged commit c5dc2c1 into main Nov 4, 2025
42 of 44 checks passed

desertaxle deleted the exectuion-expansion branch November 4, 2025 22:10

desertaxle mentioned this pull request Nov 7, 2025

Add documentation for state tracking, progress monitoring, and result persistence #191

Merged

Conversation

desertaxle commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Demo

Key Changes

Execution State Management

ExecutionProgress Class

Scheduling Refactor

Pub/Sub Event Publishing

Uh oh!

claude bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review: Add execution state tracking and progress monitoring

Review Checklist

Summary

Strengths ✅

Architecture & Design

Code Quality

Performance

Review Comments & Suggestions 🔍

1. Default Progress Total Value

2. Progress Total Validation

3. Context Manager Support

4. Event Type Definitions

5. Legacy Key Cleanup

Minor Issues 🔬

1. Redundant Comments

2. Progress Subscribe Coverage

3. TTL Configuration

Security Considerations 🔒

Performance Considerations ⚡

Test Coverage Analysis 🧪

Recommendations 📝

Conclusion

Uh oh!

github-actions bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrisguidry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chrisguidry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

desertaxle commented Oct 31, 2025 •

edited

Loading

claude bot commented Oct 31, 2025 •

edited

Loading

github-actions bot commented Oct 31, 2025 •

edited

Loading

codecov-commenter commented Nov 4, 2025 •

edited

Loading