feat: resilient background job retry with exponential backoff and monitoring#592
Open
promisingcoder wants to merge 4 commits intorohitdash08:mainfrom
Open
feat: resilient background job retry with exponential backoff and monitoring#592promisingcoder wants to merge 4 commits intorohitdash08:mainfrom
promisingcoder wants to merge 4 commits intorohitdash08:mainfrom
Conversation
… monitoring endpoints - Add retry state columns to Reminder model: retry_count, last_error, next_retry_at, failed, retry_status; include PostgreSQL ALTER TABLE compatibility patches in _ensure_schema_compatibility - Implement dispatch_reminders() in app/services/jobs.py: queries due reminders, attempts send_reminder(), and schedules up to 3 retries with 5/15/45-minute exponential backoff; permanently marks failed after max retries are exhausted - Wire APScheduler into create_app with a 1-minute interval job; suppress scheduler startup when FLASK_ENV=testing or TESTING=True - Add /jobs blueprint with GET /jobs/status (scheduler state), GET /jobs/reminders/stats (JWT, aggregate counts), POST /jobs/reminders/run (JWT, on-demand dispatch trigger) - Add 21 comprehensive tests in test_jobs.py covering success, skip, backoff delays, max-retry failure, exception capture, endpoint auth, and stats aggregation - Add _FakeRedis in-memory stub to conftest.py (autouse) so all tests run without a live Redis server; fixes pre-existing Redis-related test failures across the suite (42 → 43 passing) - Update README with retry schedule table, column docs, and monitoring endpoint reference Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…vars JOB_MAX_RETRIES (default 3) and JOB_RETRY_DELAYS (default '5,15,45') can now be set as environment variables instead of hardcoded constants. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- dispatch_reminders(candidates, sender_func, now) is now a pure function with no DB calls — only retry/backoff logic - Added run_dispatch_cycle() wrapper that handles DB fetch + commit - Updated routes to call run_dispatch_cycle - Tests refactored: dispatch logic tests use mock objects directly, DB filtering tests use run_dispatch_cycle - All 43 tests pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements resilient background job retry & monitoring for async job execution.
/claim #130
Changes
Retry Mechanism (Exponential Backoff)
JOB_MAX_RETRIES(default 3) andJOB_RETRY_DELAYS(default "5,15,45" minutes)Job State Tracking
retry_count,last_error,next_retry_at,failed,retry_statusto Reminder modelADD COLUMN IF NOT EXISTSPure Dispatch Function
dispatch_reminders(candidates, sender_func, now)— zero DB coupling, fully testablerun_dispatch_cycle()wrapper for DB operationsMonitoring Endpoints
GET /jobs/status— scheduler health (no auth)GET /jobs/reminders/stats— counts by status (JWT)POST /jobs/reminders/run— manual trigger (JWT)Tests
Documentation
Fixes #130