Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker pings #133

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from
Draft

Worker pings #133

wants to merge 8 commits into from

Conversation

RealOrangeOne
Copy link
Owner

@RealOrangeOne RealOrangeOne commented Feb 3, 2025

Fixes #46

All the time a worker process is running, it reports its status to a database table. All the time those pings are coming in, the task is considered running. If the pings stop for too long, it's a sign the worker has stopped processing the task, and the task should be considered failed (and thus retried when that's implemented)

The pings are intentionally their own separate table to avoid needing rewrite the entire task row with each ping.

To do

  • Mark tasks as failed
  • Ping only when a task is running (reduce write fatigue)
  • Loosen requirement that worker ID is a uuid (using a hostname would be useful)
  • Prune stale pings
  • Tests
  • Make pings more robust (currently won't fire if lots of small tasks are processed)

@RealOrangeOne RealOrangeOne added the database-backend Issues relating to the database backend label Feb 3, 2025
This is not only more correct, but should let the process terminate
sooner in a few edge cases.
It was probably a premature optimisation, and one to come back to in future
Whilst it's more CPU intensive, it should be faster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database-backend Issues relating to the database backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unclean exits of the worker leave tasks in limbo
1 participant