Skip to content

Conversation

@benbroadaway
Copy link
Collaborator

Current query applies a possibly very big join against the process_queue table. That can lead to never catching up if the task falls behind for some reason. This change filters-fast down to the configured limit before joining the process_queue data.

Some future considerations:

  • Add a task run limit (e.g. run duration or number of events processed) so a slow server has a termination point to allow a roll of the dice for another server to
    • currently, a "slow" server may run the task indefinitely
  • Tweak the host cache. Use weight instead of duration for eviction. Or maybe just lengthen the time beyond a minute
    • When bogged down, a single loop through events may take more than a minute (or two).
    • Either way, make it configurable so OOM-prone combo can be changed without code change

@benbroadaway benbroadaway requested review from a team and ibodrov August 11, 2025 21:26
@ibodrov ibodrov merged commit 8e2a726 into master Aug 12, 2025
4 checks passed
@ibodrov ibodrov deleted the bb/noderoster-optimize-query branch August 12, 2025 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants