Priority queue #1693
Open
Priority queue #1693
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Yes, this feature is required across all backends. |
Contributor
Author
Great, I'll see what I can do. |
As Redis used a pure list, it needed to change to ZSET so we can order on priority when popping new tasks. Added extra tests to test some edge cases here. This required a bit of refactoring, as we now need a peek-and-acquire strategy to prevent concurrency issues. 1. Use `ZRANGE` to scan the queue without removing jobs. 2. Check job tags in the application. 3. If a match is found, use a Lua script (`ACQUIRE_JOB_SCRIPT`) to atomically verify existence, remove from queue, and move to processing set. To test for this, we added a regression test `test_dequeue_skips_mismatched_tags_no_infinite_loop`.
jtwaleson
force-pushed
the
priority-queue
branch
from
c2ab0ab to
5369e57
Compare
Contributor
Author
|
Hey @kaplanelad , I've added support for sqlite and redis, and rebased my commits. The redis implementation needed some work as the native LPOP operations are incompatible with a priority queue report, and the combination with tags made it still a bit more difficult, hence the updated Lua script. While creating the test cases I found some flaky behavior but I've fixed all the issues. I've been running in production with the postgres implementation for months now, but I've not battle-tested sqlt and redis. I believe it's ready for review & merge. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here we add a priority queue to the postgres bg worker. This way, we can prioritize jobs.
Now we can do
Worker::perform_later_with_priority(ctx, params, Some(100))to schedule a high prio job or withSome(0)for a low prio job.I brought this up in #1462
What I'm using this for: we sync and analyze 100s of git repos. Each one has 3 stages: fetch, quick analysis, thorough analysis. As we want to show the results of the quick analysis as soon as possible, we give the following prios:
fetch: 5,quick analysis: 10,thorough analysis: 1. This way, we always fetch first, but after fetching, we do a quick analysis right away. When everything is fetched and "quick analyzed", we handle thethorough analysis.Please let me know what you think, and if we would want this for all queue backends or if we are fine with just postgres. If we don't want this at all, also fine.