feat: Flotilla scheduler and dispatcher actors #4375
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4375 +/- ##
==========================================
- Coverage 77.72% 76.73% -0.99%
==========================================
Files 844 846 +2
Lines 113446 115597 +2151
==========================================
+ Hits 88176 88704 +528
- Misses 25270 26893 +1623
🚀 New features to boost your workflow:
|
colin-ho
changed the title
|
Todo, add |
| } | ||
| } | ||
| } | ||
| println!("[Scheduler] Scheduler loop complete"); |
There was a problem hiding this comment.
eprintln for logs instead so they don't get buffered?
There was a problem hiding this comment.
Once this really tripped me up during debugging
| ) -> DaftResult<()> { | ||
| todo!("FLOTILLA_MS1: Implement run scheduler loop"); | ||
| let mut input_exhausted = false; | ||
| while !input_exhausted || scheduler.num_pending_tasks() > 0 { |
There was a problem hiding this comment.
Why have input_exhausted? Is it for tasks that get canceled?
There was a problem hiding this comment.
Input exhausted means that there won't be any new tasks submitted to the scheduler.
I can probably simplify this logic a bit more, the loop condition i was going for was essentially "keep looping until you have no more tasks to schedule"
There was a problem hiding this comment.
To clarify, is it possible for input_exhausted = true and scheduler.num_pending_tasks() > 0, and in what state would that be?
There was a problem hiding this comment.
yeah lets say we had 100 tasks in total for this query, but only 1 worker with 1 core.
All 100 tasks could have been submitted to the scheduler already, so input is exhausted, but scheduler could have 99 pending tasks because it only just scheduled 1
There was a problem hiding this comment.
input exhausted means the scheduler will not receive any more tasks via the task receiver channel. but pending tasks is the internal priority queue in the scheduler, which could still have tasks
Changes Made
Implement scheduler + dispatcher actors for flotilla. Including unit tests for both.
Related Issues
Checklist
docs/mkdocs.ymlnavigation