epp: add turn-priority strategy to program-aware fairness#2116
Conversation
Add a turn-priority scoring strategy alongside las. It orders waiting flows by prio = turn_number + turnPriorityTimeWeight * elapsed_seconds so deeper sessions are favored while the elapsed term ages waiting flows up to avoid starvation. Scoring engages only under contention; a lone waiting flow dispatches directly. Turn number is the program's dispatched-request count, keeping the counter per-session and resetting on eviction to match cold KV-cache state. Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com>
github-actions
Bot
added
size/L
|
CC: @praveingk @vMaroon |
| * **Selects a queue to dispatch from** using the configured strategy. Currently `las` (Least Attained Service) is supported; programs with the lowest accumulated service score highest. | ||
| * **Selects a queue to dispatch from** using the configured strategy. Two are supported: | ||
| * `las` (Least Attained Service): programs with the lowest accumulated service score highest. | ||
| * `turn-priority`: scoring engages only under contention. With no flow waiting nothing is dispatched; with exactly one waiting flow it is dispatched directly. When two or more flows are waiting, each scores by its head request's `prio = turn_number + turnPriorityTimeWeight * time_elapsed_in_seconds`, highest first. The elapsed term ages waiting flows up so lower-turn flows are not starved. Turn number is the program's dispatched-request count plus the waiting request itself: each program (fairness ID) has its own counter, advanced once per dispatch. The counter resets when idle program state is evicted: a program idle past the eviction TTL has likely lost its KV cache, so on return it restarts from turn one and re-earns priority from a cold state. |
There was a problem hiding this comment.
I'm fairly sure that individual instances of an agentic session will NOT get their own fairnessID. There is a queue in flowcontrol per fairnessID (per priorityBand). In the use case you are describibg there would be a single element in each queue, being very wastefull.
The original fairness policy was, I think, looking at each agentic workload as a whole, that is all of the requests for the same workload, vs requests for a different workload.
This PR if I understand is trying to look at the individual sessions and prioritize them. This sounds more like an in queue prioritization policy, rather than a fairness policy.
There was a problem hiding this comment.
Thanks @shmuelk .
You are correct, this PR is about scheduling sessions, and prioritizing some (the more advanced) in place of others.
Is your concern the fact that each session has its own queue, with (most of the times) only one element inside being the active request, and you are suggesting that it would be more efficient to have a single queue, with sessions as elements?
What would you recommend? Moving away from Fairness and into a more generic OrderingPolicy?
There was a problem hiding this comment.
The main point I was trying to raise is that the fairnessID is NOT a sessionID. You need to track by some sort of sessionID (there are other PRs that have dealt with sessions and sessionIDs in the past).
You are tracking by program or agenticc workload level and not what you wanted.
There was a problem hiding this comment.
I would claim that an agentID as described in the PR is not a sessionID. It identifies the agent, not the session with the agent.
There was a problem hiding this comment.
@shmuelk , maybe I hooked into the wrong handle, but at the moment the fairnessID is tied to each session, and this commit achieves the intended behaviour, at least for the OTEL traces replay via inference-perf.
If there's a better way on getting a session hook, or a better place in llm-d where to implement this, I'm happy to refactor.
|
Adding comparison plots between this PR vs a normal deployment: 
|
3 participants
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds a second scoring strategy,
turn-priority, to theprogram-aware-fairnessflow-control policy, alongside the existinglasstrategy. It orders waiting flows by their head request'sprio = turn_number + turnPriorityTimeWeight * elapsed_seconds, favoring deeper sessions while the elapsed term ages waiting flows up to avoid starvation. Scoring engages only under contention (a lone waiting flow dispatches directly). The per-program turn counter resets on eviction to match cold KV-cache state.Adds one config field,
turnPriorityTimeWeight(default0.5), with validation.Which issue(s) this PR fixes:
Fixes #2088
Release note (write
NONEif no user-facing change):Test plan (new tests, all passing locally):
turn-priority; rejects negativeturnPriorityTimeWeight