feat(flowcontrol): add VTC fairness policy with token-based cost and starvation prevention

[gyliu513]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Implements the vtc-fairness-policy plugin as described in #1182 (Task 1).

Unlike round-robin-fairness-policy (which treats every request equally regardless of size), VTC tracks a cumulative virtual token cost per flow. The flow with the lowest virtual counter wins the next dispatch slot, ensuring large-prompt tenants don't starve small-prompt tenants.

/cc @LukeAVanDrie

Which issue(s) this PR fixes:

Related to task1 for #1182

Release note (write NONE if no user-facing change):

feat(flowcontrol): add VTC fairness policy with token-based cost and starvation prevention

[roytman]

There is a PR #1248, which adds support to inference/v1/generate. Depends on which PR will be merged first, this part should be updated.
Can #1303 be relevant ?

[gyliu513]

Thanks for the heads-up, @roytman!

For #1248: good catch. The /inference/v1/generate endpoint uses GenerateRequest{TokenIDs}, which may not surface through the existing ir.Body.TokenizedPrompt path in estimateInputTokens. Depending on merge order, I'll update the cascade to handle that format when #1248 lands (or if it merges first, I'll add support before this one is merged).

For #1303: agree that centralizing token estimation is the right long-term direction. The current approach in vtc is a local heuristic to avoid a hard dependency, but if #1303 moves forward with a shared estimation backend, this function should be replaced to use that.

[github-actions]

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

[ahg-g]

@gyliu513 what is the status here? sorry this fell through the cracks, but if you think this is still relevant and show some evaluation to better understand its potential vs existing policies, I will be happy to review it.

[LukeAVanDrie]

@gyliu513 what is the status here? sorry this fell through the cracks, but if you think this is still relevant and show some evaluation to better understand its potential vs existing policies, I will be happy to review it.

This is on me. We talked and paused this since there was a lot of churn on the inflight tokens data producer and I wanted to wait for that to land so we can leverage it here. Will do a fresh review today and suggest some short-term pragmatic fixes here as the data architecture is still not ideal for flow control plugins.