[Feature]: DispatchGates should be assigned to inference targets

[evacchi]

Problem Statement

currently each DispatchGate is assigned to a specific queue

• each queue might be assigned to a different InferenceTarget;
• however, nothing potentially prevents to assign multiple queues to the same endpoint/InferenceTarget

however, in the latter case, the DispatchGate might erroneously compute the same "dispatch budget" for each input queue; while instead, the budget should be shared across them;

e.g.:

• queue 1 targets EP1
• queue 2 targets EP1
• queue 3 targets EP2

if budget(EP1) = 0.6 and budget(EP2) = 0.8:

• queue 3 should be allowed to dispatch 80% of EP2's max capacity
• however, queue 1 and queue 2 should share 60% of EP1's max capacity
  • e.g. it could be 30% each (budget(EP1)/2)) or any combination such that budget(queue_1) + budget(queue_2) = budget(EP1); for instance, if queue_1 and queue_2 were assigned priorities, then the slice of budget may be defined in terms of such priority; etc...

Proposed Solution

DispatchGate should not be (just) assigned to queues and instead it should be assigned to inference targets; or, alternatively, some sort of "hierarchy of gates" should be defined, so that the budget can be "shared" across queues.

Alternatives Considered

for the time being, it's fine to limit 1 queue : 1 gate, as long as the limitation is clearly documented.

Willingness to Contribute

Yes, I can submit a PR

Additional Context

No response

[evacchi]

note, this is a low-priority item. Gate-per-queue is fine as of now, we should just document that having multiple queues targeting the same inference pool, might require a custom gating strategy

[jtechapps]

This is similar issue to #304. Once #304 is close this issue should be resolved. Looking into this now.