Add program aware plugin

[praveingk]

This PR introduces program-aware plugin would enable identifying a request from an agentic program based on its program-id (x-gateway-inference-fairness-id) and performing scheduling decisions based on program-level metrics. Additionally, this plugin captures program-level metrics and exports to the prometheus endpoint.

Program aware plugin implements following plugin interfaces:

1. Prepare Data Interface: Extracts program information from request headers, subsequently updates the relevant program metrics and request metadata. The plugin assumes that a request arrives with a fairness ID (x-gateway-inference-fairness-id) to identify an agentic program.

2. Fairness Interface (Flow Control): We implement flow-control fairness plugin's Pick interface. Here we enable multiple strategies which can be configured. Currently, we are implemented two strategies, a simple EWMA based, and Deficit Round Robin (DRR).

3. Pre Request Interface: Updates program metrics immediately before dispatch. For example, this hook is used to calculate the wait time (time spent in EPP/queue) of requests per program, and keep track of requests sent to the vLLM inference pod. In future, this hook could also be used to add a vLLM priority to the request.

4. Response Received Interface: Updates program metrics like deficit counters (for DRR) for tokens used.

We are currently evaluating the scheduling strategies using inference-perf benchmarks.

[github-actions]

🚨 Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[hexfusion]

question: while it's true you have no consumers of the ctx in this method. should we perform this work if ctx.Err()?

[hexfusion]

question: nil band is not an error? this means the caller even when the err == nil must nil check the return is nil. also won't this give inaccurate latency metrics for work that was not done?

[D-Sai-Venkatesh]

We return (nil, nil) instead of (nil, err) because this is how the contract for fairness policy has been defined. The existing Fairness plugins also follow the same convention for example, round-robin.

From the behaviour point of view, both cases have same behaviour, that is no item will be dispatched from that band. But in error case the error will be logged.

[hexfusion]

question: how did you arrive at 0.2 for the EWMA alpha? is this based on benchmarking, or should it be configurable via the plugin config alongside strategy?

[praveingk]

We are still experimenting with the parameters. Ideally, we can make it a plugin config once we finalize the programmable parameters for each strategy.