Currently, clients must manually set the routing-strategy header to control request scheduling behavior. This adds unnecessary burden to the user, especially when different models require different routing strategies (e.g., least-request for model A, least-gpu-mem for model B).
We propose allowing the router to automatically inject the appropriate routing-strategy based on the requested model name (from workload annotation? )
Currently, clients must manually set the
routing-strategyheader to control request scheduling behavior. This adds unnecessary burden to the user, especially when different models require different routing strategies (e.g.,least-requestfor model A,least-gpu-memfor model B).We propose allowing the router to automatically inject the appropriate
routing-strategybased on the requested model name (from workload annotation? )