You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This design describes how to supply **model/gateway configuration** (routing strategy, PD bucket bounds, combined mode, etc.) via a **single annotation** (or ConfigMap), with support for **multiple named profiles** selectable at **runtime** by the client.
8
+
9
+
Motivation
10
+
----------
11
+
12
+
Today, options are encoded as many pod labels (e.g. ``model.aibrix.ai/name``, ``model.aibrix.ai/port``, ``model.aibrix.ai/routing-strategy``, ``prompt-min-length``, etc.). Adding new options requires new labels and gateway changes to read them. This does not scale. Using a single structured annotation with **multiple profiles** allows:
13
+
14
+
* One place to add new options (extend the JSON schema).
15
+
* Different configurations for the same model (e.g. ``default``, ``pd``, ``low-latency``) selectable per request via a header.
16
+
17
+
Overview
18
+
--------
19
+
20
+
* **Annotation** (on the pod): ``model.aibrix.ai/config`` holds a JSON object with a ``profiles`` map. Each profile is a set of gateway options: ``routingStrategy``, ``promptLenBucketMinLength``, ``promptLenBucketMaxLength``, ``combined``.
21
+
* **Runtime selection**: Client sends header ``config-profile: <profile-name>`` (e.g. ``pd``, ``low-latency``). If omitted, the ``defaultProfile`` (or ``"default"``) is used.
22
+
23
+
JSON Schema (Implementation)
24
+
----------------------------
25
+
26
+
The implementation parses the following structure. Extra fields (e.g. ``name``, ``port``, ``engine``) in the JSON are ignored.
27
+
28
+
Root object:
29
+
30
+
* ``defaultProfile`` (string, optional): Profile name to use when header is empty or profile not found. Default: ``"default"``.
31
+
* ``profiles`` (object, required): Map of profile name → profile object.
32
+
33
+
Profile object (``ModelConfigProfile``):
34
+
35
+
* ``routingStrategy`` (string): e.g. ``random``, ``pd``, ``least-latency``.
36
+
* ``promptLenBucketMinLength`` (int, optional): Lower bound for bucketing. Default: ``0``. If negative, normalized to ``0``.
37
+
* ``promptLenBucketMaxLength`` (int, optional): Upper bound for bucketing. Default: ``math.MaxInt32`` when ``0`` or omitted.
38
+
* ``combined`` (bool, optional): When true, indicates combined prefill/decode pod for PD routing.
39
+
40
+
Single profile (backward compatible):
41
+
42
+
.. code-block:: json
43
+
44
+
{
45
+
"profiles": {
46
+
"default": {
47
+
"routingStrategy": "pd",
48
+
"promptLenBucketMinLength": 0,
49
+
"promptLenBucketMaxLength": 2048
50
+
}
51
+
}
52
+
}
53
+
54
+
Multiple profiles with default:
55
+
56
+
.. code-block:: json
57
+
58
+
{
59
+
"defaultProfile": "pd",
60
+
"profiles": {
61
+
"default": {
62
+
"routingStrategy": "random",
63
+
"promptLenBucketMinLength": 0,
64
+
"promptLenBucketMaxLength": 4096
65
+
},
66
+
"pd": {
67
+
"routingStrategy": "pd",
68
+
"promptLenBucketMinLength": 0,
69
+
"promptLenBucketMaxLength": 2048
70
+
},
71
+
"low-latency": {
72
+
"routingStrategy": "least-latency",
73
+
"promptLenBucketMinLength": 0,
74
+
"promptLenBucketMaxLength": 2048
75
+
}
76
+
}
77
+
}
78
+
79
+
Runtime Behavior
80
+
----------------
81
+
82
+
1. Gateway resolves config from pod annotation ``model.aibrix.ai/config``. ConfigMap lookup is not yet implemented. If no annotation, fall back to existing label-based resolution.
83
+
2. Gateway reads ``config-profile`` from request headers. If missing, use ``defaultProfile`` from the JSON, or ``"default"``.
84
+
3. Gateway selects the profile via ``GetProfile(profileName)``: exact match first, then fallback to ``defaultProfile``, then ``"default"``.
85
+
4. The resolved profile is stored on ``RoutingContext.ConfigProfile`` (``ResolvedConfigProfile``) for the request.
6. PD router uses ``ResolveProfileFromPod(pod, routingCtx.ReqConfigProfile)`` with fallback to the default profile; prompt bounds and ``combined`` are read from the selected profile.
88
+
89
+
Annotation Example (StormService pod template)
90
+
----------------------------------------------
91
+
92
+
.. code-block:: yaml
93
+
94
+
template:
95
+
metadata:
96
+
labels:
97
+
app: sglang-qwen3-8b-1p1d-0-2k
98
+
model.aibrix.ai/name: qwen3-8B
99
+
annotations:
100
+
prometheus.io/scrape: "true"
101
+
prometheus.io/port: "30000"
102
+
prometheus.io/path: "/metrics"
103
+
model.aibrix.ai/config: |
104
+
{
105
+
"defaultProfile": "pd",
106
+
"profiles": {
107
+
"default": {
108
+
"routingStrategy": "random",
109
+
"promptLenBucketMinLength": 0,
110
+
"promptLenBucketMaxLength": 4096
111
+
},
112
+
"pd": {
113
+
"routingStrategy": "pd",
114
+
"promptLenBucketMinLength": 0,
115
+
"promptLenBucketMaxLength": 2048
116
+
}
117
+
}
118
+
}
119
+
120
+
Client Usage
121
+
------------
122
+
123
+
* Use default profile: do not set any header (or set ``config-profile: default``).
124
+
* Use a specific profile: set header ``config-profile: pd`` or ``config-profile: low-latency``.
125
+
126
+
Implementation
127
+
-------------
128
+
129
+
Package: ``pkg/plugins/gateway/configprofiles/``
130
+
131
+
* ``ModelConfigProfile``: struct with ``RoutingStrategy``, ``PromptLenBucketMinLength``, ``PromptLenBucketMaxLength``, ``Combined``.
132
+
* ``ModelConfigProfiles``: struct with ``DefaultProfile``, ``Profiles map[string]ModelConfigProfile``.
133
+
* ``ParseModelConfig(jsonStr)``: parses JSON; normalizes ``promptLenBucketMinLength`` (≥0) and ``promptLenBucketMaxLength`` (0→MaxInt32).
134
+
* ``GetProfile(name)``: returns profile by name; falls back to ``defaultProfile`` then ``"default"``.
135
+
* ``ResolveProfile(pods, headerProfile)``: iterates pods, returns first non-nil from ``ResolveProfileFromPod``.
136
+
* ``ResolveProfileFromPod(pod, headerProfile)``: reads ``model.aibrix.ai/config`` from pod, parses, returns ``GetProfile(headerProfile)``.
* ``HandleRequestHeaders``: captures ``config-profile`` into ``ReqConfigProfile``.
144
+
* ``HandleRequestBody``: calls ``applyConfigProfile`` which resolves config from pod annotation, sets ``routingCtx.ConfigProfile``, and provides routing strategy to ``deriveRoutingStrategyFromContext``.
145
+
* ``deriveRoutingStrategyFromContext``: chooses the routing strategy for the request using this precedence: (1) request header ``routing-strategy`` if present and non-empty; (2) ``routingCtx.ConfigProfile.RoutingStrategy`` from the resolved profile (config-profile + pod annotation); (3) environment default. Returns the strategy and whether it was explicitly set (used to validate and set ``routingCtx.Algorithm`` in ``HandleRequestBody``).
146
+
147
+
PD router:
148
+
149
+
* ``isPodSuitableForPromptLength(routingCtx, pod, promptLength)``: uses ``ResolveProfileFromPod(pod, routingCtx.ReqConfigProfile)`` for ``promptLenBucketMinLength``/``promptLenBucketMaxLength``.
150
+
* ``isCombinedPod(routingCtx, pod)``: uses ``ResolveProfileFromPod(pod, routingCtx.ReqConfigProfile)`` for ``combined``.
151
+
152
+
Backward Compatibility
153
+
----------------------
154
+
155
+
If no annotation is present, ``ResolveProfile`` returns nil. Gateway continues to use existing pod labels and env for routing strategy, port, engine, etc.
156
+
157
+
Future Work
158
+
----------
159
+
160
+
* ConfigMap lookup (wire when gateway config supports it).
161
+
* Extend profile schema: ``port``, ``metricPort``, ``engine``, ``name`` for full parity with labels.
162
+
* Use request-level ``ConfigProfile`` (from ``config-profile``) for PD bucketing instead of per-pod ``"pd"`` profile.
0 commit comments