Skip to content

Commit 86401f7

Browse files
roytmanroytman
committed
Adds coordinator sequence diagram
Signed-off-by: roytman <roytman@il.ibm.com>
1 parent d9c2e4a commit 86401f7

1 file changed

Lines changed: 97 additions & 10 deletions

File tree

‎docs/coordinator_architecture.md‎

Lines changed: 97 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -93,10 +93,14 @@ per-request context held by the coordinator, not in a sidecar on the decode pod.
9393
|
9494
| inference request (OpenAI-compatible API)
9595
v
96+
Inference Gateway --- no EPP-Phase header: default route to coordinator
97+
|
98+
v
9699
Coordinator Service ------------> side services
97100
| (media download, render / tokenize)
98101
|
99102
| one call per phase, tagged EPP-Phase: encode | prefill | decode
103+
| (re-enters the same Inference Gateway)
100104
v
101105
Inference Gateway
102106
| endpoint picker protocol
@@ -112,18 +116,20 @@ per-request context held by the coordinator, not in a sidecar on the decode pod.
112116
In short:
113117

114118
```
115-
one request+response per phase (encode, prefill, decode)
116-
+---------------------------------------------------------+
117-
| v
118-
Client <--> Coordinator --> Inference Gateway --> EPP --> vLLM worker pool
119-
^ |
120-
+---------------------------------------------------------+
121-
response streamed/returned
119+
one request+response per phase (encode, prefill, decode)
120+
+------------------------------------------------------+
121+
| v
122+
Client <--> Inference Gateway <--> Coordinator --> Inference Gateway --> EPP --> vLLM worker pool
123+
^ |
124+
+------------------------------------------------------+
125+
response streamed/returned
122126
```
123127

124-
The client opens a single connection to the coordinator. Behind it, the coordinator
125-
issues several requests to the Inference Gateway and consumes each response before issuing
126-
the next: it is an active client of the Gateway, not a one-shot proxy. A single step
128+
The client opens a single connection through the Inference Gateway to the coordinator
129+
(the gateway's default route, taken when a request carries no `EPP-Phase` header).
130+
Behind it, the coordinator issues several requests back to that same Gateway and
131+
consumes each response before issuing the next: it is an active client of the Gateway,
132+
not a one-shot proxy. A single step
127133
can also fan out into several parallel requests: `replace-media-urls` downloads media
128134
concurrently (to the source URLs), and `encode` issues one Gateway request per
129135
multimodal item in parallel. The number and shape of these requests depend on the input
@@ -134,6 +140,87 @@ download or encode at all. Across phases the coordinator sequences the round-tri
134140
request. Each Gateway call routes it by the `EPP-Phase` header to the EPP, which runs
135141
the matching scheduling profile and picks a pod from that phase's pool.
136142

143+
Every call the Coordinator makes for a phase (`conditional-decode`, `encode`, `prefill`,
144+
`decode`) goes through the same Gateway and the same EPP, which picks the pod for that
145+
phase via the [Endpoint Picker protocol](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/004-endpoint-picker-protocol)
146+
(`ext_proc`). `conditional-decode` tries decode first, optimistically, before running
147+
encode or prefill at all: if the chosen decode pod already has what it needs (e.g. the
148+
prompt is already cached), it serves the request directly and the full pipeline never
149+
runs. Only if decode responds `412 Precondition Failed` does the Coordinator fall back
150+
to the full `encode → prefill → decode` cascade — which is also how a request with
151+
multiple multimedia entries fans encode out in parallel, one call per entry.
152+
153+
```mermaid
154+
sequenceDiagram
155+
participant Client
156+
participant Gateway
157+
participant Coordinator
158+
participant EPP
159+
participant Media as External Media Service
160+
participant Render as Render (side service)
161+
participant Encode as Encode pool
162+
participant Prefill as Prefill pool
163+
participant Decode as Decode pool
164+
165+
Client->>Gateway: Request
166+
Gateway->>Coordinator: Forward request
167+
168+
opt replace-media-urls step configured and request has media entries
169+
par one call per media entry
170+
Coordinator->>Media: Fetch / cache media URL
171+
Media-->>Coordinator: Resolved reference
172+
end
173+
end
174+
175+
opt render step configured
176+
Coordinator->>Render: Normalize multimodal input
177+
Render-->>Coordinator: Normalized tensors
178+
end
179+
180+
Note over Coordinator,Decode: conditional-decode — try decode alone first
181+
Coordinator->>Gateway: Call (EPP-Phase: decode)
182+
Gateway->>EPP: ext_proc: pick pod (profile=decode)
183+
EPP-->>Gateway: Chosen decode pod
184+
Gateway->>Decode: Forward request
185+
186+
alt Decode can serve the request
187+
Decode-->>Gateway: 200 OK (tokens)
188+
Gateway-->>Coordinator: Response
189+
Coordinator-->>Client: Final response
190+
else Decode cannot serve it (e.g. no KV cache for this prompt)
191+
Decode-->>Gateway: 412 Precondition Failed
192+
Gateway-->>Coordinator: 412
193+
Note over Coordinator: Fall back to the full pipeline
194+
195+
opt request has multimodal entries
196+
par one call per media entry
197+
Coordinator->>Gateway: Call (EPP-Phase: encode)
198+
Gateway->>EPP: ext_proc: pick pod (profile=encode)
199+
EPP-->>Gateway: Chosen encode pod
200+
Gateway->>Encode: Forward request
201+
Encode-->>Gateway: Embeddings
202+
Gateway-->>Coordinator: Embeddings
203+
end
204+
end
205+
206+
Note over Prefill,Decode: Shown here: kv-nixl. Other KV connectors (kv-shared-storage,<br/>kv-sglang) transfer KV differently - see KV and EC transfer protocols.
207+
Coordinator->>Gateway: Call (EPP-Phase: prefill)
208+
Gateway->>EPP: ext_proc: pick pod (profile=prefill)
209+
EPP-->>Gateway: Chosen prefill pod
210+
Gateway->>Prefill: Forward request
211+
Prefill-->>Gateway: KV transfer descriptor
212+
Gateway-->>Coordinator: Response
213+
214+
Coordinator->>Gateway: Call (EPP-Phase: decode)
215+
Gateway->>EPP: ext_proc: pick pod (profile=decode)
216+
EPP-->>Gateway: Chosen decode pod
217+
Gateway->>Decode: Forward request (pulls KV via NIXL)
218+
Decode-->>Gateway: 200 OK (tokens)
219+
Gateway-->>Coordinator: Response
220+
Coordinator-->>Client: Final response
221+
end
222+
```
223+
137224
Steps are skipped at runtime when they do not apply (for example, `encode` is a no-op
138225
when the request has no multimodal entries, and `render` short-circuits when the
139226
completions prompt is already a token array). See

0 commit comments

Comments
 (0)