Summary
The Responses API stub backend accepts the standard new_request flag but ignores it. Its fake token sequence is stored in the module-global token_queue, so a new API request starts wherever the previous request stopped rather than from the beginning of the deterministic stub response.
Current behavior
def stub_infer_next_token(
tokens: list[int], temperature: float = 0.0, new_request: bool = False
) -> int:
global token_queue
next_tok = token_queue.pop(0)
...The server sets new_request=True on the first inference call for each StreamResponsesEvents request, and the Metal backend already uses that flag to reset its per-request output state.
Impact
Stub-backed API tests and development sessions are order-dependent across requests. A second request can receive a suffix of the fake response instead of the same deterministic response used by the first request.
Proposed resolution
When new_request=True, reset token_queue to fake_tokens.copy() before returning the first token.
Add a regression that consumes several tokens, starts a new request, and verifies the sequence restarts from fake_tokens[0].
Summary
The Responses API stub backend accepts the standard
new_requestflag but ignores it. Its fake token sequence is stored in the module-globaltoken_queue, so a new API request starts wherever the previous request stopped rather than from the beginning of the deterministic stub response.Current behavior
The server sets
new_request=Trueon the first inference call for eachStreamResponsesEventsrequest, and the Metal backend already uses that flag to reset its per-request output state.Impact
Stub-backed API tests and development sessions are order-dependent across requests. A second request can receive a suffix of the fake response instead of the same deterministic response used by the first request.
Proposed resolution
When
new_request=True, resettoken_queuetofake_tokens.copy()before returning the first token.Add a regression that consumes several tokens, starts a new request, and verifies the sequence restarts from
fake_tokens[0].