Stop injecting placeholder tokens in Responses Ollama inference

[sylvesterkaczmarek]

Summary

Stop the Responses API Ollama backend from manufacturing token id 0 when a streamed token is temporarily unavailable, without blocking the async server while waiting for the next token.

The Ollama inference callable now performs a non-blocking poll. It returns None while the stream is active with no buffered token, and the shared Responses event loop treats that as an internal retry state: it yields asynchronously, appends and parses nothing, and re-checks client disconnects on the next iteration.

Fixes #279.

Fix

• return only real Ollama tokens or EOS_TOKEN; never manufacture token 0;
• preserve the 30-second first-byte timeout and 15-second post-output inactivity timeout;
• preserve stream completion and error handling;
• yield the async event loop between empty Ollama polls so other requests and disconnect checks are not blocked;
• leave integer-returning inference backends unchanged.

Regression coverage

Adds focused tests verifying that:

• an active Ollama stream with no buffered token returns None immediately without calling time.sleep;
• a delayed real token is subsequently returned unchanged;
• a completed stream with no buffered token returns EOS_TOKEN;
• the first-byte timeout returns EOS_TOKEN;
• the post-output inactivity timeout returns EOS_TOKEN.

Ollama request payloads, tokenization, error propagation, and timeout values are otherwise unchanged.

The branch is based directly on current main and collapsed to one signed-off commit.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29c9b70357

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Note

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.