fix(rag): serve streaming chat completions as text/event-stream (#2837)

[Anai-Guo]

Summary

Fixes #2837. When serving with ramalama serve --rag, the RAG proxy's streaming
chat-completion path returned the SSE stream with media_type="text/plain" instead
of text/event-stream. Strict OpenAI-compatible clients (e.g. Open WebUI) require
content-type: text/event-stream for "stream": true responses and reject the
stream otherwise — the chat appears to hang and then fails with a decode error, even
though the model server underneath is generating tokens normally. Non-streaming
requests are unaffected (they correctly return application/json).

Fix

_stream_completion in container-images/scripts/rag_framework already emits correct
SSE framing (data: {...}\n\n chunks terminated by data: [DONE]\n\n); only the
response media type was wrong. Set it to text/event-stream to match the OpenAI Chat
Completions streaming contract so unmodified clients can consume the stream.

         return StreamingResponse(
             generate(),
-            media_type="text/plain",
+            media_type="text/event-stream",
             headers={"Cache-Control": "no-cache", "Connection": "keep-alive"},
         )

Assisted-by: Claude

🤖 Generated with Claude Code

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes

  • Updated streamed chat responses to use the standard Server-Sent Events (SSE) payload type (text/event-stream), improving compatibility with streaming clients.

• Tests

  • Extended the streaming media-type regression test to verify the SSE content type and cache-control: no-cache.
  • Added a version guard to skip the SSE test on Python versions below 3.10.

Walkthrough

The RAG streaming chat-completions response now declares text/event-stream instead of text/plain, with a unit test verifying the SSE media type and no-cache header.

Changes

RAG streaming responses

Layer / File(s)	Summary
Set and validate SSE response media type container-images/scripts/rag_framework, test/unit/test_rag_stream_content_type.py	The streamed response uses text/event-stream; a regression test loads the container script with stubs and verifies the media type and cache-control: no-cache header.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jhjaggars, olliewalsh, maxamillion, rhatdan, mikebonnet

Poem

A rabbit streams through fields of green,
With SSE carrots crisp and clean.
No plain-text hop, the headers gleam,
Data bounces down the stream.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: serving RAG streaming completions as text/event-stream.
Description check	✅ Passed	The description matches the change and explains the SSE content-type fix and client impact.
Linked Issues check	✅ Passed	The code and test summaries align with #2837: content-type fixed, SSE framing preserved, and non-streaming behavior unchanged.
Out of Scope Changes check	✅ Passed	The added regression test and Python-version skip guard are directly related to validating the fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request updates the media type of the StreamingResponse in the RagService from 'text/plain' to 'text/event-stream' to correctly handle streaming responses. No review comments were provided, and there is no additional feedback to offer.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[rhatdan]

@Anai-Guo are you AI or Human?

@bmahabirbu PTAL and see if this makes sense.

[mikebonnet]

@Anai-Guo Please add a test-case to validate the new behavior, and ensure it doesn't regress in the future.

[bmahabirbu]

Hmm small change make sense

@Anai-Guo would love a test case to confirm this fix adds functionality to things that need it

Im the long run serve should really be an mcp server but since this is a small change im ok as long as I can test it

[Anai-Guo]

@bmahabirbu @mikebonnet Added a regression test in test/unit/test_rag_stream_content_type.py.

Since rag_framework is a standalone container script that imports packages only present in the RAG image (fastapi, pydantic, openai, qdrant_client, uvicorn, aiohttp), the test loads it via SourceFileLoader and stubs those deps when they're absent from the unit-test env. It then calls the real _stream_completion and asserts media_type == "text/event-stream" (plus Cache-Control: no-cache). I verified it passes on this branch and fails against main (text/plain), so it guards against a regression either way.

Also restored the executable bit on the script, which an earlier revision had dropped.

[rhatdan]

@Anai-Guo "An AI" looks like you are an AI and not a Human.

[Anai-Guo]

Pushed 4aa0f43: the Python 3.9 unit-test job was failing because container-images/scripts/rag_framework uses PEP 604 unions (float | None) in Pydantic model class bodies, which are evaluated eagerly at import time and are a TypeError on 3.9. The script only ever runs inside the RAG container image (Python 3.11+), so the test is now guarded with pytest.mark.skipif(sys.version_info < (3, 10)) rather than changing the script's annotations.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/unit/test_rag_stream_content_type.py`:
- Around line 99-102: Move the rag_framework SourceFileLoader bootstrap and
exec_module call behind the Python version guard so it cannot run during
collection on Python versions below 3.10. In
test_stream_completion_uses_sse_media_type, perform the version check before
loading the framework, or add a module-level pytest skip before the bootstrap
while preserving the existing test behavior on supported versions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d6d241d7-37a6-473f-8768-a418e0058422

📥 Commits

Reviewing files that changed from the base of the PR and between cbcbd17 and 4aa0f43.

📒 Files selected for processing (1)

• test/unit/test_rag_stream_content_type.py

[coderabbitai]

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n 'SourceFileLoader|exec_module|load_module|pytest\.mark\.skipif|pytest\.skip' \
  test/unit/test_rag_stream_content_type.py
sed -n '1,110p' test/unit/test_rag_stream_content_type.py

Repository: containers/ramalama

Length of output: 3857

---

Move the RAG framework bootstrap behind the version guard.

SourceFileLoader(...).exec_module(module) runs during test collection, so Python 3.9 can hit the PEP 604 syntax in rag_framework before pytest.mark.skipif takes effect. Load rag_framework only inside test_stream_completion_uses_sse_media_type after checking sys.version_info, or add pytest.skip(..., allow_module_level=True) before the bootstrap code.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/unit/test_rag_stream_content_type.py` around lines 99 - 102, Move the
rag_framework SourceFileLoader bootstrap and exec_module call behind the Python
version guard so it cannot run during collection on Python versions below 3.10.
In test_stream_completion_uses_sse_media_type, perform the version check before
loading the framework, or add a module-level pytest skip before the bootstrap
while preserving the existing test behavior on supported versions.

Source: Coding guidelines

[olliewalsh]

@Anai-Guo are you AI or Human?

@Anai-Guo please respond to this question. You most definitely appear to be an AI agent.