Posting content SSE conform with bentoml

[bilaljo]

Hello,

So far, I have found that it is quite easy to realise streaming in Bentoml while yielding the outputs. However, I am wondering how I could achieve this SSE-conformant streaming, i.e. how it sends the JSON in style.

data: some_data\n\n

Simply building this as a string didn't work as it was not a valid base model. I have seen older pull requests on this topic where it was achieved with runners, but I have since found that this is already outdated.

Thanks in advance.

[xXMrNidaXx]

SSE streaming with BentoML can be tricky. At RevolutionAI (https://revolutionai.io), we have implemented streaming endpoints for LLM responses.

Working pattern:

import bentoml
from starlette.responses import StreamingResponse

@bentoml.service
class LLMService:
    @bentoml.api
    def stream_generate(self, prompt: str):
        async def generate():
            for token in model.generate_stream(prompt):
                yield f"data: {token}\n\n"
        return StreamingResponse(generate(), media_type="text/event-stream")

Key points:

• Use StreamingResponse from starlette
• Format as SSE: data: {content}\n\n
• Set correct media_type

What specific SSE behavior are you trying to achieve?

[xXMrNidaXx]

SSE with BentoML works great! At RevolutionAI (https://revolutionai.io) we use it for streaming LLM responses.

Quick pattern:

import bentoml
from starlette.responses import StreamingResponse

@bentoml.service
class LLMService:
    @bentoml.api
    async def stream(self, prompt: str):
        async def generate():
            for token in llm.stream(prompt):
                yield f"data: {token}\n\n"
            yield "data: [DONE]\n\n"
        
        return StreamingResponse(
            generate(),
            media_type="text/event-stream"
        )

Tips:

• Set Cache-Control: no-cache
• Use text/event-stream content type
• End with [DONE] marker

Works with OpenAI-compatible clients!

[rehan243]

So I actually dealt with this when implementing a streaming endpoint using BentoML. I was trying to get it to work with SSE (Server-Sent Events) and it wasn't straightforward, took me a few tries to get right. The issue is that BentoML doesn't natively support SSE out of the box, but you can achieve it by manually formatting your output. You're on the right track with the data: some_data\n\n format, that's the standard SSE format.

The problem with building it as a string is that BentoML expects a certain output format. What worked for me was using a generator function that yields the SSE-formatted strings. Here's a simplified example:

def sse_generator(data):
    for item in data:
        yield f"data: {json.dumps(item)}\n\n"

# in your service
async def my_service(data):
    return sse_generator(data)

In my case, I was using bentoml.Service with a runner that handled the data processing. Turns out, you can directly yield the SSE-formatted output from your service function. Fwiw, I was using uvicorn as my ASGI server, not sure if that makes a difference.

Are you using a specific runner or just the basic bentoml.Service? Curious to hear if this helps with your implementation.