Streaming chat endpoint

[TamiTakamiya]

Jira Issue: https://issues.redhat.com/browse/AAP-39044

Description

The implementation of /streaming_chat endpoint.

Testing

Steps to test

1. Pull down the PR
2. Run unit tests

Scenarios tested

Production deployment

• This code change is ready for production on its own
• This code change requires the following considerations before going to production:

1. The streaming API returns almost immediately and actual chat data are sent through the stream, which is returned with API. It is likely we need to manage and limit the number of concurrent streams. This PR does not contain that logic.
2. An operational telemetry event (StreamingChatBotOperationalEvent) is implemented in this PR, but it is sent right after the API is returned, i.e. chat data are not received yet. We need to design more meaningful telemetry for the streaming API.

[manstis]

I had a peek @TamiTakamiya

I don't think we need the async_invoke function as you've now moved streaming support to a new type of pipeline (ModelPipelineStreamingChatBot).

[manstis]

Other than the selftest errors; LGTM 👍

I made a comment about where HTTP exception handling should possibly happen.

IDK if you're able to test both where you have exceptions handled (e.g. backend disconnects or something) vs where I've proposed it.

[romartin]

@TamiTakamiya that's an AWESOME work! congrats! 👏

I just added a few comments, but mostly about Q's I have, I think the code changes are looking good , also tests.

Just another Q - If I want to test it, I don't see any change in this PR on the chatbot UI app. I understand this is because we are just enabling the backend side of it, but for now not allowing end users to stream. Anyway, if at some point we want to enable streaming, at least in local env, it is just about changing the endpoint in the useChatbot.tsx, something else? Worth documenting this in a quick README?

Great great stuff, thanks!

[romartin]

Assuming we are just using the same chatbot event structure for when streaming, as when non-streaming, correct?

It is absolutely fine, but just asking, should I make sense to collect any other chatbot attribute of interest only when streaming? Just brainstorming, because as far as I remember, when playing locally with Ollama, for example, you get, for every stream data, also metadata associated for it, as well as some other metadata at the end of the stream response.

[TamiTakamiya]

The STREAM_DATA used in test_pipeline.py was taken from an actual log of ansible-chatbot-service. As you see in the first start data, conversation_id is sent with the start data. For the first API call, chatbot UI issues /chat or /streaming_chat API without specifying conversation_id expecting chatbot service will assing a new conversation_id. In such case, conversation_id is not found in the API response and therefore it won't appear in the operational event as well.

So I think at least we want to add two more operational event types that correspond to the start and end data.

On Ollama, I also saw some data is received from ollama server at the end of stream. There might be some missing parts in the current road-core/service streaming API implementation.

[manstis]

This class is needed. It becomes the fallback implementation should one not be provided. For example, you configured the ChatBot to use wca as it's provider in the JSON configuration. We only have a concrete implementation for http.

[TamiTakamiya]

@manstis Thank you for correcting my misunderstanding and provide clarification.

[manstis]

IDK why my comment appeared here 🤣

It was meant for here.

[romartin]

Just due to my ignorance... :) why do we need CSRF trusted origins? Why are they used for?

[TamiTakamiya]

This is what I do not understand 100% yet. It is not needed when the AI Connect server is started from source, which uses Daphne for all endpoints. When the server is loaded from the image with podman-compose, it is needed. I think it may be because we are using Daphne only for the /streaming_chat endpoint and others are processed with uwsgi, i.e. when the chatbot page is loaded, all HTML/Javascript/css, etc. are loaded through uwsgi, and the streaming_chat request goes to Daphne. I guess it breaks something on CSRF code used by Django, but I have not clearly figured it out yet.

[manstis]

A few minor comments on this iteration @TamiTakamiya but looking good.

I have not looked at the concerns reported by @romartin or @jameswnl. I'll leave those to them.

[manstis]

LGTM 👍

I didn't consider @jameswnl and @romartin comments.. you may want to get their approval too.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[TamiTakamiya]

@jameswnl Thank you for your approval!! I opened https://github.com/ansible/ansible-wisdom-ops/pull/1255 to set the CSRF_TRUSTED_ORIGINS envvar, which was added with this PR, in stage/prod clusters and the ops PR has to be merged before this is merged so that streaming char functions correctly. Would you take a look? Thanks!