Add initial support for images in the ai chat

[eneufeld]

What it does

Initial implementation of #15407

How to test

Copy an image into the clipboard. Paste it using CTRL+V.

Follow-ups

• Untested and probably missing support for Drag&Drop (2.2)
• Missing support for Upload via Button (2.3)
• Missing support to select from the workspace (2.4)
• Missing feedback if selected LLM does not support image data (6)

As this is the first introduction of non text data to the request, it must be decided where and how this kind of data should be stored. All adaptations on the communication from input widget to the llm must be reviewed and discussed also in regard to more kind of input data (e.g. audio).

Breaking changes

• This PR introduces breaking changes and requires careful review. If yes, the breaking changes section in the changelog has been updated.

Attribution

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[planger]

This is awesome! Thank you!

I'm wondering whether we could integrate the handling of images with the existing context concept and extend it with images, rather than doing it alongside it? Conceptually it would feel like a good fit, alongside other context, like files.

[eneufeld]

yes I thought about that. Basically we need to decide how we want to handle non message data, eg images and audio. as part of the request or just as "additional information" and thus part of the context?

[JonasHelming]

This is awesome! Thank you!

I'm wondering whether we could integrate the handling of images with the existing context concept and extend it with images, rather than doing it alongside it? Conceptually it would feel like a good fit, alongside other context, like files.

Maybe even both should be possible?

[planger]

I tested this PR a bit more and I think it is a great feature that I really would like to get in!

Besides cleaning up the change (some unnecessary comments, out-commented code and formatting changes), I'm mostly thinking about how to best represent images in the chat session and in the UI. Currently it is added orthogonal to all available infrastructure we have.

I'm wondering whether we couldn't just reuse the file context variable or create an image context variable adjacent to it. This would give us the UI more or less for free:

• The drag'n'drop support to add images
• The + button support to add images
• Show images just like files in the chat input if they have been attached with the existing context variable UI

A minor disadvantage would be that we wouldn't show a preview of the image in the chat input. But I'm not sure this is a big deal.

We could still handle the images in the generic chat agent code (like now in this PR) but just filter out all image context variables from the context instead of reading the new request.images. I like this as it is customizable by concrete chat agents. It might be worth though to give the getMessages method access to the language model that is being used and add a property to it whether it even supports images. If it doesn't, we could avoid creating the image messages and show a warning to the user?

What do you think?

Great work with this PR, in any case!!! Looking forward to having this available in Theia AI!

[eneufeld]

So my thoughts about the context and why I didn't add it directly to it:

• In my understanding the context is used to provide information to resolve variables/placeholders of the text.
• An image and in the future audio input is in my opinion an integral part of the user request. It can even be the only thing you send to the llm, eg if the image contains a screenshot of your console. Doing so by using the context sounds weird to me.
• On the other hand it is nice to reuse all the other features 🤷

Yes I agree language models need a possibility to provide a feature list so that we can check what is supported.

[planger]

@eneufeld Thanks for the feedback! That makes sense. The way we use context so far is indeed a bit different in semantics, especially when thinking about audio. For images, though, I feel the advantages of reusing the existing infrastructure outweigh the semantic mismatch.

For audio, I agree—it’s semantically equivalent to the user input, like a spoken prompt. But for images and files, they usually augment the input. Sure, it may also make sense to send an image without a user prompt (spoken or written). Actually, this may actually also be true for any file, e.g. when I copy a log file and want to send it to the LLM without writing a prompt. I don't see it as part of this PR, but we may want to consider enabling the send button if there is at least one context element (image or file).

All in all, going the context-element route seems reasonable to me—especially as it simplifies integrating the UI behaviors (e.g. Drag&Drop, + button, consistent view in the chat input). But I’m totally open to your thoughts here and happy to go with whatever direction you feel is more robust long term.

Thank you!

[eneufeld]

I totally see the advantages of the context. We should just define a consistent logic for us what goes into the context (and thus what it is) and what stays outside of it.

@sdirix any ideas here

[sdirix]

Images and Voice are transferred to the LLM via separate messages within the request. They are high priority items the user will always want to be included. Modeling these as additional fields on a request makes sense to me.

The way the context is currently set up, it's optional content. The only way the LLM even knows about the context is by adding the context variable and/or the context functions deliberately to the prompt. Even then, an LLM still needs to perform tool calls to retrieve the content and might just ignore it. Therefore this does not fit to the deliberate copy of images or voice input. We still might want to (auto-)populate the context with images and audio later on for the LLM to offer for optional consumption, but this is not the current use case.

[martin-fleck-at]

I rebased the change and moved the image property to the context. As a follow-up we definitely want to support that images can be handled the same way other context variables (tasks, files) are handled at the moment. Still I think this adds a valuable feature and a good baseline for image support.

[planger]

Thanks, looks great to me! 👍 Excellent work.

[xpomul]

@eneufeld Is it possible that this change breaks pasting text into the chat with the CTRL+V shortcut? It was working after pulling this noon, but seems to be broken after a pull just now… 😕

[martin-fleck-at]

@xpomul Unfortunately, it does! There is a fix in https://github.com/eclipse-theia/theia/pull/15765/files#diff-7b86f78bfb9759d841ed4e6a074768ed761c0f18079b23dc1d705566bb8bcaab but if it is urgent and the PR review takes a longer time, we can pull the fix out.

[xpomul]

@martin-fleck-at ah thanks. Fine for me. I just wanted to give a heads-up so that it does not go unnoticed :)