Feature request
The current implementation of the VLLMClient accepts only text inputs. However, in the VLM scenario, the input is a mix of text and images.
The current implementation raises an error if the input contains images, as it converts the input to JSON to send the request to the vLLM server.
Motivation
It is very useful to have support for image input as well, as the class name is VLLM client, not LLM.
So it makes sense to have support for multi-modal input.
Your contribution
I think it is not straightforward to implement such a feature.
Although it is easy from the client side to send the image by encoding it first so we can send it as a JSON request, it is still very challenging, I guess, to handle it from the vLLM side, and it may require some considerable adaptation from the vLLM library side.
Feature request
The current implementation of the VLLMClient accepts only text inputs. However, in the VLM scenario, the input is a mix of text and images.
The current implementation raises an error if the input contains images, as it converts the input to JSON to send the request to the vLLM server.
Motivation
It is very useful to have support for image input as well, as the class name is VLLM client, not LLM.
So it makes sense to have support for multi-modal input.
Your contribution
I think it is not straightforward to implement such a feature.
Although it is easy from the client side to send the image by encoding it first so we can send it as a JSON request, it is still very challenging, I guess, to handle it from the vLLM side, and it may require some considerable adaptation from the vLLM library side.