feat: pass an image as part of the evaluation

[giladgd]

When llama.cpp's support for this will be stable.
Hopefully, there will be an official API for this after ggml-org/llama.cpp#9643 is implemented.

[samlhuillier]

Interested in this kind of multimodal support. Any update on progress?

[fozziethebeat]

Does this encompass adding support for running llava models or should that be a separate feature request? I noticed that llama-cpp-python already includes llava support from llama.cpp so this shouldn't be too hard with setting up the bindings.

[giladgd]

I haven't started working on this yet, but it is planned as part of the roadmap.
The plan is to add support for llama.cpp's ability to pass an image to a model, which right now only supports LLaVA.

I'll work on this once llama.cpp's API for this is final, to prevent frequent breaking API changes (unlike what happens on some other libraries)

[fozziethebeat]

Make sense. Hopefully llama.cpp finalizes that API.

[AlexTech314]

Any update on this? Would love to leverage multimodal models! Love the library so far :)

[giladgd]

An official API for this is in active development on llama.cpp; one it's ready we can start working on adding support for it in node-llama-cpp.
It appears that the API being worked on will also include support for other modalities, such as audio and video, so this is going to be a major feature once it lands (node-llama-cpp will support everything).

[AlexTech314]

An official API for this is in active development on llama.cpp; one it's ready we can start working on adding support for it in node-llama-cpp.
It appears that the API being worked on will also include support for other modalities, such as audio and video, so this is going to be a major feature once it lands (node-llama-cpp will support everything).

Beautiful. Looking forward to it, this library is insane.

[wisng]

Any updates on this feature, it seems there were some experimental support for gemma 3 vision last week ggml-org/llama.cpp#12344