Setting up external LLM and embedding models via communication protocol OpenAI

[BavariaR]

I would want to ask an extension of the configuration for the model settings to be used with embeddings and chat LLMs in such a way that you add an option under OpenAI where it would be possible to configure an own base URL just using the OpenAI communication protocol but directing the requests to some other cloud services (ex:https://api.infomaniak.com/1/ai/<PRODUCT_ID>/openai).

In my case, for example, Infomaniak in Switzerland has some local private models running which I would like to use and I could access them through the communication protocol that OpenAI has defined. But I would need to be able to change the base URL or to have an option where the configuration of the base URL is made available in the settings. I would need to be able to configure different models for the embedding and for the chat because it's two different models that I need to address in order to get things going.

Is any feature enhancement in this direction conceivable? I have difficulties using my local GPU for running the smart connections. The version 3 embeddings are simply not working, and I would like to have access to faster models which are still private but not locally hosted. This would be a workaround that I could go for.

[brianpetro]

There is a custom API adapter for the chat models, this same pattern could be used for what you're looking for with embeddings 🌴

[BavariaR]

Yeah, it would be cool if you put this on your bucket list for enhancements, especially for the embeddings, since it's really annoying that I can't use the GPU of my graphics card.