vLLM compatibility ?

[naatyu]

Hi,

I currently use vLLM for other services, and I am deeply interested in connecting your extension with a vLLM server. Do you think it's possible? I'm using the OpenAI API format, so if there is a possibility to connect the extension to any server using the OpenAI API, that would be great.

Thanks for the awesome work!

---

Please reply with a 👍 if you want this feature.

[wsxiaoys]

We explored the direction but ultimately decided against pursuing it. This decision was influenced by the fact that most OpenAI-like solutions lack control over decoding steps, a crucial component that could use code completion specific optimization. For instance, handling long lists of stop words and applying grammar constraints for each decoding step becomes challenging.

Revisiting this approach might be viable if a decoding step-level API becomes widely adopted in the future.

[d0rc]

Do you think it's so hard and complex so it's not even a thing to formulate it as protocol? Like Tabby Inference Protocol, which should be either supported, or inference engine is not compatible...? If you will pick the protocol which is ok for your code, it shouldn't be hard to support it, and even for Tabby's codebase in a long run it may become beneficial.

[wsxiaoys]

:) It's not so much about complexity as it is about capability. With an interface like OpenAI, we've relinquished control over accessing intermediate decoding steps. Many optimizations, upon which the current tabby relies, cannot be easily implemented—for instance, a lengthy stop words list.

[d0rc]

I see no problem implementing even very long stop list dictionary in my setup, even long list of long stop words with optimal stopping/lookup, I've been through "a lot" with open models😂

[d0rc]

I can even provide extended methods support, like forcing response to be json etc.

[naatyu]

Thank you for the explanations

We explored the direction but ultimately decided against pursuing it. This decision was influenced by the fact that most OpenAI-like solutions lack control over decoding steps, a crucial component that could use code completion specific optimization. For instance, handling long lists of stop words and applying grammar constraints for each decoding step becomes challenging.

Revisiting this approach might be viable if a decoding step-level API becomes widely adopted in the future.

Thank you for the explanations

[sundaraa-deshaw]

Continuing from #854. Currently Tabby supports llama.cpp bindings and http bindings to vertex-ai and fastchat. Are there plans to support other bindings like OpenAI endpoints via HTTP or similar protocol.?

Thanks for the responses.

[wsxiaoys]

Hey @sundaraa-deshaw , #795 (comment) explains the reason that why we don’t want a openai like http interface

[sundaraa-deshaw]

Thanks, I was wondering if we can have a binding to exllama[v2] inference engine like how it is done for llama.cpp today?

[wsxiaoys]

That’s possible - the trait is defined at https://github.com/TabbyML/tabby/blob/main/crates/tabby-inference/src/lib.rs

Could you share some of your findings that exllama has advantage against llama.cpp?

[sundaraa-deshaw]

Thanks, are there plans to add such a binding?

exllama turned out to be good for inference on GPU, compared to llama.cpp on CPU.

The memory usage for a GPTQ quantized model was 2-3x less than running the non-quantized model (Llama 13B) on llama.cpp on GPU.
The performance (in terms of tokens/second) was 1.5-2x higher compare to llama.cpp.

[r7l]

Since Tabby seems to supports Fastchat, would it be possible to support Ollama HTTP bindings? They have a decent list of integrations already. Ollama is also using llama.cpp under the hood.

[wsxiaoys]

Fastchat isn't supported; it's a part of the exploration mentioned in an earlier reply and was eventually abandoned due to the reasons discussed above (lack of control during decoding).