Adding vLLM, PostgreSQL/pg_vector, and UV

[thesteve0]

Hey folks - I just spun up the default Allycat with Docling and several different Open Weights models. Kudos to those who have contributed before - basically worked right out of the box and nice and simple flow. Also love that you also included a pure code version for every jupyter notebook.
Couple of questions:

1. I found some typos or instructions that could be updated. Should I just PR those?
2. I want to add using vLLM rather than Ollama to Allycat as the inference server. Once I do that I would like to feed that back into the project.
   a. Is that welcome or should I just leave it in my own fork?
   b. If it is welcome, how should I approach contributing it back?
3. Same request again but this time using PostgreSQL pg_vector rather than Milvus.
4. How are people feeling about converting this to a UV project for dependency management? https://docs.astral.sh/uv/

[thesteve0]

@sujee please feel free to tag anyone else in that you think could help with this discussion

[nat3058]

Hi Steve,

I am also working on adding vLLM support as a contribution and it seems non-trivial to support (especially for CPU-only machines).

Were you able to add vLLM to Allycat? Also, is there a specific reason for using vLLM instead of Ollama (perhaps you want local GPU inference)? Ollama seems to be pretty good for inference on local CPU-only machines (correct me if I am wrong).

I have looked into adding vLLM and have noted my thoughts here. Any help is appreciated.

[thesteve0]

Hey Nat
I did do this, along with some other refactoring, in a separate fork but in the meantime AllyCat moved to a new framework making the integration much easier
https://github.com/thesteve0/allycat/tree/vllm-update-rebased

I will be adding vLLM to the newer version of AllyCat soon

I work at Red Hat and one of my tasks is the vLLM community work. I chose it because I thought it would be a good addition to show alternates to model serving for inference. It is not that hard to start for CPU only machines but its use case is primarly for multi-user serving. If you just want to run something locally on your machine then ollama is a simpler solution.