Uv & Apertus support

[loleg]

I'm using the uv package manager to run my project locally, and experiment with the Apertus model - which I have running fine in vLLM and Ollama, using the same quantizations as your Llama defaults.

With the current release version of llama-cpp-python I get this error:

RuntimeError: Internal C++ object (PySide6.QtNetwork.QNetworkReply) already deleted.
Exception ignored in: <function LlamaModel.__del__ at 0x7f612e5d0a40>
Traceback (most recent call last):
  File "OKFN/opendataeditor/.venv/lib/python3.12/site-packages/llama_cpp/_internals.py", line 78, in close
    if self.sampler is not None:
       ^^^^^^^^^^^^
AttributeError: 'LlamaModel' object has no attribute 'sampler'
Traceback (most recent call last):
  File "/home/oleg/Localdev/OKFN/opendataeditor/ode/main.py", line 820, in on_llm_init_error

Even after bumping to a patched source version of the library, it is not yet working. But I'm going to try again once there's an October release.

[pdelboca]

Thanks for this! We're excited to test the Apertus model! It is definitely a nice replacement for Llama :D

I have also in my todo list to add uv support since the Makefile is too linux oriented and uv is just great.

[loleg]

Tested again yesterday, still no official update on llama-cpp-python. Works fine with the latest PR.

[loleg]

@pdelboca do you have thoughts on how to support multiple models, at least the use of an alternative model via environment variable?

[pdelboca]

@loleg We used to have a dialog to select between different models. It should be fairly easy to restore it. Here is the PR in which we simplified the feature: https://github.com/okfn/opendataeditor/pull/1051/changes. The "backend" supports it, it just a matter of creating the UI interactions and adding some help text to guide the user on which to choose and why.

Some context

Supporting multiple models was removed for simplification and because it caused confusion on users regarding which one was best, what to choose, why is there an option, they didn't understand the concept of model and why they need to download, etc. Based on that experience we decided that for the user base of ODE having only one model is the best UX. We are more than happy to default to Apertus as it aligns perfectly with the tool and the philosophy.