Questions about API/ABI stability

[ckastner]

Hi,

with semantic versioning in place, and also the recent addition of a SOVER, I have a few questions that are relevant to some distributions. For example on Debian we've so far shipped the libggml in a private directory because the public locations (/usr/lib etc.) have stability requirements. With these recent improvements, we think we can move this into a public location now.

1. What would you consider the public interface?

   Asking so that we can adapt our automated checks for API/ABI breakage (abi-compliance-checker, abigail-tools). For example, libggml-base.so is not part of this public interface, right? As in: it's used by libggml.so, not by users directly, so any breakage there is "internal" and would not require a SOVER bump

2. What about the interface of the backends? Also, is this dependent on GGML_BACKEND_DL=ON/OFF?

   I think this one is a bit tricky. For example, what happens when libggml-cpu.so has a breaking change? Would this be an internal change, or a public one? If public, would libggml-cpu.so get its own SOVER bump, or would all of ggml get a SOVER bump? (AFAICT, there is currently one SOVER project-wide)

3. Extending the previous questions: Even taking the libggml.so public, we'd still keep the backends like (libggml-cuda.so) in a private directory. Do you think this is wrong? Should these, be public, too?

   Can these backends have any standalone value, that would make sense in a public directory? Or are they "only" to be accessed through libggml{,base}.so?

[ggerganov]

For example, libggml-base.so is not part of this public interface, right?

Yes, libggml-base.so should not be considered public.

What about the interface of the backends?

Similarly, 3rd party projects should not link directly to backend libs. I think the only public binary should be libggml.so and nothing else.

Regarding the interfaces - everything in ggml/include is part of the public interface, so we have to bump the ggml version to reflect changes to these headers. I think it is fine to have all those interfaces under the same SOVER.

Also, is this dependent on GGML_BACKEND_DL=ON/OFF?

I think for package distribution we always want to have GGML_BACKEND_DL=ON?

[ckastner]

Great, thanks for clearing it up. Some replies below, but feel free to close the issue.

On a side note, is a 0.9.5 release on the radar? It would be needed to build a newer llama.cpp, because it needs GGML_ROPE_TYPE_IMROPE.

I think for package distribution we always want to have GGML_BACKEND_DL=ON?

Indeed, and we have individually installable packages for each of those backends. This is actually where my questions come from, because we need to get the inter-package dependencies right, to guarantee an always clean upgrade path in any given scenario.

For example, libggml.so.0 is stable. In Debian, we ship it as a standalone library. So a llama.cpp built against libggml.so.0 version 0.9.4 will also work with libggml.so.0 from 0.9.5, because 0.9.5 is backwards-compatible to 0.9.4.

And our tooling automatically takes care of these things. What we do additionally is track the symbols, as a double-check of upstream (mistakes can happen). For that, I just needed to know what part of the libraries are seen as the public contract, which is now clear to me.

Regarding the backends, they also have a SOVER, suggesting also some kind of public contract. But that felt like perhaps a step too far (they felt private), so to err on the side of caution, we treated them as private. And we currently specify the backend relationship as "backend must come from the exact same build that libggml.so came from". If I understood you correctly, this approach does not conflict with the way you'd like libggml used, so we can continue this.

[ggerganov]

Yes, I think what you have explained makes sense. Mind you, I am a bit uncertain myself about some of these points. But as soon as we hit any problem with the existing contract/method, we will adapt. Feedback is very appreciated and this is quite useful already. Thanks.