Skip to content

fix: handle llama_cpp/llguidance version incompatibility gracefully#1452

Open
octo-patch wants to merge 1 commit into
guidance-ai:mainfrom
octo-patch:fix/issue-1384-llama-cpp-graceful-import
Open

fix: handle llama_cpp/llguidance version incompatibility gracefully#1452
octo-patch wants to merge 1 commit into
guidance-ai:mainfrom
octo-patch:fix/issue-1384-llama-cpp-graceful-import

Conversation

@octo-patch

Copy link
Copy Markdown

Fixes #1384

Problem

When llama_cpp is installed but at a version incompatible with llguidance.llamacpp
(e.g. llama-cpp-python==0.2.x with llguidance>=1.5.0), the module-level
import llguidance.llamacpp raises an AttributeError (e.g. module 'llama_cpp' has no attribute 'llama_vocab_p') that crashes the entire guidance import.

This makes the library completely unusable even for users who are not using LlamaCpp
at all, since guidance/models/__init__.py imports from _llama_cpp.py at module level.

Verified on: llama-cpp-python==0.2.7 + llguidance==1.5.0:

ImportError while loading conftest ...
guidance/models/_llama_cpp.py:25: in <module>
    import llguidance.llamacpp
AttributeError: module 'llama_cpp' has no attribute 'llama_vocab_p'

Solution

Consolidate both import llama_cpp and import llguidance.llamacpp into a single
try/except block that catches ModuleNotFoundError, ImportError, and
AttributeError. When either package is missing or incompatible, is_llama_cpp
is set to False and the rest of guidance loads normally.

Users who attempt to instantiate LlamaCpp will still receive a clear error message
at instantiation time (the existing check at the top of LlamaCppEngine.__init__).

Testing

  • Verified import guidance succeeds with llama-cpp-python==0.2.7 installed
  • Unit tests pass: pytest tests/unit/library/test_gen.py tests/unit/test_decorator.py
    (44 passed, 1 xfailed)

When llama_cpp is installed but at a version incompatible with
llguidance.llamacpp (e.g. llama-cpp-python 0.2.x with llguidance 1.5+),
the module-level import llguidance.llamacpp raised an AttributeError
that crashed the entire guidance import, making the library unusable
even for users not relying on LlamaCpp.

This change consolidates both imports into a single try/except block
that catches ModuleNotFoundError, ImportError, and AttributeError,
setting is_llama_cpp=False when either package is unavailable or
incompatible. Users who attempt to use LlamaCpp will still receive a
clear error message at instantiation time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Some LlamaCpp models incoherent in Guidance, works fine when using llama-cpp-python directly

1 participant