Submission checklist
Package (Required)
Related Issues / PRs
No response
Reproduction Steps / Example Code (Python)
import asyncio
from langchain_core.caches import RETURN_VAL_TYPE, BaseCache
from langchain_core.language_models import FakeListLLM
class SizedCache(BaseCache):
"""A cache that reports its entry count via __len__ (a common pattern)."""
def __init__(self) -> None:
self._cache: dict[tuple[str, str], RETURN_VAL_TYPE] = {}
def lookup(self, prompt, llm_string):
return self._cache.get((prompt, llm_string))
def update(self, prompt, llm_string, return_val):
self._cache[prompt, llm_string] = return_val
def clear(self, **kwargs):
self._cache = {}
def __len__(self) -> int:
return len(self._cache)
# Empty cache -> len() == 0 -> falsy
llm = FakeListLLM(cache=SizedCache(), responses=["foo", "bar"])
# Both of these raise KeyError: 0
llm.generate(["hello"])
asyncio.run(llm.agenerate(["hello"]))Error Message and Stack Trace (if applicable)
Traceback (most recent call last):
File "repro.py", line 31, in <module>
llm.generate(["hello"])
File ".../langchain_core/language_models/llms.py", line 1070, in generate
generations = [existing_prompts[i] for i in range(len(prompts))]
File ".../langchain_core/language_models/llms.py", line 1070, in <listcomp>
generations = [existing_prompts[i] for i in range(len(prompts))]
KeyError: 0Description
The cache lookup/update helpers in langchain_core.language_models.llms decide whether a cache is configured using a truthiness check rather than an identity check:
get_prompts (sync lookup): if llm_cache:aget_prompts (async lookup): if llm_cache:aupdate_cache (async write): if llm_cache:
(update_cache, the sync write, already correctly uses if llm_cache is not None:.)
In get_prompts / aget_prompts that check guards the entire lookup loop body. For a BaseCache that implements __len__ (which many real caches do, to report their entry count), an empty cache is falsy, so the loop body is skipped. The prompt is then added to neither existing_prompts nor missing_prompts — it simply disappears. Generation is skipped because there are no missing prompts, and the final generations = [existing_prompts[i] for i in range(len(prompts))] raises KeyError.
This happens on both generate and agenerate, so caching is effectively unusable (it hard-crashes) for any cache that implements __len__ and starts empty.
I expect: an empty-but-present cache should be treated as present, exactly like the non-empty case — the call should run normally and populate the cache.
Instead: KeyError: 0.
The fix is to use is not None in the three places above (matching update_cache). Happy to open a PR with a regression test if a maintainer can assign me.
System Info
langchain-core==1.4.8 (also reproduces on current main)
Python 3.10
Submission checklist
Package (Required)
Related Issues / PRs
No response
Reproduction Steps / Example Code (Python)
Error Message and Stack Trace (if applicable)
Description
The cache lookup/update helpers in
langchain_core.language_models.llmsdecide whether a cache is configured using a truthiness check rather than an identity check:get_prompts(sync lookup):if llm_cache:aget_prompts(async lookup):if llm_cache:aupdate_cache(async write):if llm_cache:(
update_cache, the sync write, already correctly usesif llm_cache is not None:.)In
get_prompts/aget_promptsthat check guards the entire lookup loop body. For aBaseCachethat implements__len__(which many real caches do, to report their entry count), an empty cache is falsy, so the loop body is skipped. The prompt is then added to neitherexisting_promptsnormissing_prompts— it simply disappears. Generation is skipped because there are no missing prompts, and the finalgenerations = [existing_prompts[i] for i in range(len(prompts))]raisesKeyError.This happens on both
generateandagenerate, so caching is effectively unusable (it hard-crashes) for any cache that implements__len__and starts empty.I expect: an empty-but-present cache should be treated as present, exactly like the non-empty case — the call should run normally and populate the cache.
Instead:
KeyError: 0.The fix is to use
is not Nonein the three places above (matchingupdate_cache). Happy to open a PR with a regression test if a maintainer can assign me.System Info
langchain-core==1.4.8 (also reproduces on current
main)Python 3.10