llm-cache

Production LLM call layer for AI agents and tools: keep OpenAI/Anthropic/AI SDK/LiteLLM, hot-swap models with MDA presets, and add cache, retries, circuit breakers, key rotation, singleflight, and Python/TypeScript/Rust parity.

Drop-in OpenAI-compatible cache that replays repeated computer-use and agent workflows at near-zero cost.

Unified multi-layer caching library for AI/agent pipelines — LangChain, LangGraph, AutoGen, CrewAI, Agno, A2A

Redis Vector Library (RedisVL) -- the AI-native Java client for Redis.

Context-aware caching proxy for Anthropic and OpenAI — maximizes prompt-cache hit rates by anchoring long-lived context into stable boundaries. Drop-in for Claude Code, Codex, Cursor, and custom harnesses.

Read-only performance diagnostics for DeepSeek Harness: session load/restore timing, spill-hit counts, compaction count and trigger, context-injection volume (AGENTS.md/skills/tool-schema token share), and LLM cache hit rate — surfaced via /fast, persisted as reconstructable session events with async sampling off the model path.

Python implementations, C++ structures, and CUDA GPU kernels that demonstrate the core algorithms, memory allocators, quantization schemes, disaggregation transports, and latency profilers used in modern LLM serving engines like vLLM, SGLang, TensorRT-LLM, llama.cpp, NVIDIA Dynamo, and LMCache.

VCAL Core — high-performance semantic cache and vector cache library for LLM applications.

The LLM call framework. Resilience, observability, and control for AIs.

Deterministic LLM caching layer with context optimization, LRU eviction, model-aware token counting, and concurrency-safe request deduplication.

Five-decimal cache-hit readout beside the chat stats line for the DeepSeek Harness Web GUI (dsh-plugin)

Adaptive semantic cache for LLMs with streaming support, ML-based thresholds, and real-time cost tracking. Built in Rust for sub-millisecond performance.

⚡ Zero-latency semantic caching, agent loop accelerator & cost optimization gateway for LLMs. Slash OpenAI & Anthropic bills by 40%-75%.

Reduce LLM API costs and speed up responses by caching completions with NeuroCache’s intelligent, provider-agnostic caching layer.

Chengeta AI is a unified memory and caching platform for AI agents, workflows, and autonomous systems.

CacheEval is a prompt level caching accuracy evaluation system which is intended to check the accuracy and performance of a prompt caching system like GPTCache etc.

Something related to LLM caching

Semantic caching layer for LLM calls. Exact-match and embedding-similarity caching with model-version-aware invalidation, use-case segmentation, and cost-saved tracking. Adapters for Redis and DynamoDB.

Pre-indexed code knowledge graph for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local

Correctness-aware semantic cache for LLM responses, built for regulated workloads where a wrong cache hit is a defect. Zero-dependency Go: similarity floor + verification, PII-aware keying, knowledge-epoch staleness, negative cache.