Awesome AI Gateways

A curated list of tools, libraries, and resources for LLM routing, orchestration, and gateway infrastructure.

Managing 100+ LLM APIs, handling rate limits, implementing fallbacks, and tracking token costs is a massive operational headache. This list curates the best open-source tools, managed services, and resources to help you build production-grade AI infrastructure.

Contents

• AI Gateways
• Semantic Caching
• Cost Tracking and Analytics
• Load Balancing and Routing
• API Management and Rate Limiting
• Observability and Logging
• Agentic Orchestration
• SDKs and Libraries
• Guardrails and Security
• Tutorials and Case Studies
• Communities

AI Gateways

Unified API gateways that route requests across multiple LLM providers.

Open Source

• Kong AI Gateway Plugin - Kong plugin for LLM provider integration with request/response transformation. • Traefik Forward Auth for AI - Use forward auth middleware for AI-based authentication and authorization. • Traefik AI Plugin - Traefik plugin for AI-based request routing and middleware. • Traefik Pilot AI - AI-driven configuration and optimization for Traefik proxies. • APISIX AI Plugin - Apache APISIX plugin for LLM provider integration with load balancing and observability. • Ambassador AI Gateway - Kubernetes-native AI gateway built on Envoy with rate limiting and fallback.

• LiteLLM - Python SDK and proxy server calling 100+ LLMs in OpenAI format.
• Portkey Gateway - Blazing fast AI gateway with 250+ LLMs, 50+ guardrails.
• Bifrost - High-performance AI gateway in Go with adaptive load balancing.
• Ferro Labs AI Gateway - Go-native gateway for 29 providers with caching & guardrails.
• Envoy AI Gateway - Unified access to GenAI services built on Envoy Gateway.
• LLM Gateway - Unified interface for running and managing LLMs with analytics.
• Inference Gateway - Cloud-native gateway unifying multiple LLM providers.
• OpenGateLLM - Open-source API gateway focused on self-hosted LLMs.
• Routerly - Self-hosted LLM gateway with intelligent multi-policy routing.
• Plexus - Unified API gateway with OAuth auth, quota tracking, and 15+ providers.
• OpenZiti LLM Gateway - Zero-trust LLM gateway with semantic routing and E2E encryption.
• Kong - Enterprise API gateway with LLM routing plugin.
• LocalAI - Self-hosted, drop-in replacement for OpenAI API.
• lm-proxy - Lightweight OpenAI-compatible proxy for multi-provider inference.
• LLM API Proxy - Edge runtime proxy supporting OpenAI, Anthropic, Gemini, and more.
• LLMProxy - High-performance reverse proxy for LLM inference with SSE streaming.
• Zuul - Netflix edge gateway.
• RouteLLM - Open-source framework from LMSYS for training and deploying lightweight LLM routers that balance cost and quality.
• Manifest - TypeScript gateway connecting agents with any provider, with cost optimization and BYOK support.
• ClawRouter - Agent-native LLM router with 41+ models, sub-millisecond routing, and USDC payments.
• NadirClaw - Self-hosted LLM router that routes simple prompts to cheap models and complex to premium, saving 40-70% on costs.
• Olla - High-performance lightweight Go proxy and load balancer for LLM infrastructure with intelligent routing.
• BitRouter - Rust-based agentic LLM gateway and router for cost-optimizing agentic workflows.
• Nyro - Self-hosted Rust AI gateway with protocol translation for Claude Code, Codex, and Gemini CLI.
• 1Flowbase - Rust AI gateway for fusion-style multi-model workflows published as OpenAI-compatible virtual models.
• SmarterRouter - VRAM-aware router for Ollama and llama.cpp with semantic caching, model profiling, and automatic failover.
• Llamactl - Unified management and routing for llama.cpp, MLX, and vLLM models with web dashboard.
• Nexus Gateway - Enterprise AI traffic gateway with compliance, routing across 20+ providers, semantic cache, quotas, and audit.
• AI Firewall - Rust OpenAI-compatible gateway using Redis exact cache and Qdrant semantic cache.
• Hecate - Local AI operations console for supervised agent work with MCP and OpenTelemetry.
• Alvus - Lightweight Go proxy for handling AI API rate limits.

Managed Services

• OCI AI Services API Gateway - Oracle Cloud Infrastructure AI services exposed via API Gateway. • Alibaba Cloud API Gateway for AI - Alibaba Cloud's API gateway with AI capabilities for model serving. • Azure API Management with Cognitive Services - Integrate Azure Cognitive Services with API Management for AI-enhanced APIs. • IBM API Connect AI - Use AI for API lifecycle management and security in IBM API Connect. • AWS API Gateway with Bedrock Integration - Expose Bedrock models via REST/WebSocket APIs with throttling and auth. • GCP Apigee AI Gateway - Manage and secure LLM APIs with Apigee's traffic management and analytics.

• Vercel AI Gateway - Single endpoint for hundreds of AI models.
• Cloudflare AI Gateway - Unified interface for AI providers at the edge.
• OpenRouter - Unified API for 500+ models from 60+ providers.
• Portkey Hosted - Managed AI gateway with 1600+ LLMs and enterprise features.
• Braintrust - Unified API with encrypted caching and integrated evaluation.
• Inworld Router - LLM plus TTS pipelining for high-interactivity use cases.
• Maxim AI - End-to-end platform for simulation, evaluation, and monitoring.
• Together AI - Cloud platform for 200+ open-source models with optimized inference.
• Fireworks AI - Fast inference platform optimized for latency and throughput.
• DeepInfra - Serverless GPU inference for open-source LLMs.
• Replicate - Cloud platform for running and fine-tuning open-source models.
• Anyscale - AI platform built on Ray for scaling LLM applications.

Semantic Caching

• RedisAI - Redis module for serving AI models and caching embeddings. • Vald - Highly scalable distributed fast approximate nearest neighbor search engine. • Elasticsearch Semantic Cache - Use Elasticsearch for semantic caching with dense vector search. • Weaviate - Open-source vector database for storing and querying AI embeddings with hybrid search capabilities. • Weaviate Semantic Cache - Vector-based semantic caching using Weaviate hybrid search. • Milvus AI Cache - Use Milvus vector store for prompt similarity caching.

Reduce costs 60-80% by caching semantically similar responses.

• RedisVL SemanticCache - Semantic caching built on Redis vector search.
• GPT-Cache - Semantic cache for LLM responses.
• semantic-prompt-cache - RAG plus Semantic Cache system with FAISS.
• vCache - Verified semantic prompt caching with adaptive thresholds.
• VectorQ - Adaptive similarity thresholds for semantic caching.
• Qdrant - Vector similarity search engine.
• Pinecone - Managed vector database.
• Chroma - Vector database for AI apps.
• pgvector - Vector similarity in PostgreSQL.
• FAISS - Facebook vector search library.
• ModelCache - LLM semantic caching system for reducing response time via cached query-result pairs.
• Mimir - Go drop-in proxy that caches LLM API responses using semantic similarity.
• Semcache - Rust semantic caching layer for LLM applications.
• OmniCache - Unified multi-layer caching library for AI and agent pipelines.
• Graft - Local-first semantic cache for AI agents with cross-session memory via MCP.

Cost Tracking and Analytics

• Kubecost - Cost monitoring for Kubernetes workloads, adaptable for LLM costs. • AWS Cost Explorer with AI Insights - AI-driven cost anomaly detection and forecasting. • OpenCost - Open-source cost monitoring for cloud-native workloads, can be adapted for LLM costs. • Cloudability - Cloud cost management platform with AI-driven insights. • Prometheus LLM Exporter - Export LLM metrics (tokens, latency, cost) to Prometheus via LiteLLM. • OpenLLMetry - OpenTelemetry instrumentation for LLM applications.

Monitor, attribute, and optimize LLM spend.

• Langfuse - Open-source LLM engineering platform.
• Helicone - Open-source LLM observability platform.
• Arize Phoenix - ML and LLM observability platform.
• Opik - LLM development platform by Comet.
• tokenmeter - Track AI API usage locally.
• tokentap - Terminal dashboard for LLM token tracking.
• tokenator - Monitor LLM token usage.
• LLM Cost Guardian - Multi-provider cost tracking in Go.
• tokenx - Python decorators for cost and latency monitoring.
• llm-performance-tracker - Multi-tenant LLM analytics dashboard.
• Weave - LLM observability from Weights and Biases.
• Datadog LLM Observability - Enterprise monitoring with LLM metrics.
• PostHog - Product analytics with LLM event tracking.
• Confident AI - Evaluation-first observability platform.
• Maxim AI - End-to-end platform for simulation, evaluation, and monitoring.
• Coze Loop - Next-gen AI agent optimization platform with full-lifecycle management.
• Pydantic Logfire - AI observability platform for production LLM and agent systems.
• Agenta - Open-source LLMOps platform with prompt playground, management, evaluation, and observability.
• Latitude - Open-source AI monitoring platform.
• Acontext - Agent skills as a memory layer for context engineering.
• Laminar - Open-source observability platform purpose-built for AI agents.
• Judgeval - Continuous-improvement stack for agents with environment data and evals.
• TraceRoot - Open-source observability and self-healing layer for AI agents.
• Agent Prism - React components for visualizing traces from AI agents.
• Aegis - Runtime policy enforcement for AI agents with cryptographic audit trail and kill switch.
• AgentWatch - AI observability framework for comprehensive agent interaction insights.
• VoltAgent - AI Agent Engineering Platform built on open-source TypeScript framework.

Load Balancing and Routing

• F5 NGINX Plus AI Security - AI-powered web application and API protection. • Citrix ADC with AI - Application delivery controller with AI-based traffic management. • Envoy AI Extension - Use Envoy extensions for AI-driven load balancing. • Istio AI Traffic Management - AI-based traffic routing in Istio service mesh. • NGINX Plus AI Module - Dynamic module for request routing, fallback, and token limiting. • HAProxy LLM Runtime - Use HAProxy ACLs and stick tables for provider failover and latency-based routing.

Distribute traffic, implement failovers, and optimize costs.

• LiteLLM Router - Retry/fallback logic and least-busy routing.
• Portkey - Conditional routing and percentage-based distribution.
• Bifrost Load Balancing - Adaptive load balancer with cluster mode.
• Ferro Labs Router - Multi-provider routing with 29 providers.
• Routerly Policies - 9 configurable policies including LLM-native routing.
• Lovable: 1.8B tokens per minute load balancing - PID-controlled dynamic load balancing.
• RouteLLM - Cost-quality routing with trainable lightweight routers.
• vLLM Semantic Router - System-level intelligent router for Mixture-of-Models at cloud, data center, and edge.
• NVIDIA LLM Router - Route LLM requests to the best model for the task at hand.
• Dario - Claude Pro/Max subscription proxy for any tool at subscription pricing.
• RouterArena - Open framework for evaluating LLM routers with standardized datasets, metrics, and live leaderboard.
• Infermux - Go-based inference routing across providers with load balancing and cost tracking.
• FreeRouter - Self-hosted AI model router with 14-dimension classifier. OpenRouter alternative using your own API keys.
• Claude Model Router Hook - Claude Code hooks that auto-switch model tier based on task complexity.
• a3m Router - Top-ranked router on RouterArena with 96.77% accuracy at $0.0768/1K tokens.

API Management and Rate Limiting

• MuleSoft AI Governance - AI-powered API governance and security. • Apigee API Hub with AI Recommendations - Discover and recommend APIs using AI-powered insights. • Apigee AI-powered API Management - Google Cloud's Apigee with AI for intelligent API lifecycle management. • 3Scale AI API Management - AI-enhanced API management for performance and security insights. • Tyk AI Gateway - Open-source API gateway with AI-specific rate limiting, token quotas, and prompt guardrails. • Kong AI Rate Limiting Advanced - Plugin for token-based rate limiting and semantic caching.

Control access, prevent abuse, and enforce quotas.

• RateLimit4j - Java rate limiting library.
• Guava RateLimiter - Google token bucket implementation.
• Resilience4j RateLimiter - Java rate limiting and circuit breaker library.
• express-rate-limit - Rate limiting middleware for Express.js and Node.js.
• bottleneck - Promise-based rate limiter for Node.js with clustering support.

Observability and Logging

• Splunk AI for IT - AI-powered IT observability and security. • Dynatrace AI - AI-powered observability and automation. • Datadog AI Monitoring - AI-powered monitoring for applications and infrastructure. • New Relic AI - AI-driven observability and incident response. • LangSmith - Unified platform for tracing, evaluation, and prompt management from LangChain. • Helicone AI - Open-source LLM observability with prompt tracing and cost monitoring.

Full visibility into LLM behavior, performance, and costs.

• OpenTelemetry - Vendor-neutral observability framework.
• Traceloop OpenLLMetry - OpenTelemetry for LLMs.
• Grafana - Metrics visualization.
• Loki - Log aggregation for LLM logs.
• LiteLLM Logging - Logging to object storage.
• LiteLLM Admin UI - Built-in spend and usage dashboards.
• LiteLLM Grafana Dashboard - Prometheus metrics visualization.
• AgentOps - Observability for agentic loops with tool usage tracking.

Agentic Orchestration

• LangFlow - Visual framework for building agents and agent-based applications. • Promptflow - Development tool for streamlining LLM application development. • LlamaIndex Agents - Build agents with LlamaIndex for data-aware applications. • LangChain Agents - Framework for building agents with LangChain. • Microsoft Semantic Kernel - SDK for orchestrating AI agents with planners, memory, and connectors. • AutoGen Studio - Low-code UI for building and debugging multi-agent workflows.

Build and manage autonomous agents, long-running tasks, and multi-agent coordination.

• LangGraph - Low-level orchestration framework for building stateful, multi-agent applications.
• Agency Swarm - Multi-agent framework building collaborative networks of AI agents.
• CrewAI - Multi-agent framework with LLM routing.
• AutoGen - Microsoft multi-agent framework.
• UiPath Maestro - Enterprise orchestrator blending LLM agents with RPA and human-in-the-loop.

SDKs and Libraries

• Together AI SDK - Python SDK for Together AI's inference API. • Replicate SDK - SDK for running and fine-tuning open-source models. • Cohere Python SDK - Official Python client for Cohere's NLP models. • AI21 Labs SDK - Python SDK for AI21 Labs' Jurassic models. • Mistral AI SDK - Official Python client for Mistral AI endpoints. • Google AI Python SDK - Generative AI SDK for Gemini models.

Multi-provider abstractions and LLM client libraries.

• OpenAI Python SDK - Official OpenAI client.
• Anthropic Python SDK - Official Claude client.
• Vercel AI SDK - AI SDK for Next.js and Svelte.
• LiteLLM Python SDK - Unified interface for 100+ providers.
• Portkey Python SDK - Multi-provider with tracing.
• LangChain - LLM orchestration framework.
• LlamaIndex - RAG framework with gateway integrations.
• tiktoken - Fast tokenization by OpenAI.
• tokenizers - Hugging Face tokenizers.
• httpx - Async HTTP client.
• Ferro Labs Go SDK - Go SDK for Ferro Labs gateway.

Guardrails and Security

• Lakera Guard - Real-time AI application security for prompt injection, data loss, and more. • HiddenLayer - ML model security platform for protecting AI systems. • Microsoft Presidio - Open-source service for PII detection and redaction. • Amazon Comprehend - NLP service for insights and content moderation. • Guardrails AI - Open-source framework for adding validation and correction to LLM outputs. • NeMo Guardrails - Programmable guardrails for LLM applications (NVIDIA).

Content filtering, PII redaction, and prompt injection protection.

• Portkey Guardrails - Input and output filtering.
• LiteLLM Guardrails - Pre and post call hooks.
• Ferro Labs Guardrails - Word/phrase filtering and token limits.
• PromptGuard - Prompt injection detection.
• NeMo Guardrails - NVIDIA dialogue guardrails.
• LLM Guard - Security toolkit for LLM applications.
• SlashLLM - Enterprise security platform.
• Azure AI Content Safety - Microsoft content moderation.
• AWS AI Services - Amazon content moderation.
• DeepEval - Open-source testing framework for LLM outputs.
• OpenZiti Security - Zero-trust access with E2E encryption.
• Agentic Security - Agentic LLM vulnerability scanner and AI red teaming kit.
• DeepTeam - Framework to red team LLMs and AI agents.
• Fast LLM Security Guardrails - The fastest trust layer for AI agents.
• Last Layer - Ultra-fast, low latency LLM prompt injection and jailbreak detection.
• Open Bias - Open-source reliability harness for enforcing, tracing, and improving agent rule compliance.
• Trylon Gateway - Self-hosted firewall for LLMs with powerful guardrails.
• PROMPTPurify - Prompt-injection guardrail using compact model that outperforms larger open-source guards.
• Control Layer - Production-grade control layer with input validation, schema enforcement, and circuit breaking.
• HAI Guardrails - TypeScript library providing guards for LLM applications.
• VeilArmor - Enterprise-grade security framework for LLMs with multi-layered protection.
• Agent Guardrails - Merge gates and safety checks for AI coding agents via MCP.

Tutorials and Case Studies

• Microsoft Azure AI Gateway Documentation - Guide to building AI gateways on Azure. • HashiCorp Consul for AI Service Mesh - Using Consul for AI service discovery and security. • AWS AI/ML Gateway Patterns - AWS blog on building AI/ML gateways. • Google Cloud AI Gateway Best Practices - Google Cloud blog. • Building AI Gateways with Kong - Step‑by‑step guide to adding LLM routing, caching, and guardrails. • Multi‑Provider LLM Routing with AWS API Gateway - AWS tutorial on Bedrock integration.

• LiteLLM Quick Start - Official LiteLLM documentation.
• Bifrost Documentation - Bifrost setup guide.
• Portkey Quick Start - Portkey getting started guide.
• Building Multi-Provider LLM Infrastructure - Architecture guide.
• LLM Orchestration with Bifrost - Implementation guide.
• Failover Routing Strategies - Production patterns.
• Semantic Caching Guide - Implementation best practices.
• Cost-Aware Routing Patterns - Routing strategies.
• OpenRouter Multi-Provider Routing - OpenRouter guide.
• Rasa Multi-LLM Routing - Rasa routing documentation.
• Cortex - Semantic-aware knowledge caching for LLM agents.
• Ferro Labs Getting Started - Go-native gateway setup.
• Routerly Documentation - Intelligent routing guide.
• OSS LLMOps Stack - Modular open-source LLMOps stack combining LiteLLM and Langfuse.
• awesome-mllm-guardrails - Curated list of LLM guardrails, safety benchmarks, guard models, and jailbreak attacks.
• awesome-free-llm-apis - List of permanent free LLM API keys.

Communities

• MLOps World Community - Community for MLOps practitioners and enthusiasts. • Dataiku Community - Community for collaborative data science and ML. • LLMops Subreddit - Reddit community for LLM operations. • MLops.community - Community for MLOps practitioners. • Discord: AI Gateway Developers - Community for discussing gateway patterns, tools, and best practices.

• /r/LLMOps - Reddit community for LLM operations.
• LangChain Discord - Framework community.
• LiteLLM Discord - Gateway community.
• Hugging Face Community - Model and deployment discussions.

Related Awesome Lists

• rothgar/awesome-tuis - 18K+ stars, comprehensive TUI list.
• msmps/awesome-opentui - Curated open TUI resources.