[Collection Request] LLM Gateway & Proxy Infrastructure - Unified Access, Routing & Cost Optimization for 100+ AI Models

[sykp241095]

[Collection Request] LLM Gateway & Proxy Infrastructure - Unified Access, Routing & Cost Optimization for 100+ AI Models

Summary

Track the rapidly growing LLM Gateway/Proxy ecosystem - infrastructure that provides unified access, intelligent routing, load balancing, and cost optimization across 100+ LLM providers. This is distinct from #2159 (LLM Inference & Serving) which focuses on running models locally.

Why This Matters

• Unified API: Single endpoint for 100+ LLM providers (OpenAI, Anthropic, Bedrock, VertexAI, etc.)
• Cost Optimization: Automatic routing to cheapest/fastest models, fallback strategies
• Enterprise Features: Rate limiting, caching, guardrails, observability, multi-tenant isolation
• Production Ready: Load balancing, retry logic, circuit breakers, high availability

Key Repositories to Track

Major Players (10K+ Stars)

Repo	Stars	Forks	Description
Kong/kong	43K+	5.1K	The API and AI Gateway - traditional API gateway adding AI capabilities
BerriAI/litellm	40K+	6.6K	Python SDK + Proxy Server (AI Gateway) for 100+ LLM APIs with cost tracking, guardrails, loadbalancing
apache/apisix	16K+	2.8K	Cloud-Native API Gateway and AI Gateway
Portkey-AI/gateway	11K+	954	Blazing fast AI Gateway with integrated guardrails, 200+ LLMs, 50+ AI Guardrails
tensorzero/tensorzero	11K+	797	LLMOps platform unifying LLM gateway, observability, evaluation, optimization

Emerging Players (1K-10K Stars)

Repo	Stars	Forks	Description
alibaba/higress	7.9K+	1K+	AI Gateway
IBM/mcp-context-forge	3.5K+	592	AI Gateway, registry, and proxy for MCP, A2A, or REST/gRPC APIs
maximhq/bifrost	3.1K+	338	Claims 50x faster than LiteLLM, adaptive load balancer, <100µs overhead at 5k RPS

Specialized Solutions (<1K Stars)

Repo	Stars	Forks	Description
envoyproxy/ai-gateway	1.4K+	197	Unified Access to Generative AI Services built on Envoy Gateway
traceloop/hub	174+	31	High-scale LLM gateway in Rust with OpenTelemetry observability
Helicone/ai-gateway	555+	46	Fastest, lightest AI gateway - fully open-sourced
labring/aiproxy	-	-	High performance AI gateway with intelligent error handling, multi-channel management

Ecosystem Categories

1. Traditional API Gateways adding AI: Kong, Apache APISIX, Higress
2. LLM-Native Gateways: LiteLLM, Portkey, Bifrost
3. LLMOps Platforms with Gateway: TensorZero, Helicone
4. Service Mesh / Proxy-based: Envoy AI Gateway
5. MCP/A2A Integration: IBM MCP Context Forge

Key Features to Track

• Model routing & fallback strategies
• Cost optimization & tracking
• Rate limiting & quota management
• Caching & response optimization
• Guardrails & safety filters
• Observability & tracing (OpenTelemetry)
• Multi-tenant isolation
• Load balancing & high availability
• Protocol unification (OpenAI-compatible APIs)
• MCP/A2A integration

Related Issues

• LLM Inference & Serving Infrastructure Collection (200K+ Stars) #2159 - LLM Inference & Serving Infrastructure (running models, not gateway)
• AI Observability Ecosystem: Trace, Monitor & Debug Production LLM Agents #2131 - AI Observability (tracing/monitoring, not gateway routing)
• [Growth] MCP Server Directory - Unified Collection for 80K+ Star Ecosystem #2109 - MCP Ecosystem (gateway can integrate with MCP)
• [Growth] 🤝 A2A Protocol - Track Agent-to-Agent Interoperability Ecosystem (22.7k Stars, Google-Led) #2122 - A2A Protocol (gateway can integrate with A2A)

Suggested Labels

area/growth, type/feature, priority/p2, topic/ai-infra, topic/llm-gateway

[sykp241095]

New collection created in PR #2271.