Skip to content

envoy-ai-gateway

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

envoy-ai-gateway

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
multi-model-routing
multi-model-routing
 
 
rate-limiting
rate-limiting
 
 
README.md
README.md
 
 
gateway.yaml
gateway.yaml
 
 
model-backends.yaml
model-backends.yaml
 
 

README.md

Envoy AI Gateway Blueprint

Organizations deploying AI applications face a fundamental challenge: no single model serves all needs. Developers may choose Claude for long-context analysis, OpenAI for reasoning tasks, and DeepSeek for cost-sensitive workloads. The problem is that each model provider uses different APIs. Without centralized control, teams can't easily switch providers, get visibility into utilization, or enforce quotas.

Envoy AI Gateway is an open source project that solves this challenge by providing a single, scalable OpenAI-compatible endpoint that routes to multiple supported LLM providers. It gives Platform teams cost controls and observability, while developers never touch provider-specific SDKs.

Key objectives of Envoy AI Gateway

  • Provide a unified layer for routing and managing LLM/AI traffic
  • Support automative failover mechanisms to ensure service reliability
  • Ensure end-to-end security, including upstream authorization for LLM/AI traffic
  • Implement a policy framework to support usage limiting use cases
  • Foster an open-source community to address GenAI-specific routing and quality of service needs

Envoy Gateway Fundamentals

:::info If you're already familiar with Envoy Gateway, you can skip this section. :::

As Envoy AI Gateway builds on top of the standard Kubernetes Gateway API and Envoy Gateway extensions, it's necessary to familiarize yourself with the underlying Envoy Gateway primitives:

  • GatewayClass - Defines which controller manages the Gateway. Envoy AI Gateway uses the same GatewayClass as Envoy Gateway.
  • Gateway - The entry point for traffic. A Gateway resource defines listeners (HTTP/HTTPS ports). When you create a Gateway, Envoy Gateway deploys the actual Envoy proxy pods and a corresponding Kubernetes Service (typically a LoadBalancer).
  • HTTPRoute - The instruction for routing traffic HTTP based on hostnames, paths, or headers. Conceptually, this is similar to ingress rules or listener rules in ALB.
  • Backend - A Kubernetes Service or an external endpoint.
  • BackendTrafficPolicy - Configures connection behavior like timeouts, retries, and rate limiting of an HTTPRoute.
  • ClientTrafficPolicy - Configures how the Envoy proxy server behaves with downstream clients.
  • EnvoyExtensionPolicy - A way to extend Envoy's traffic processing capabilities.

Envoy AI Gateway introduces following CRDs:

  • AIGatewayRoute - Defines unified API and routing rules for AI traffic
  • AIServiceBackend - Represents individual AI service backends like Bedrock
  • BackendSecurityPolicy - Configures authentication for backend access
  • BackendTLSPolicy - Defines TLS parameters for backend connections

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────────┐
│   Client Apps   │───▶│  Envoy Gateway   │───▶│   AI Model Services │
└─────────────────┘    └──────────────────┘    └─────────────────────┘
                              │                        │
                              ▼                        ▼
                       ┌──────────────────┐    ┌─────────────────────┐
                       │  AI Gateway      │    │   AWS Bedrock       │
                       │  Controller      │    │   Claude Models     │
                       └──────────────────┘    └─────────────────────┘

This envoy gateway blueprint deploys Envoy AI Gateway on Amazon EKS and supports two use cases:

  • Multi-model routing
  • Rate limiting

Prerequisites

Before we begin, ensure you have all the necessary prerequisites in place to make the deployment process smooth. Make sure you have installed the following tools on your machine:

  1. aws cli
  2. kubectl
  3. terraform
  4. envsubst

Deploy

Clone the repository

git clone https://github.com/awslabs/ai-on-eks.git

Important Note:

Step1: Ensure that you update the region in the blueprint.tfvars file before deploying the blueprint. Additionally, confirm that your local region setting matches the specified region to prevent any discrepancies.

For example, set your export AWS_DEFAULT_REGION="<REGION>" to the desired region:

Step2: Run the installation script.

cd ai-on-eks/infra/envoy-ai-gateway/ && chmod +x install.sh
./install.sh

Verify the resources

Once the installation finishes, verify the Amazon EKS Cluster.

Creates k8s config file to authenticate with EKS.

aws eks --region us-west-2 update-kubeconfig --name envoy-gateway-cluster
kubectl get nodes

Deploy AI Models

# Create Hugging Face token secret
kubectl create secret generic hf-token --from-literal=token=your_huggingface_token

# Add helm chart repository
helm repo add ai-on-eks https://awslabs.github.io/ai-on-eks-charts/
helm repo update

# Deploy qwen3 model
helm install qwen3-1.7b ai-on-eks/inference-charts \
  -f https://raw.githubusercontent.com/awslabs/ai-on-eks-charts/refs/heads/main/charts/inference-charts/values-qwen3-1.7b-vllm.yaml \
  --set nameOverride=qwen3 \
  --set fullnameOverride=qwen3 \
  --set inference.serviceName=qwen3

# Deploy gpt-oss model
helm install gpt-oss ai-on-eks/inference-charts \
  -f https://raw.githubusercontent.com/awslabs/ai-on-eks-charts/refs/heads/main/charts/inference-charts/values-gpt-oss-20b-vllm.yaml \
  --set nameOverride=gpt-oss \
  --set fullnameOverride=gpt-oss \
  --set inference.serviceName=gpt-oss

Multi-model routing

Route requests to different AI models based on the x-ai-eg-model header. This header enables Envoy AI gateway to identify appropriate route configured within the gateway and routes client traffic to relevant backend kubernetes service. In this case, it's a service that exposes a self-hosted model or Amazon Bedrock model.

Deploy common gateway infrastructure

cd ../../blueprints/gateways/envoy-ai-gateway
kubectl apply -f gateway.yaml

Configure model backends

kubectl apply -f model-backends.yaml

Configure model routes

kubectl apply -f multi-model-routing/ai-gateway-route.yaml

Test multi-model routing

python3 multi-model-routing/client.py

Expected Output:

🚀 AI Gateway Multi-Model Routing Test
============================================================
Gateway URL: http://your-gateway-url
=== Testing Qwen3 1.7B ===
Status Code: 200
✅ SUCCESS: Qwen3 - [response content]

=== Testing Self-hosted GPT ===
Status Code: 200
✅ SUCCESS: GPT - [response content]

=== Testing Bedrock Claude ===
Status Code: 200
✅ SUCCESS: Bedrock Claude - [response content]

🎯 Final Results:
• Qwen3 1.7B: ✅ PASS
• GPT OSS 20B: ✅ PASS
• Bedrock Claude: ✅ PASS

📊 Summary: 3/3 models working

Rate limiting

Token-based rate limiting with automatic tracking for AI workloads.

Features:

  • Token-based rate limiting (input, output, and total tokens)
  • User-based rate limiting using x-user-id header
  • Redis backend for distributed rate limiting (automatically deployed)
  • Configurable limits per user per time window

Deploy rate limiting

kubectl apply -f rate-limiting/ai-gateway-route.yaml
kubectl apply -f rate-limiting/ai-gateway-rate-limit.yaml
kubectl apply -f rate-limiting/backend-traffic-policy.yaml

Test rate limiting

python3 rate-limiting/client.py

Configuration Details

Routing Configuration

The AI Gateway routes requests based on the x-ai-eg-model header:

Header Value Backend Endpoint Model Type
Qwen/Qwen3-1.7B qwen3 /v1/chat/completions Self-hosted
openai/gpt-oss-20b gpt-oss /v1/chat/completions Self-hosted
anthropic.claude-3-haiku-20240307-v1:0 bedrock /anthropic/v1/messages AWS Bedrock

Bedrock Integration Details

  • Authentication: Pod Identity (automatically configured via installation script)
  • Schema: AWSAnthropic for native Bedrock support
  • Endpoint: /anthropic/v1/messages (Anthropic Messages API format)
  • Region: Configurable in backend-security-policy.yaml (default: us-west-2)

Resources

Important Notes

  • Multi-Model Routing: Requires deployed AI model services and AWS Bedrock access
  • Rate Limiting: Requires actual AI models that return real token usage data and Redis for storage
  • Bedrock Integration: Requires AWS Bedrock API access, proper IAM setup, and Pod Identity configuration
  • Authentication: Pod Identity for Bedrock is automatically configured when deploying via the installation script

These are working configuration examples that demonstrate AI Gateway capabilities with real AI model deployments and AWS Bedrock integration.