Cloud Native LLM Platform

Cloud Native LLM Platform is a personal AI infrastructure project for running OpenAI-compatible LLM serving on AWS EKS. It focuses on the platform engineering layer around model serving: multi-tenant gateway governance, model routing, GPU scheduling, observability, and reproducible benchmark reporting.

What is implemented

• OpenAI-compatible Gateway in Go
  • Authorization: Bearer API key authentication
  • tenant model allow-list
  • request-per-minute and token-per-minute rate limiting
  • monthly token quota accounting
  • model routing to vLLM-compatible backends
  • streaming response pass-through
  • Prometheus metrics for QPS, latency, TTFT, tokens, and inflight requests
• Direct vLLM Helm chart
  • DeepSeek and Qwen values examples
  • PVC model cache, /dev/shm, GPU resource requests, nodeSelector, tolerations
• KServe-managed vLLM example
  • InferenceService running vLLM OpenAI server
• GPU scheduling examples
  • Karpenter GPU NodePool
  • Volcano queue and scheduler-based vLLM deployment
• Observability
  • Prometheus alerting and recording rules
  • Grafana dashboard config map
  • DCGM exporter metric references for GPU utilization and memory
• Benchmarking
  • YAML experiment config
  • Python benchmark client
  • JSON, CSV, and Markdown report output

Repository layout

cmd/gateway/                 Gateway entry point
internal/config/             YAML config loader and validation
internal/gateway/            Auth, quota, rate limit, routing, proxy, metrics
configs/gateway.yaml         Local gateway config
deploy/terraform/            EKS Auto Mode infrastructure
deploy/helm/gateway/         Gateway Helm chart
deploy/helm/direct-vllm/     Direct vLLM Helm chart
deploy/kserve/               KServe-managed vLLM example
deploy/scheduling/           Karpenter and Volcano examples
deploy/monitoring/           Prometheus and Grafana assets
benchmarks/                  Experiment configs and result directory
scripts/benchmark_openai.py  Benchmark runner

Deployment architecture

flowchart TB
  User["Client / OpenAI SDK / curl"] --> LB["Kubernetes Service / Ingress"]
  LB --> Gateway["LLM Gateway Pods"]

  subgraph GatewayGovernance["Gateway governance"]
    Auth["API Key auth"]
    Tenant["Tenant model allow-list"]
    RateLimit["Request and token rate limit"]
    Quota["Monthly token quota"]
    Metrics["Prometheus metrics"]
  end

  Gateway --> Auth
  Auth --> Tenant
  Tenant --> RateLimit
  RateLimit --> Quota
  Gateway --> Metrics

  Quota --> Router["Model router"]
  Router --> DeepSeekSvc["deepseek-vllm Service"]
  Router --> QwenSvc["qwen-vllm Service"]

  subgraph Runtime["Model runtime namespace"]
    DeepSeekSvc --> DeepSeekPod["vLLM DeepSeek Pod"]
    QwenSvc --> QwenPod["vLLM Qwen Pod"]
    KServe["Optional KServe InferenceService"] --> VLLMRuntime["vLLM OpenAI server"]
  end

  subgraph Scheduling["GPU scheduling"]
    Karpenter["Karpenter GPU NodePool"]
    Volcano["Volcano Queue / Scheduler"]
    GPUNode["EKS GPU Node"]
  end

  DeepSeekPod --> GPUNode
  QwenPod --> GPUNode
  VLLMRuntime --> GPUNode
  Karpenter --> GPUNode
  Volcano --> GPUNode

  subgraph Observability["Observability"]
    Prom["Prometheus"]
    Grafana["Grafana Dashboard"]
    DCGM["NVIDIA DCGM Exporter"]
  end

  Metrics --> Prom
  GPUNode --> DCGM
  DCGM --> Prom
  Prom --> Grafana

Loading

Model backend options

The project intentionally keeps two model backend deployment paths. Use one path for a given model, not both at the same time.

Option A: Direct vLLM Deployment

This path uses Helm to create normal Kubernetes resources:

Deployment + Service + PVC

The Deployment runs vllm serve ... directly. This is the simplest path for learning, debugging, and proving that the model can run on EKS GPU nodes.

Gateway routes to the direct Service with a backend URL like:

backendURL: http://deepseek-vllm.vllm.svc.cluster.local

Option B: KServe-managed vLLM

This path creates a KServe InferenceService. KServe then manages the underlying serving resources, and the predictor container still runs vllm serve ....

Use this path when you want to show Kubernetes-native model serving lifecycle management.

The example also creates a ReadWriteOnce gp3 PVC and mounts it to /root/.cache/huggingface. On first startup, vLLM downloads the Hugging Face model into this cache. Later Pod restarts can reuse the cached model files instead of downloading the model again.

Because gp3/EBS is ReadWriteOnce, this example keeps maxReplicas: 1. For multiple KServe predictor replicas, use one of these production patterns:

• Use an RWX storage backend such as EFS or FSx for Lustre for a shared model cache.
• Give each replica its own model cache instead of sharing one PVC.
• Pre-bake the model into an image or load it from an object-store based model repository.

Gateway routes to the KServe predictor Service with a backend URL like:

backendURL: http://deepseek-r1-qwen-7b-predictor.llm-platform.svc.cluster.local

See configs/gateway-kserve.yaml for a full Gateway config example.

In both options, vLLM is still the inference engine. KServe is only the management layer around the vLLM container.

Local gateway test

Run unit tests:

go test ./...

Run the gateway against the example config:

go run ./cmd/gateway -config configs/gateway.yaml

Example request:

curl -N http://localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer dev-platform-key' \
  -H 'content-type: application/json' \
  -d '{
    "model": "deepseek-r1-qwen-7b",
    "messages": [{"role": "user", "content": "Explain vLLM continuous batching."}],
    "max_tokens": 128,
    "stream": true
  }'

For local development without a real vLLM backend, change configs/gateway.yaml model backendURL to a local mock server that implements the OpenAI response shape.

Mock backend:

python3 scripts/mock_openai_backend.py --port 18080

Then point one model in configs/gateway.yaml to http://127.0.0.1:18080 and run the gateway.

Deploy on EKS

Create an EKS Auto Mode cluster:

make terraform-init
AWS_PROFILE=<profile> make terraform-plan
AWS_PROFILE=<profile> make terraform-apply
aws eks update-kubeconfig --name cloud-native-llm-platform --region us-west-2 --profile <profile>

Install scheduling, one model backend path, gateway, and monitoring:

make install-scheduling
make install-direct-vllm
make install-gateway IMAGE_REPOSITORY=ghcr.io/daemonxiao/cloud-native-llm-platform/gateway IMAGE_TAG=latest
make install-observability

To use the KServe backend path instead of direct vLLM:

make install-kserve-model

Monitoring prerequisites are documented in deploy/monitoring/service-monitor-notes.md.

Benchmark

Port-forward the gateway:

kubectl -n llm-platform port-forward svc/llm-gateway 8080:80

Run an experiment:

python3 -m venv .venv
. .venv/bin/activate
pip install -r scripts/requirements.txt
python3 scripts/benchmark_openai.py --config benchmarks/experiments/deepseek-g6e.yaml

The report includes QPS, P95 latency, TTFT, TPOT, success count, error count, and per-request CSV records.

Design notes

See docs/architecture.md for the architecture diagram and component responsibilities.