A repository for exploring LLM Serving and MLOps practices on AWS Kubernetes (EKS) using Infrastructure as Code (AWS CDK).
| Project | Description | Blog Post |
|---|---|---|
| Serving LLMs on EKS | Creating EKS cluster and serving Llama-3.1-8B-Instruct (AWQ quantized) LLM |
LLM on EKS: Serving with vLLM |
| Scaling LLMs on EKS | Autoscaling Llama-3.1-8B-Instruct inference with Ray + KubeRay, GPU nodes provisioned on demand via Karpenter |
LLM on EKS: Scaling with Ray & Karpenter |