llm-d-latency-predictor

Latency prediction service for the llm-d ecosystem.

Overview

Prerequisites

• Python 3.11+
• Docker (for container builds)
• pre-commit (for local development)

Quick Start

# Clone the repo
git clone https://github.com/llm-d/llm-d-latency-predictor.git
cd llm-d-latency-predictor

# Install pre-commit hooks
pre-commit install

# Install Python dependencies
make install

# Run tests
make test

# Run linters
make lint

Development

See CONTRIBUTING.md for development guidelines, coding standards, and how to submit changes.

Common Commands

make help           # Show all available targets
make install        # Install Python dependencies
make test           # Run Python tests
make lint           # Run Python linter (ruff)
make fmt            # Format Python code
make image-build    # Build container images (prediction, training, test)
make pre-commit     # Run pre-commit hooks

Layout

common/
  types.py                    # Shared enums and model types
prediction/
  prediction_server.py        # FastAPI prediction server (serves latency predictions)
  Dockerfile                  # Image for the prediction server
training/
  training_server.py          # FastAPI training server (trains models from request traces)
  Dockerfile                  # Image for the training server
tests/
  test_dual_server_client.py  # integration / load-test client exercising both servers
  Dockerfile                  # Image that runs the test client as a Job
deploy/                       # Kubernetes manifests and kustomization
build-deploy.sh               # Helper script for building images and deploying to GKE

After make install, the servers can be run with:

uvicorn prediction.prediction_server:app --port 8001
uvicorn training.training_server:app --port 8000

Architecture

Configuration

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

All commits must be signed off (DCO). See PR_SIGNOFF.md for instructions.

Security

To report a security vulnerability, please see SECURITY.md.

License

This project is licensed under the Apache License 2.0 - see LICENSE for details.