llm-shim

llm-shim exposes an OpenAI-compatible API and routes requests to configured providers via pydantic-ai.

Endpoints

• POST /v1/chat/completions
• POST /v1/embeddings
• GET /v1/models
• GET /healthz and GET /livez

Requirements

• Python >=3.14
• uv (recommended)

Quick Start (Local)

1. Install dependencies:

uv sync

2. Create config file in data/:

cp config.example.yaml data/config.yaml

3. Edit data/config.yaml with your providers and credentials.

4. Start the server:

uv run main.py

5. Open:

• http://localhost:8000/docs
• http://localhost:8000/redoc

Quick Start (Docker Compose)

docker compose up --build

compose.yaml mounts ./data to /config, and the container reads config from /config/config.yaml.

Configuration

• providers: a mapping of provider names to their configuration.
  • chat_models: list of allowed chat model patterns (supports wildcards, example: gpt-*).
  • embedding_models: list of allowed embedding model patterns (supports wildcards, example: text-embedding-3-*).
  • chat_model_settings: optional kwargs to pass to chat model calls. See pydantic-ai docs for supported settings.
  • embedding_model_settings: optional kwargs to pass to embedding model calls. See pydantic-ai docs for supported settings.
  • env: environment variables to set for this provider (required for things like API keys).
• server: FastAPI server settings.
  • host: server host (default: "0.0.0.0")
  • port: server port (default: 8000)
  • reload: enable auto-reload (default: false)
  • workers: number of worker processes (default: 1)
  • log_level: logging level (default: "info")

providers:
  openai:
    chat_models: ["gpt-*"]
    embedding_models: ["text-embedding-3-*"]
    chat_model_settings: {}
    embedding_model_settings: {}
    env:
      OPENAI_API_KEY: "sk-..."

  google-gla:
    chat_models: ["gemini-*"]
    embedding_models: ["gemini-embedding-*"]
    env:
      GOOGLE_API_KEY: "AIza..."

server:
  host: "0.0.0.0"
  port: 8000
  reload: false
  workers: 1
  log_level: "info"

Routing Rules

• Requests must send model as provider:model (example: openai:gpt-4o-mini).
• The provider must exist in providers.
• Chat models are validated against chat_models, and embeddings against embedding_models.
• Model patterns support exact values and wildcards (*) using shell-style matching.

Minimal API Examples

Chat completion:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openai:gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello, world!"}]
  }'

Embeddings:

curl -s http://localhost:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openai:text-embedding-3-small",
    "input": "Hello, world!"
  }'

List configured models:

curl -s http://localhost:8000/v1/models

External References

• OpenAI API format: Chat Completions, Embeddings, Models
• pydantic-ai providers and model naming: docs

This project is a thin shim that routes OpenAI-compatible requests to pydantic-ai providers. Those docs will serve you better on how to make requests and configure providers.