Skip to content
BranchesTags

Repository files navigation

LocalLMQ

Cloud-to-Local LLM Inference Gateway — Resource-aware queue system that routes cloud LLM requests to your local Mac, processes them through Ollama, and returns results. When your machine is busy, it returns 429 so the cloud app can fallback to cloud models.

Cloud App ──HTTPS──▶ Cloudflare Tunnel ──▶ LocalLMQ (:8000) ──▶ Ollama (:11434)
                                              │
                                     Queue + Resource Monitor
                                     (RAM / Thermal / Queue depth)

Why

Running LLM inference locally on Apple Silicon is free and private. But your Mac has limited resources — if 100 requests hit at once, the system will swap and freeze. LocalLMQ solves this by:

  • Queuing requests and processing them one at a time (M1 GPU can't parallelize efficiently)
  • Monitoring RAM, CPU, and thermal state in real-time
  • Rejecting excess load with HTTP 429 + Retry-After header so the cloud app falls back to cloud models
  • Exposing an OpenAI-compatible API — your cloud app only changes the base URL

Features

  • 🔄 OpenAI-compatible API — drop-in replacement (/v1/chat/completions)
  • 📡 SSE Streaming — optional token-by-token streaming ("stream": true)
  • 🧠 Resource-aware — monitors RAM, CPU, thermal state via psutil + pmset
  • 🚦 Backpressure — configurable queue size, instant 429 on overflow
  • 💾 Persistent queue — SQLite-backed job history survives restarts
  • 🔒 API key auth — Bearer token authentication
  • 📊 Monitoring dashboard — real-time web UI at http://localhost:8000
  • 🔁 Auto-restart — launchd integration for crash recovery

Requirements

  • macOS with Apple Silicon (M1/M2/M3/M4)
  • Python 3.9+
  • Ollama installed and running

Quick Start

1. Install

git clone <repo-url> locallmq
cd locallmq
pip3 install -e .

2. Configure

cp .env.example .env
# Edit .env — at minimum change the API key:
# LOCALLMQ_API_KEY=your-secure-key-here

3. Run

# Make sure Ollama is running
ollama serve

# Start LocalLMQ
./locallmq.sh start

Open http://localhost:8000 to see the monitoring dashboard.

Configuration

All settings are configured via environment variables in .env:

Variable Default Description
LOCALLMQ_API_KEY change-me-to-a-secure-key Bearer token for API authentication
LOCALLMQ_OLLAMA_URL http://localhost:11434 Ollama server URL
LOCALLMQ_DEFAULT_MODEL qwen3:8b Default model when not specified in request
LOCALLMQ_MAX_QUEUE_SIZE 5 Max pending jobs. Beyond this → 429
LOCALLMQ_JOB_TIMEOUT 300 Per-job timeout in seconds
LOCALLMQ_MEMORY_PRESSURE_THRESHOLD 80 Memory % above which new jobs are rejected
LOCALLMQ_HOST 0.0.0.0 Server bind address
LOCALLMQ_PORT 8000 Server port

API Reference

Chat Completions

OpenAI-compatible endpoint. Your cloud app just changes the base URL.

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:latest",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello!"}
    ],
    "temperature": 0.7,
    "max_tokens": 1024
  }'

Streaming (SSE):

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:latest",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Health Check

No authentication required. Useful for load balancers and Cloudflare Tunnel health checks.

curl http://localhost:8000/v1/health
{
  "status": "healthy",
  "ollama_connected": true,
  "memory_percent": 61.7,
  "memory_available_gb": 6.1,
  "thermal_state": "nominal",
  "cpu_percent": 15.2,
  "queue_depth": 0,
  "can_accept_work": true
}

status values: healthy, degraded (high memory or thermal), unhealthy.

Queue Status

curl http://localhost:8000/v1/queue/status \
  -H "Authorization: Bearer your-api-key"

Job Details

# Get job status
curl http://localhost:8000/v1/queue/{job_id} \
  -H "Authorization: Bearer your-api-key"

# Cancel a pending job
curl -X DELETE http://localhost:8000/v1/queue/{job_id} \
  -H "Authorization: Bearer your-api-key"

Dashboard Data

# JSON feed (no auth required)
curl http://localhost:8000/api/dashboard

Cloud-Side Integration

Your cloud application just uses the OpenAI SDK with a custom base URL:

from openai import OpenAI

# Try local first, fallback to cloud
try:
    client = OpenAI(
        base_url="https://llm.yourdomain.com/v1",  # LocalLMQ via tunnel
        api_key="your-locallmq-key",
        timeout=30,
    )
    response = client.chat.completions.create(
        model="gemma4:latest",
        messages=[{"role": "user", "content": "Hello!"}],
    )
except Exception:
    # Fallback to cloud (OpenAI, Anthropic, etc.)
    client = OpenAI(api_key="sk-...")
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello!"}],
    )

429 handling (recommended):

import httpx

try:
    response = client.chat.completions.create(...)
except openai.RateLimitError:
    # LocalLMQ is at capacity — use cloud model
    response = cloud_client.chat.completions.create(...)

Process Management

LocalLMQ uses launchd (macOS's native service manager) for process lifecycle:

./locallmq.sh install     # Install as launchd service (auto-start on login)
./locallmq.sh start       # Start
./locallmq.sh stop        # Stop
./locallmq.sh restart     # Restart
./locallmq.sh status      # Health + process info
./locallmq.sh logs        # Show recent logs
./locallmq.sh tail        # Follow logs in real-time
./locallmq.sh uninstall   # Remove the service

What launchd provides:

  • ✅ Auto-start on macOS login
  • ✅ Auto-restart on crash (10s throttle)
  • ✅ Lower CPU priority than UI apps (Nice=5)
  • ✅ Logs at ~/.locallmq/logs/

Manual mode (without installing the service):

./locallmq.sh start   # Runs in foreground if not installed
# or directly:
python3 -m uvicorn locallmq.main:app --host 0.0.0.0 --port 8000 --workers 1

Cloudflare Tunnel

Cloudflare Tunnel exposes your local Mac to the internet securely — no port forwarding, no static IP needed.

Quick Test (no account needed)

brew install cloudflared
cloudflared tunnel --url http://localhost:8000

Gives you a temporary https://xxxxx.trycloudflare.com URL. No signup, no config.

Production Setup (permanent subdomain)

1. Login to Cloudflare (one-time):

cloudflared login
# Opens browser → select your domain → saves ~/.cloudflared/cert.pem

2. Create the tunnel:

cloudflared tunnel create locallmq
# Output: Created tunnel locallmq with id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

3. Configure routing:

Create ~/.cloudflared/config.yml:

tunnel: locallmq
credentials-file: /Users/YOUR_USER/.cloudflared/TUNNEL_ID.json

ingress:
  - hostname: llm.yourdomain.com
    service: http://localhost:8000
    originRequest:
      noTLSVerify: true
  - service: http_status:404

4. Add DNS record:

cloudflared tunnel route dns locallmq llm.yourdomain.com

5. Run the tunnel:

# Foreground
cloudflared tunnel run locallmq

# Or install as a macOS service (auto-start)
cloudflared service install

6. Test it:

curl https://llm.yourdomain.com/v1/health

Note: Cloudflare Tunnel is free. No API keys needed — it uses OAuth via browser for initial auth. Add your domain to Cloudflare DNS first.

Docker

Docker is available for Linux deployment or CI environments. On macOS, native execution is recommended because Docker can't access Apple Silicon GPU.

Linux (with NVIDIA GPU)

docker compose up -d
# Ollama runs containerized with GPU passthrough

macOS (native Ollama + containerized gateway)

ollama serve  # Native Ollama with Apple Silicon GPU
docker compose -f docker-compose.yml -f docker-compose.mac.yml up locallmq

Testing

pip3 install -e ".[dev]"
python3 -m pytest tests/ -v
53 passed in 1.07s

All tests use a mocked Ollama backend — no running Ollama server needed.

Suite Tests Coverage
Auth & Security 6 Public/protected endpoints, valid/invalid keys
Health & Dashboard 5 Schema validation, Ollama status, data structure
Chat Completions 8 Blocking, streaming, validation, defaults
Queue Management 5 Status, job lookup, cancel, history
Resource Monitor 16 Memory/thermal thresholds, lifecycle
Job Model & Store 10 CRUD, stats, ordering, cleanup
Backpressure 3 Memory 429, thermal 429, Retry-After

Architecture

┌─────────────────────────────────────────────────────┐
│                   FastAPI Gateway                    │
│                                                     │
│  /v1/chat/completions ─┐                            │
│  /v1/health            │    ┌──────────────────┐    │
│  /v1/queue/*           ├───▶│  Queue Manager   │    │
│  /api/dashboard        │    │  (asyncio.Queue  │    │
│  /                     │    │   + SQLite)      │    │
│                        │    └───────┬──────────┘    │
│  ┌─────────────┐       │            │               │
│  │ Auth Guard  │───────┘            ▼               │
│  │ Bearer Token│          ┌──────────────────┐      │
│  └─────────────┘          │ Inference Worker │      │
│                           │ (single, serial) │      │
│  ┌─────────────────┐      └───────┬──────────┘      │
│  │ Resource Monitor │             │                  │
│  │ psutil + pmset   │────gate────▶│                  │
│  │ RAM/CPU/Thermal  │             ▼                  │
│  └─────────────────┘      ┌──────────────┐          │
│                           │    Ollama    │          │
│                           │ :11434       │          │
│                           └──────────────┘          │
└─────────────────────────────────────────────────────┘

Key design decisions:

  1. Single worker — Apple Silicon UMA can't efficiently parallelize GPU inference
  2. Queue size 5 — Fast 429 rejection for overflow → cloud fallback
  3. Memory threshold 80% — macOS idles at ~65–75% on 16GB with Ollama loaded
  4. asyncio.Queue + SQLite — Zero overhead runtime queue, persistent job history
  5. OpenAI API compatible — Cloud app only changes base URL to switch providers

License

MIT

About

Resource-aware queue system that routes cloud LLM requests to your local Mac, processes them through Ollama, and returns results.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages