llm-mock-provider

llm-mock-provider is an OpenAI-compatible mock provider for testing LLM gateways, API routers, billing layers, retry logic, streaming behavior, and observability pipelines without calling a real model provider.

It can be used as a lightweight upstream for any OpenAI-compatible gateway:

k6 / client / gateway test runner
  -> your LLM gateway
  -> llm-mock-provider

You can also call llm-mock-provider directly when validating mock behavior.

Security

This is a public project. Do not commit real API keys, cloud credentials, kubeconfigs, tokens, production URLs, or private deployment values.

The examples use placeholder values such as dummy-key, localhost, and mock-gpt. Replace them only in your private deployment environment or secret manager.

Features

• OpenAI-compatible chat completions endpoint
• OpenAI-compatible embeddings endpoint
• Non-streaming and SSE streaming responses
• Configurable latency, TTFT, chunk delay, token counts, error rates, and timeout simulation
• Channel-style scenario paths for testing gateway failover with one mock-provider deployment
• k6 scripts for smoke, load, streaming, stress, and soak tests
• Docker and docker compose support
• No OpenAI SDK dependency
• Go standard library HTTP server, no Gin/Echo/Fiber

Project Structure

llm-mock-provider/
├── mock-provider/
│   ├── main.go
│   ├── go.mod
│   ├── Dockerfile
│   └── README.md
├── k6/
│   ├── smoke.js
│   ├── chat-non-stream.js
│   ├── chat-stream.js
│   ├── stress.js
│   └── soak.js
├── tools/
│   ├── load-probe/
│   └── stream-probe/
├── docker-compose.yml
└── README.md

Kubernetes manifests are intentionally not included here because deployment topology is gateway-specific. Keep cluster credentials and production values in a private infrastructure repository.

Image Publishing

The included GitHub Actions workflow tests and builds pull requests. After a change is merged into main, it publishes linux/amd64 images tagged with both latest and the commit SHA.

Configure these GitHub repository variables:

• CONTAINER_REGISTRY
• CONTAINER_NAMESPACE
• CONTAINER_USERNAME

Configure this GitHub repository secret:

• CONTAINER_PASSWORD

Registry credentials must remain in GitHub Secrets and must not be committed to this public repository.

Quick Start

docker compose up --build

The mock provider listens on:

http://localhost:3001

Verify:

curl -sS http://localhost:3001/healthz
curl -sS http://localhost:3001/v1/models

API

GET /healthz

{"status":"ok"}

GET /v1/models

The model name comes from DEFAULT_MODEL; the default is mock-gpt.

POST /v1/chat/completions

Supports both stream=false and stream=true.

Non-streaming:

curl --request POST \
  --url 'http://localhost:3001/v1/chat/completions' \
  --header 'Content-Type: application/json' \
  --data '{"model":"mock-gpt","messages":[{"role":"user","content":"hello"}],"stream":false}'

Streaming:

curl --no-buffer --request POST \
  --url 'http://localhost:3001/v1/chat/completions?ttft_ms=300&chunk_delay_ms=50&completion_tokens=10' \
  --header 'Content-Type: application/json' \
  --data '{"model":"mock-gpt","messages":[{"role":"user","content":"hello"}],"stream":true}'

POST /v1/embeddings

Returns a fixed OpenAI-compatible embedding response. It does not call a real model.

Scenario Paths

Use scenario paths when you want multiple gateway channels to point at the same mock-provider deployment but behave differently:

/scenario/{scenario}/v1/models
/scenario/{scenario}/v1/chat/completions
/scenario/{scenario}/v1/embeddings

Built-in scenarios:

Scenario	Behavior
healthy	No additional injected fault
flaky-500	20% of requests return 500
flaky-429	20% of requests return 429
timeout	5% of requests wait 30s, then return 504
slow-ttft	Streaming TTFT is forced to 2000ms; non-streaming delay is forced to 2000ms
always-500	100% of requests return 500, useful for smoke/debug
always-429	100% of requests return 429, useful for smoke/debug

Scenarios are applied after query-string controls are parsed. That makes the path authoritative for channel-level behavior. For example, /scenario/always-429/v1/chat/completions?error_rate=0 still returns 429.

Mock Controls

Control behavior with query parameters:

Parameter	Description	Example
delay_ms	Whole-response delay for non-streaming requests	delay_ms=1000
ttft_ms	Time to first token for streaming requests	ttft_ms=300
chunk_delay_ms	Delay between streaming chunks	chunk_delay_ms=50
prompt_tokens	Simulated input tokens	prompt_tokens=100
completion_tokens	Simulated output tokens; also controls streaming content chunk count	completion_tokens=200
error_rate	Random error probability, from 0 to 1	error_rate=0.05
error_status	Error status code, default 500	error_status=429
timeout_rate	Random timeout probability, from 0 to 1	timeout_rate=0.01
timeout_ms	Timeout sleep duration, default 30000	timeout_ms=30000

Examples:

/v1/chat/completions?delay_ms=1000
/v1/chat/completions?error_rate=0.05&error_status=429
/v1/chat/completions?ttft_ms=300&chunk_delay_ms=50&completion_tokens=200
/v1/chat/completions?timeout_rate=0.01&timeout_ms=30000

Error response:

{
  "error": {
    "message": "mock upstream error",
    "type": "mock_error",
    "code": "mock_error"
  }
}

Gateway Integration

For an OpenAI-compatible gateway, configure a provider/channel/upstream like this:

Field	Value
Provider type	OpenAI-compatible or OpenAI
Base URL	http://mock-provider:3001/v1 or http://localhost:3001/v1
API key	dummy-key
Model	mock-gpt

To test channel failover, configure several gateway channels that point to the same mock-provider service with different scenario paths:

Gateway channel	Base URL
healthy	http://mock-provider:3001/scenario/healthy/v1
flaky 500	http://mock-provider:3001/scenario/flaky-500/v1
flaky 429	http://mock-provider:3001/scenario/flaky-429/v1
timeout	http://mock-provider:3001/scenario/timeout/v1
slow TTFT	http://mock-provider:3001/scenario/slow-ttft/v1

If the gateway and mock provider run in the same docker compose network:

http://mock-provider:3001/v1

If the gateway runs on the host:

http://localhost:3001/v1

k6 Scripts

All scripts support:

Environment variable	Default	Description
BASE_URL	http://localhost:3001	Target gateway or mock provider URL
API_KEY	dummy-key	Placeholder bearer token
MODEL	mock-gpt	Model name
QUERY	Script-specific default	Query string appended to /v1/chat/completions

Smoke test:

k6 run k6/smoke.js

Run through a gateway:

BASE_URL=http://localhost:3000 API_KEY=dummy-key MODEL=mock-gpt k6 run k6/smoke.js

Non-streaming load test:

k6 run k6/chat-non-stream.js

Simulate upstream latency:

QUERY='?delay_ms=1000' k6 run k6/chat-non-stream.js

Streaming test:

k6 run k6/chat-stream.js

The default streaming query is:

?ttft_ms=300&chunk_delay_ms=50&completion_tokens=200

The k6 streaming script validates completed streams and measures full response duration. For true time-to-first-token measurement, use the incremental SSE stream probe:

API_KEY=your-key go run ./tools/stream-probe/main.go \
  -url 'http://localhost:3001/v1/chat/completions?ttft_ms=300&chunk_delay_ms=50&completion_tokens=200' \
  -concurrency 20 \
  -duration 2m

It reports HTTP response-header latency, time to the first SSE data: chunk, and complete stream duration separately.

Stress test:

k6 run k6/stress.js

Soak test:

k6 run k6/soak.js

Suggested Test Flow

1. Smoke test

Goal: verify basic connectivity.

Suggested load: 1 VU for 30 seconds.

2. Load test

Goal: understand normal gateway capacity.

Suggested load: 50 / 100 / 200 VUs for 10-20 minutes.

3. Stress test

Goal: find gateway bottlenecks.

Suggested load: 100 / 200 / 500 / 1000 VUs.

4. Streaming test

Goal: validate SSE long-lived connections and buffering behavior.

Suggested load: 100 / 500 / 1000 / 3000 concurrent connections.

Suggested mock parameters:

ttft_ms=300
chunk_delay_ms=50
completion_tokens=200

5. Fault injection

Goal: validate retry behavior, billing correctness, logs, error handling, and circuit breaking.

Suggested mock parameters:

error_rate=0.01&error_status=429
error_rate=0.02&error_status=500
timeout_rate=0.01&timeout_ms=30000

6. Soak test

Goal: validate long-running stability.

Suggested load: 50%-70% of expected production concurrency for 1-6 hours.

Metrics to Watch

Gateway:

• QPS
• P50 / P95 / P99 latency
• Error rate
• HTTP 429 / 500 / 502 / 504
• Active SSE connections
• Request log write latency
• Quota or billing deduction correctness
• Whether failed requests are billed repeatedly
• Retry amplification

System:

• CPU
• Memory
• Goroutine / thread count
• File descriptors
• Network connections
• Database connection pool
• Cache connection pool
• Slow queries
• Log table or log sink growth

Profiling:

• CPU profile
• Heap profile
• Goroutine profile
• Flame graphs
• Mutex / block profile

Roadmap

• Long-lived connection profiles for streaming tests, including configurable idle periods and heartbeat chunks
• More SSE patterns, such as role-only chunks, empty deltas, tool-call chunks, abrupt disconnects, and malformed frames
• Richer error scenarios: 401, 403, 408, 409, 429, 500, 502, 503, 504, and provider-specific error bodies
• Retry-oriented scenarios, including partial failures, slow-first-attempt then success, and deterministic failure sequences
• Request-size controls for large prompts, long message histories, and large tool/function call payloads
• Token accounting modes for validating gateway quota and billing behavior
• Per-route controls for embeddings, chat, model listing, and future OpenAI-compatible endpoints
• Metrics endpoint for mock-provider-side request counts, latency buckets, active streams, and injected failures
• Config-file based scenarios in addition to query-string controls
• Optional response templates for custom provider shapes while keeping OpenAI-compatible defaults

Acceptance Checklist

• docker compose up starts the mock provider
• GET /healthz returns 200
• GET /v1/models returns mock-gpt
• POST /v1/chat/completions works with stream=false
• POST /v1/chat/completions works with stream=true
• delay_ms works
• ttft_ms works
• chunk_delay_ms works
• error_rate works
• timeout_rate works
• k6 smoke script runs
• k6 non-streaming script runs
• k6 streaming script runs