|
| 1 | +--- |
| 2 | +name: write-fixtures |
| 3 | +description: Use when writing test fixtures for @copilotkit/llmock — mock LLM responses, tool call sequences, error injection, multi-turn agent loops, or debugging fixture mismatches |
| 4 | +--- |
| 5 | + |
| 6 | +# Writing llmock Test Fixtures |
| 7 | + |
| 8 | +## What llmock Is |
| 9 | + |
| 10 | +Zero-dependency mock LLM server. Fixture-driven. Multi-provider (OpenAI, Anthropic, Gemini). Runs a real HTTP server on a real port — works across processes, unlike MSW-style interceptors. WebSocket support for OpenAI Responses/Realtime and Gemini Live APIs. |
| 11 | + |
| 12 | +## Core Mental Model |
| 13 | + |
| 14 | +- **Fixtures** = match criteria + response |
| 15 | +- **First-match-wins** — order matters |
| 16 | +- All providers share one fixture pool (provider adapters normalize to `ChatCompletionRequest`) |
| 17 | +- Fixtures are stateless — no built-in multi-turn sequencing |
| 18 | +- Fixtures are live — mutations after `start()` take effect immediately |
| 19 | + |
| 20 | +## Match Field Reference |
| 21 | + |
| 22 | +| Field | Type | Matches Against | |
| 23 | +| ------------- | ----------------------------------------- | ------------------------------------------------------------------------- | |
| 24 | +| `userMessage` | `string` | Substring of last `role: "user"` message text | |
| 25 | +| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text | |
| 26 | +| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) | |
| 27 | +| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | |
| 28 | +| `model` | `string` | Exact match on `req.model` | |
| 29 | +| `model` | `RegExp` | Pattern test on `req.model` | |
| 30 | +| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request | |
| 31 | + |
| 32 | +**AND logic**: all specified fields must match. Empty match `{}` = catch-all. |
| 33 | + |
| 34 | +Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically extracted — `userMessage` matching works regardless of content format. |
| 35 | + |
| 36 | +## Response Types |
| 37 | + |
| 38 | +### Text |
| 39 | + |
| 40 | +```typescript |
| 41 | +{ |
| 42 | + content: "Hello!"; |
| 43 | +} |
| 44 | +``` |
| 45 | + |
| 46 | +### Tool Calls |
| 47 | + |
| 48 | +```typescript |
| 49 | +{ |
| 50 | + toolCalls: [{ name: "get_weather", arguments: '{"city":"SF"}' }]; |
| 51 | +} |
| 52 | +``` |
| 53 | + |
| 54 | +**`arguments` MUST be a JSON string**, not an object. This is the #1 mistake. |
| 55 | + |
| 56 | +### Error |
| 57 | + |
| 58 | +```typescript |
| 59 | +{ error: { message: "Rate limited", type: "rate_limit_error" }, status: 429 } |
| 60 | +``` |
| 61 | + |
| 62 | +## Common Patterns |
| 63 | + |
| 64 | +### Basic text fixture |
| 65 | + |
| 66 | +```typescript |
| 67 | +mock.onMessage("hello", { content: "Hi there!" }); |
| 68 | +``` |
| 69 | + |
| 70 | +### Tool call → tool result → final response (3-step agent loop) |
| 71 | + |
| 72 | +The most common pattern. Fixture 1 triggers the tool call, fixture 2 handles the tool result. |
| 73 | + |
| 74 | +```typescript |
| 75 | +// Step 1: User asks about weather → LLM calls tool |
| 76 | +mock.onMessage("weather", { |
| 77 | + toolCalls: [{ name: "get_weather", arguments: '{"city":"SF"}' }], |
| 78 | +}); |
| 79 | + |
| 80 | +// Step 2: Tool result comes back → LLM responds with text |
| 81 | +mock.addFixture({ |
| 82 | + match: { predicate: (req) => req.messages.at(-1)?.role === "tool" }, |
| 83 | + response: { content: "It's 72°F in San Francisco." }, |
| 84 | +}); |
| 85 | +``` |
| 86 | + |
| 87 | +**Why predicate, not userMessage?** After a tool call, the client replays the same conversation with the tool result appended. The user message hasn't changed — `userMessage: "weather"` would match the SAME fixture again, creating an infinite loop. |
| 88 | + |
| 89 | +### Predicate-based routing (same user message, different context) |
| 90 | + |
| 91 | +Common in supervisor/orchestrator patterns where the system prompt changes: |
| 92 | + |
| 93 | +```typescript |
| 94 | +mock.addFixture({ |
| 95 | + match: { |
| 96 | + predicate: (req) => { |
| 97 | + const sys = req.messages.find((m) => m.role === "system")?.content ?? ""; |
| 98 | + return typeof sys === "string" && sys.includes("Flights found: false"); |
| 99 | + }, |
| 100 | + }, |
| 101 | + response: { toolCalls: [{ name: "search_flights", arguments: "{}" }] }, |
| 102 | +}); |
| 103 | +``` |
| 104 | + |
| 105 | +### Catch-all (always add one) |
| 106 | + |
| 107 | +Prevents unmatched requests from returning 404 and crashing the test: |
| 108 | + |
| 109 | +```typescript |
| 110 | +mock.addFixture({ |
| 111 | + match: { predicate: () => true }, |
| 112 | + response: { content: "I understand. How can I help?" }, |
| 113 | +}); |
| 114 | +``` |
| 115 | + |
| 116 | +### Tool result catch-all with prependFixture |
| 117 | + |
| 118 | +Must go at the front so it matches before substring-based fixtures: |
| 119 | + |
| 120 | +```typescript |
| 121 | +mock.prependFixture({ |
| 122 | + match: { predicate: (req) => req.messages.at(-1)?.role === "tool" }, |
| 123 | + response: { content: "Done!" }, |
| 124 | +}); |
| 125 | +``` |
| 126 | + |
| 127 | +### Stream interruption simulation (v1.3.0+) |
| 128 | + |
| 129 | +```typescript |
| 130 | +mock.onMessage( |
| 131 | + "long response", |
| 132 | + { content: "This will be cut short..." }, |
| 133 | + { |
| 134 | + truncateAfterChunks: 3, // Stop after 3 SSE chunks |
| 135 | + disconnectAfterMs: 500, // Or disconnect after 500ms |
| 136 | + }, |
| 137 | +); |
| 138 | +``` |
| 139 | + |
| 140 | +### Error injection (one-shot) |
| 141 | + |
| 142 | +```typescript |
| 143 | +mock.nextRequestError(429, { message: "Rate limited", type: "rate_limit_error" }); |
| 144 | +// Next request gets 429, then fixture auto-removes itself |
| 145 | +``` |
| 146 | + |
| 147 | +### JSON fixture files |
| 148 | + |
| 149 | +```json |
| 150 | +{ |
| 151 | + "fixtures": [ |
| 152 | + { |
| 153 | + "match": { "userMessage": "hello" }, |
| 154 | + "response": { "content": "Hi!" } |
| 155 | + } |
| 156 | + ] |
| 157 | +} |
| 158 | +``` |
| 159 | + |
| 160 | +JSON files cannot use `RegExp` or `predicate` — those are code-only features. |
| 161 | + |
| 162 | +Load with `mock.loadFixtureFile("./fixtures/greetings.json")` or `mock.loadFixtureDir("./fixtures/")`. |
| 163 | + |
| 164 | +## Critical Gotchas |
| 165 | + |
| 166 | +1. **Order matters** — first match wins. Specific fixtures before general ones. Use `prependFixture()` to force priority. |
| 167 | + |
| 168 | +2. **`arguments` must be a JSON string** — `"arguments": "{\"key\":\"value\"}"` not `"arguments": {"key":"value"}`. The type system enforces this but JSON fixtures can get it wrong silently. |
| 169 | + |
| 170 | +3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). |
| 171 | + |
| 172 | +4. **Tool result messages don't change the user message** — after a tool call, the client sends the same conversation + tool result. Matching on `userMessage` will hit the SAME fixture again → infinite loop. Always use `predicate` checking `role === "tool"` for tool results. |
| 173 | + |
| 174 | +5. **`clearFixtures()` preserves the array reference** — uses `.length = 0`, not reassignment. The running server reads the same array object. |
| 175 | + |
| 176 | +6. **Journal records everything** — including 404 "no match" responses. Use `mock.getLastRequest()` to debug mismatches. |
| 177 | + |
| 178 | +7. **All providers share fixtures** — a fixture matching "hello" works whether the request comes via `/v1/chat/completions` (OpenAI), `/v1/messages` (Anthropic), or Gemini endpoints. |
| 179 | + |
| 180 | +8. **WebSocket uses the same fixture pool** — no special setup needed for WebSocket-based APIs (OpenAI Responses WS, Realtime, Gemini Live). |
| 181 | + |
| 182 | +## Debugging Fixture Mismatches |
| 183 | + |
| 184 | +When a fixture doesn't match: |
| 185 | + |
| 186 | +1. **Inspect what the server received**: `mock.getLastRequest()` → check `body.messages` array |
| 187 | +2. **Check fixture order**: `mock.getFixtures()` returns fixtures in registration order |
| 188 | +3. **For `userMessage`**: match is against the LAST `role: "user"` message only, substring match (not exact) |
| 189 | +4. **Check the journal**: `mock.getRequests()` shows all requests including which fixture matched (or `null` for 404) |
| 190 | + |
| 191 | +## E2E Test Setup Pattern |
| 192 | + |
| 193 | +```typescript |
| 194 | +import { LLMock } from "@copilotkit/llmock"; |
| 195 | + |
| 196 | +// Setup — port: 0 picks a random available port |
| 197 | +const mock = new LLMock({ port: 0 }); |
| 198 | +mock.loadFixtureDir("./fixtures"); |
| 199 | +await mock.start(); |
| 200 | +process.env.OPENAI_BASE_URL = `${mock.url}/v1`; |
| 201 | + |
| 202 | +// Per-test cleanup |
| 203 | +afterEach(() => mock.reset()); // clears fixtures AND journal |
| 204 | + |
| 205 | +// Teardown |
| 206 | +afterAll(async () => await mock.stop()); |
| 207 | +``` |
| 208 | + |
| 209 | +### Static factory shorthand |
| 210 | + |
| 211 | +```typescript |
| 212 | +const mock = await LLMock.create({ port: 0 }); // creates + starts in one call |
| 213 | +``` |
| 214 | + |
| 215 | +## API Quick Reference |
| 216 | + |
| 217 | +| Method | Purpose | |
| 218 | +| ------------------------------------- | ---------------------------------- | |
| 219 | +| `addFixture(f)` | Append fixture (last priority) | |
| 220 | +| `addFixtures(f[])` | Append multiple | |
| 221 | +| `prependFixture(f)` | Insert at front (highest priority) | |
| 222 | +| `clearFixtures()` | Remove all fixtures | |
| 223 | +| `getFixtures()` | Read current fixture list | |
| 224 | +| `on(match, response, opts?)` | Shorthand for `addFixture` | |
| 225 | +| `onMessage(pattern, response, opts?)` | Match by user message | |
| 226 | +| `onToolCall(name, response, opts?)` | Match by tool name in `tools[]` | |
| 227 | +| `onToolResult(id, response, opts?)` | Match by `tool_call_id` | |
| 228 | +| `nextRequestError(status, body?)` | One-shot error, auto-removes | |
| 229 | +| `loadFixtureFile(path)` | Load JSON fixture file | |
| 230 | +| `loadFixtureDir(path)` | Load all JSON files in directory | |
| 231 | +| `start()` | Start server, returns URL | |
| 232 | +| `stop()` | Stop server | |
| 233 | +| `reset()` | Clear fixtures + journal | |
| 234 | +| `getRequests()` | All journal entries | |
| 235 | +| `getLastRequest()` | Most recent journal entry | |
| 236 | +| `clearRequests()` | Clear journal only | |
| 237 | +| `url` / `baseUrl` | Server URL (throws if not started) | |
| 238 | +| `port` | Server port number | |
0 commit comments