Commit d3eebc2
authored
test: expand sequential response coverage across providers and edge cases (#52)
## Summary
Comprehensive test coverage expansion and two implementation
improvements across the entire llmock codebase.
### Implementation Changes
**Chaos injection feature** — New fault tolerance testing capability:
- `ChaosConfig` with `dropRate`, `malformedRate`, `disconnectRate` (all
0-1 probability)
- Three override levels: server defaults < fixture-level < per-request
headers (`X-LLMock-Chaos-*`)
- All HTTP handlers wired (Chat Completions, Responses, Messages,
Gemini, Bedrock, Embeddings)
- CLI flags: `--chaos-drop`, `--chaos-malformed`, `--chaos-disconnect`
- Journal recording with `chaosAction` field
- `setChaos()`/`clearChaos()` on LLMock class (propagates to running
server)
**Provider-specific error formats** — Previously all providers returned
identical OpenAI-style error JSON. Now:
- Anthropic/Bedrock: `{ type: "error", error: { type, message } }`
- Gemini: `{ error: { code, message, status } }`
- OpenAI: unchanged (already correct)
### Test Coverage (848 → 906 tests, +58)
**Sequential Responses** (+9): error responses, tool calls, gap indices,
Anthropic/Gemini sequences, predicate+sequenceIndex, model matching,
resetMatchCounts, concurrent requests
**Embeddings** (+14): base64 encoding_format, empty input, sequential
embeddings, Unicode, error status codes (401, 503)
**Structured Output** (+9): streaming json_object, json_schema matching,
combined responseFormat+model+userMessage routing, malformed
response_format, sequenceIndex+error interleaving
**Streaming Physics** (+6): streamingProfile+truncateAfterChunks
combined, streamingProfile+disconnectAfterMs, truncateAfterChunks:1
boundary, large chunkSize, empty content streaming, Anthropic truncation
**Chaos** (+12): server-level drop/malformed on
Anthropic/Gemini/Bedrock, fixture-level chaos scoping, journal
recording, setChaos/clearChaos integration
**Error Format** (+6): Anthropic/Gemini/Bedrock format conformance,
streaming error returns JSON not SSE, cross-provider format verification
**Provider Coverage** (+4): Azure streaming, onToolResult live
verification, onToolCall response verification, Groq streaming through
proxy
**Fixtures** (+7): streamingProfile/chaos JSON passthrough, validation
boundary conditions (status 99/100/599), error status default 500
**WebSocket** (+5): concurrent message serialization (no interleaving),
frame fragmentation (continuation frames), multiple tool calls (Realtime
+ Responses WS)
### Bug Fixes Found During Testing
- `setChaos()`/`clearChaos()` silently failed on running servers
(captured snapshot at startup, not live reference)
- Error responses used OpenAI format for all providers (Anthropic,
Gemini, Bedrock now use native formats)
## Test plan
- [x] All 906 tests pass
- [x] Prettier clean
- [x] ESLint clean
- [x] Full build passes
- [x] 7-agent code review loop — zero actionable findings27 files changed
Lines changed: 2868 additions & 71 deletions
File tree
- src
- __tests__
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
271 | 271 | | |
272 | 272 | | |
273 | 273 | | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
274 | 322 | | |
275 | 323 | | |
276 | 324 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
192 | 192 | | |
193 | 193 | | |
194 | 194 | | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
195 | 218 | | |
196 | 219 | | |
197 | 220 | | |
| |||
0 commit comments