Skip to content

Commit d3eebc2

Browse files
authored
test: expand sequential response coverage across providers and edge cases (#52)
## Summary Comprehensive test coverage expansion and two implementation improvements across the entire llmock codebase. ### Implementation Changes **Chaos injection feature** — New fault tolerance testing capability: - `ChaosConfig` with `dropRate`, `malformedRate`, `disconnectRate` (all 0-1 probability) - Three override levels: server defaults < fixture-level < per-request headers (`X-LLMock-Chaos-*`) - All HTTP handlers wired (Chat Completions, Responses, Messages, Gemini, Bedrock, Embeddings) - CLI flags: `--chaos-drop`, `--chaos-malformed`, `--chaos-disconnect` - Journal recording with `chaosAction` field - `setChaos()`/`clearChaos()` on LLMock class (propagates to running server) **Provider-specific error formats** — Previously all providers returned identical OpenAI-style error JSON. Now: - Anthropic/Bedrock: `{ type: "error", error: { type, message } }` - Gemini: `{ error: { code, message, status } }` - OpenAI: unchanged (already correct) ### Test Coverage (848 → 906 tests, +58) **Sequential Responses** (+9): error responses, tool calls, gap indices, Anthropic/Gemini sequences, predicate+sequenceIndex, model matching, resetMatchCounts, concurrent requests **Embeddings** (+14): base64 encoding_format, empty input, sequential embeddings, Unicode, error status codes (401, 503) **Structured Output** (+9): streaming json_object, json_schema matching, combined responseFormat+model+userMessage routing, malformed response_format, sequenceIndex+error interleaving **Streaming Physics** (+6): streamingProfile+truncateAfterChunks combined, streamingProfile+disconnectAfterMs, truncateAfterChunks:1 boundary, large chunkSize, empty content streaming, Anthropic truncation **Chaos** (+12): server-level drop/malformed on Anthropic/Gemini/Bedrock, fixture-level chaos scoping, journal recording, setChaos/clearChaos integration **Error Format** (+6): Anthropic/Gemini/Bedrock format conformance, streaming error returns JSON not SSE, cross-provider format verification **Provider Coverage** (+4): Azure streaming, onToolResult live verification, onToolCall response verification, Groq streaming through proxy **Fixtures** (+7): streamingProfile/chaos JSON passthrough, validation boundary conditions (status 99/100/599), error status default 500 **WebSocket** (+5): concurrent message serialization (no interleaving), frame fragmentation (continuation frames), multiple tool calls (Realtime + Responses WS) ### Bug Fixes Found During Testing - `setChaos()`/`clearChaos()` silently failed on running servers (captured snapshot at startup, not live reference) - Error responses used OpenAI format for all providers (Anthropic, Gemini, Bedrock now use native formats) ## Test plan - [x] All 906 tests pass - [x] Prettier clean - [x] ESLint clean - [x] Full build passes - [x] 7-agent code review loop — zero actionable findings
2 parents f3ad7f5 + bbd3554 commit d3eebc2

27 files changed

Lines changed: 2868 additions & 71 deletions

src/__tests__/api-conformance.test.ts

Lines changed: 393 additions & 0 deletions
Large diffs are not rendered by default.

src/__tests__/azure.test.ts

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,54 @@ describe("Azure OpenAI: journal recording", () => {
271271
});
272272
});
273273

274+
describe("Azure OpenAI: streaming", () => {
275+
it("streaming through Azure deployment path", async () => {
276+
const fixtures: Fixture[] = [
277+
{
278+
match: { userMessage: "stream-test" },
279+
response: { content: "Azure streamed!" },
280+
},
281+
];
282+
instance = await createServer(fixtures);
283+
284+
const { status, body } = await httpPost(
285+
`${instance.url}/openai/deployments/my-gpt4/chat/completions?api-version=2024-02-01`,
286+
{
287+
model: "gpt-4",
288+
stream: true,
289+
messages: [{ role: "user", content: "stream-test" }],
290+
},
291+
);
292+
293+
expect(status).toBe(200);
294+
295+
// Parse SSE events
296+
const events: unknown[] = [];
297+
for (const line of body.split("\n")) {
298+
if (line.startsWith("data: ") && line !== "data: [DONE]") {
299+
events.push(JSON.parse(line.slice(6)));
300+
}
301+
}
302+
303+
expect(events.length).toBeGreaterThanOrEqual(3);
304+
305+
// All chunks should have chat.completion.chunk object type
306+
for (const event of events) {
307+
const ev = event as { object: string };
308+
expect(ev.object).toBe("chat.completion.chunk");
309+
}
310+
311+
// Content should be present across the chunks
312+
const contentParts = events
313+
.map((e) => (e as { choices: [{ delta: { content?: string } }] }).choices[0].delta.content)
314+
.filter(Boolean);
315+
expect(contentParts.join("")).toBe("Azure streamed!");
316+
317+
// Body ends with [DONE]
318+
expect(body).toContain("data: [DONE]");
319+
});
320+
});
321+
274322
describe("Azure OpenAI: 404 when no fixture matches", () => {
275323
it("returns 404 when no fixture matches the request", async () => {
276324
const fixtures: Fixture[] = [

src/__tests__/bedrock.test.ts

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,29 @@ describe("POST /model/{modelId}/invoke (error handling)", () => {
192192
expect(body.error.message).toBe("Rate limited");
193193
});
194194

195+
it("returns error in Anthropic format: { type: 'error', error: { type, message } }", async () => {
196+
instance = await createServer(allFixtures);
197+
const res = await post(
198+
`${instance.url}/model/anthropic.claude-3-5-sonnet-20241022-v2:0/invoke`,
199+
{
200+
anthropic_version: "bedrock-2023-05-31",
201+
max_tokens: 512,
202+
messages: [{ role: "user", content: "fail" }],
203+
},
204+
);
205+
206+
expect(res.status).toBe(429);
207+
const body = JSON.parse(res.body);
208+
// Bedrock uses Anthropic Messages format for errors
209+
expect(body.type).toBe("error");
210+
expect(body.error).toBeDefined();
211+
expect(body.error.type).toBe("rate_limit_error");
212+
expect(body.error.message).toBe("Rate limited");
213+
// Should NOT have OpenAI-style fields
214+
expect(body.status).toBeUndefined();
215+
expect(body.error.code).toBeUndefined();
216+
});
217+
195218
it("returns 404 when no fixture matches", async () => {
196219
instance = await createServer(allFixtures);
197220
const res = await post(

0 commit comments

Comments
 (0)