Skip to content

fix(recorder): multi-turn runs must produce distinct fixtures (1.19.3)#166

Merged
AlemTuzlak merged 2 commits intomainfrom
worktree-snazzy-swimming-hamming
May 8, 2026
Merged

fix(recorder): multi-turn runs must produce distinct fixtures (1.19.3)#166
AlemTuzlak merged 2 commits intomainfrom
worktree-snazzy-swimming-hamming

Conversation

@AlemTuzlak
Copy link
Copy Markdown
Contributor

@AlemTuzlak AlemTuzlak commented May 8, 2026

Summary

  • --record mode now writes turnIndex and hasToolResult to the saved fixture's match block, so multi-turn agent runs (tool-call → tool-result → follow-up) record as N distinct fixtures instead of collapsing into one and silently dropping the follow-up turn on replay.
  • Matcher and types already supported both fields (added in 1.16.0); only the recorder was lagging. This commit closes that gap.
  • Bumps to 1.19.3 (patch — purely additive on the recorder side, existing fixtures continue to match unchanged).

What was broken

A single agent run usually issues several LLM calls that share the same last user message:

  1. Turn 0: messages: [user] → tool call.
  2. Turn 1: messages: [user, assistant(tool_call), tool(result)] → text reply.

buildFixtureMatch wrote only userMessage. Recording call 0 wrote a fixture with match: { userMessage: "..." } and pushed it into the in-memory fixtures array. The matcher then matched that fixture for call 1 (substring of the last user message is unchanged), the recorder was skipped, the canned tool-call response was replayed, and the follow-up turn was silently lost. Replays of the resulting single fixture looped on the tool call until the framework's recursion limit fired.

CopilotKit's beautiful-chat showcase compose worked around this by hot-patching dist/recorder.js at container start — fragile and dropped on every container restart.

What changed

  • src/recorder.tsbuildFixtureMatch writes turnIndex (count of assistant messages) and hasToolResult (any prior role:"tool" present) for chat-shaped requests. Embedding short-circuit and multimedia endpoints (no messages) are guarded out. Composes cleanly with the model field added in 1.18.0 for fal.ai.
  • src/recorder.tsisEmptyMatch narrowed to check only userMessage/inputText/endpoint so the empty-match-warning behavior is preserved (the new disambiguators don't constitute a meaningful match key on their own).
  • src/__tests__/recorder.test.ts — new recorder multi-turn disambiguation describe block with 4 tests:
    • 2 LLM calls in one tool-using run record as 2 fixtures with distinct match.
    • 3 LLM calls (chained tool calls) record as 3 fixtures with distinct match.
    • Recorded fixtures replay deterministically against a fresh aimock.
    • Two demos with the same first-turn userMessage but distinct disambiguated follow-ups stay isolated.
  • CHANGELOG.md + package.json — 1.19.3 release notes and version bump.

Compatibility

  • Hand-written fixtures: no schema change. turnIndex and hasToolResult were already accepted by the validator and matcher (1.16.0). Existing fixtures that lack these fields continue to match exactly as before.
  • Recorded fixtures from prior versions: replay unchanged — matcher only checks the new fields when the fixture defines them.
  • Embeddings, image, speech, transcription, video, fal.ai: unaffected — embedding short-circuit untouched, the new field writes are gated on messages.length > 0, and the model field added in 1.18.0 for fal.ai composes orthogonally with the new fields.

Test plan

  • pnpm test src/__tests__/recorder.test.ts — 85/85 pass (4 new + 81 existing)
  • pnpm test src/__tests__/turn-index.test.ts — 12/12 pass (matcher coverage unchanged)
  • pnpm test — 2832/2835 pass; the 3 failures are pre-existing Windows-specific infrastructure tests (symlink invocation, SIGTERM, XDG_CACHE_HOME path separators), all unrelated to this change.
  • pnpm run lint — clean.
  • pnpm exec prettier --check src/recorder.ts src/__tests__/recorder.test.ts CHANGELOG.md package.json — clean.
  • pnpm run build — clean (tsdown, 245 files, no type errors).

@AlemTuzlak AlemTuzlak force-pushed the worktree-snazzy-swimming-hamming branch from 75658a1 to dd03edf Compare May 8, 2026 14:15
@AlemTuzlak AlemTuzlak changed the title fix(recorder): multi-turn runs must produce distinct fixtures (1.16.5) fix(recorder): multi-turn runs must produce distinct fixtures (1.19.3) May 8, 2026
@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 8, 2026

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@166

commit: dd03edf

@AlemTuzlak AlemTuzlak merged commit bfd1081 into main May 8, 2026
22 checks passed
@AlemTuzlak AlemTuzlak deleted the worktree-snazzy-swimming-hamming branch May 8, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants