Harden replay-proxy "Available tools" normalization against built-in tool set changes

[stephentoub]

Problem

The Copilot SDK C# tests leg in github/copilot-agent-runtime (a non-blocking canary that replays these E2E snapshots against runtime HEAD) started timing out at the 45-minute cap, e.g. this run. Every request logged No cached response found for POST /chat/completions and retried forever.

Root cause

When a model calls a tool that doesn't exist (the harness deliberately triggers this via a fake report_intent tool), the runtime replies with:

Tool 'report_intent' does not exist. Available tools that can be called are <full built-in tool list>.

The replay proxy previously only normalized the platform-specific shell tool names inside that list and froze the rest of the enumeration verbatim in ~54 snapshots. The runtime graduated multi-turn subagents to always-on and added a new default built-in tool, write_agent, to that list. The stored snapshots (..., read_agent, list_agents, grep, glob, task.) no longer matched the live request (..., read_agent, list_agents, write_agent, grep, glob, task.), so no snapshot matched, each request retried until it exhausted the suite budget, and the whole leg timed out.

This is a by-design runtime change (multi-turn subagents GA); the brittleness lives in this harness, which shouldn't re-record on every built-in tool set change.

Fix

Collapse the entire enumeration after Available tools that can be called are (up to the terminating .) to a stable ${available_tools} placeholder:

• normalizeAvailableToolNames now replaces the whole list on the live-request side.
• New normalizeStoredToolMessages, applied in loadStoredData, re-normalizes the same string in stored snapshots at load time (the result normalizers otherwise only run against live requests, so the stored side would keep its stale list).

Both sides collapse to ${available_tools} and match without re-recording, and stay matched as the built-in tool set evolves across runtime versions and platforms. Removed the now-unused shell-tool-family normalizer that only served the old list-preserving path.

Testing

• cd test/harness && npm test → all 37 harness tests pass.
• Added a regression test proving a legacy snapshot (tool list without write_agent) still matches a live request whose tool result includes write_agent. Verified it fails (HTTP 500 / no match) without the fix and passes with it.

Generated by Copilot

[Copilot]

Pull request overview

Hardens replay matching against runtime built-in tool-set changes.

Changes:

• Normalizes available-tool enumerations in live and stored messages.
• Removes obsolete shell-family normalization.
• Adds recording and legacy-snapshot regression tests.

Show a summary per file

File	Description
test/harness/replayingCapiProxy.ts	Adds stable available-tools normalization.
test/harness/replayingCapiProxy.test.ts	Tests normalization and replay compatibility.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Medium