Context. Replay mode intentionally compares the request side, so a changed prompt trips the PR check. That is deliberate: it is the signal that catches prompt-edit regressions. But not every workflow wants that. Someone iterating on prompts, or who only cares that the agent still calls the right tools with sane arguments and produces an equivalent output, currently has no way to exclude the prompt/request content from the comparison.
Proposed. An opt-in mode (config flag, e.g. ignore_request_content=True or a replay_scope option) that skips the request-side diff and only asserts on tool call shape (structural + arguments + model tool requests) and the final output. Strict full comparison stays the default.
Acceptance. Config plumbing through DiffConfig/AgentAsserter/config.py/pytest ini; unit tests for both modes; docs note on when to use it (prompt-iteration workflows).
Source: community feedback (r/LLMDevs launch thread).
Context. Replay mode intentionally compares the request side, so a changed prompt trips the PR check. That is deliberate: it is the signal that catches prompt-edit regressions. But not every workflow wants that. Someone iterating on prompts, or who only cares that the agent still calls the right tools with sane arguments and produces an equivalent output, currently has no way to exclude the prompt/request content from the comparison.
Proposed. An opt-in mode (config flag, e.g.
ignore_request_content=Trueor areplay_scopeoption) that skips the request-side diff and only asserts on tool call shape (structural + arguments + model tool requests) and the final output. Strict full comparison stays the default.Acceptance. Config plumbing through
DiffConfig/AgentAsserter/config.py/pytest ini; unit tests for both modes; docs note on when to use it (prompt-iteration workflows).Source: community feedback (r/LLMDevs launch thread).