Skip to content

[Platform] Verify the recorded request signature when replaying a cassette #2443

Description

@wachterjohannes

Test\Replay\HttpCassette writes a signature for every recorded request (method, URL and body hash), but nothing ever reads it. next() returns only the response half of an interaction, so replay is pure FIFO: the cassette decides what comes back, regardless of what the bridge actually sent.

That leaves half the pipeline uncovered. Contract normalizers and ModelClient payload building can regress without a single replay test noticing. Sending a completely wrong model through the recorded examples of #2437 changes nothing:

// src/platform/src/Bridge/OpenResponses/ModelClient.php
'body' => $this->encodeJsonBody(array_merge($options, ['model' => 'totally-wrong-model'], $payload)),
OK (5 tests, 10 assertions)

A dropped option, a wrong endpoint or a mangled message payload behaves the same way.

The data is already there, so the fix is small: compare the outgoing request against the recorded signature on replay and fail loudly on a mismatch.

Two things to decide:

  • Strictness. Failing hard on any mismatch is the point, but it makes every cassette brittle against unrelated payload changes (a new default option would invalidate every cassette at once). An opt-in flag on CassetteHttpClient, defaulting to on for AbstractBridgeReplayTestCase and off for the examples harness, may be the better balance.
  • Granularity. The signature is a hash, so a mismatch can only say "differs", not what differs. Keeping the normalized request body in the cassette (it already is, redacted) allows a readable diff in the failure message, which is what makes such a failure actionable.

Follow-up to #2128, which introduced the scaffolding, and #2437, which puts it in CI as bridge integration testing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureNew featurePlatformIssues & PRs about the AI Platform component

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions