Search before asking
Description
What problem does it solve: today, validating a behavior change to an agent (a new prompt, a changed tool set, an updated routing strategy) against realistic state generally means either testing against synthetic fixtures or deploying the change and watching production. Flink Agents already carries durable per-key state through Flink's own checkpoint/savepoint mechanism (the recovery model #1016 is about aligning), which is exactly the ingredient a safer middle option needs.
What value does it bring: a dry-run mode that takes a savepoint from a running (or previously run) agent job, replays a defined set of keys/events against a candidate agent definition (the new prompt, tool set, or routing strategy under evaluation), and reports the resulting decisions/outputs side-by-side with what the current definition would produce for the same input — without those replayed actions taking effect against any real external tool, model call side effect that isn't idempotent, or downstream sink. It's the same idea as a Flink savepoint-based canary, specialized to "did my agent's behavior actually change the way I expect, on data shaped like production."
Do you already have a solution in mind: no PR yet — the open design question is how much of the replay path can reuse existing durable-state and event-recording machinery (the same Event Log format #970's Phase 1 OTel work already consumes) versus needing a genuinely new replay-only execution mode, so I'd like input on which direction the maintainers would actually want to carry before proposing a concrete shape.
Are you willing to submit a PR?
Search before asking
Description
What problem does it solve: today, validating a behavior change to an agent (a new prompt, a changed tool set, an updated routing strategy) against realistic state generally means either testing against synthetic fixtures or deploying the change and watching production. Flink Agents already carries durable per-key state through Flink's own checkpoint/savepoint mechanism (the recovery model #1016 is about aligning), which is exactly the ingredient a safer middle option needs.
What value does it bring: a dry-run mode that takes a savepoint from a running (or previously run) agent job, replays a defined set of keys/events against a candidate agent definition (the new prompt, tool set, or routing strategy under evaluation), and reports the resulting decisions/outputs side-by-side with what the current definition would produce for the same input — without those replayed actions taking effect against any real external tool, model call side effect that isn't idempotent, or downstream sink. It's the same idea as a Flink savepoint-based canary, specialized to "did my agent's behavior actually change the way I expect, on data shaped like production."
Do you already have a solution in mind: no PR yet — the open design question is how much of the replay path can reuse existing durable-state and event-recording machinery (the same Event Log format #970's Phase 1 OTel work already consumes) versus needing a genuinely new replay-only execution mode, so I'd like input on which direction the maintainers would actually want to carry before proposing a concrete shape.
Are you willing to submit a PR?