Add counterfactual checks for workload created tests

[SeanTAllen]

At the moment, workload creates tests and we don't know their quality. We need to introduce a step as part of tests that requires the agent to see the test fail before calling it "good". This could be introducing a brown m&m to find. It could be creating a regression workload for known fixed issues and rolling back before the fix to make sure it actually is fixed.

I think this is probably at least 2 parts. There is the deeper knowledge we get from the above, but we should also do intentional simple changes to the tests themselves to see them fail. This simple test manipulation that works for example based tests and more focused "unit level properties" might not be appropriate in the context of a "we don't see data corruption" level property.

The how here is up in the air. But tests that we haven't seen fail are tests that we can't know have any value.

[SeanTAllen]

After some thinking, the "deeper" validation has two distinct strategies:

Regression against known bugs. Run the current workload against a known-buggy SUT version. If the property fires, you have real evidence it catches what it claims to catch. This requires a history of known bugs and which versions had them, but doesn't require source access or the ability to build — you're just swapping the SUT image.

Intentional defect introduction. Reason backward from the property to find the point in the SUT where the invariant gets established, then break that specific thing and confirm the property fires. This is basically lineage-driven fault injection — you're not randomly mutating code, you're tracing the causal chain from "this property holds" back to "this code path makes it hold" and introducing a realistic violation at that point. Undo the change after validation. This requires source access and the ability to build, but doesn't require any bug history. It works for brand-new properties where there's nothing to regress against.

The quality of the intentional defect matters. Too shallow (hardcoded wrong return value) and you prove the assertion triggers but not that it catches real bugs. Too deep and it might not trigger within the test's time budget. The sweet spot is a defect that could plausibly be an actual bug.

Both strategies belong in the skill. Regression is strongest evidence but needs history. Intentional defect is more general-purpose but harder — the agent has to understand the SUT's internals to reason backward from the property.

[SeanTAllen]

On the "simple" counterfactual case — the quick flip-and-check that works so well for unit tests — I don't think it has independent value in the Antithesis context.

The reason simple counterfactuals work for unit tests is the fast local feedback loop. Flip an assertion, run the test in seconds, see it fail, flip it back. You've proved the assertion is reachable and meaningful. But Antithesis SDK assertions are no-ops outside the Antithesis environment. You can run a test command locally and verify it doesn't crash, connects to the SUT, and exits cleanly — but snouty validate already covers that ground. The assertions themselves can't fire locally, so you can't see them fail.

The only way to verify that an assertion actually catches what it claims to catch is to run it in Antithesis against something that should trigger it. And at that point you're in "deeper" territory — either regression against a known-buggy version or intentional defect introduction.

So the scope here is the two deeper strategies from the previous comment. The simple case doesn't carry its weight for Antithesis workloads.