Background
The advanced_* benches drive a single real participant against recorded snapshot messages via run_simulated_protocol, but pass its Result straight to criterion, which discards it:
|preps| run_simulated_protocol(preps.participant, preps.protocol, preps.simulator),
A protocol that errors mid-run is simply timed as a fast sample. Adding an .expect() shows two benches are already failing on main:
advanced_ot_based_ecdsa (triples group):
AssertionFailed("q check failed") — the reported ~13.6ms is time-to-error, not triple generation time.
advanced_dkg: replay also returns Err.
The replay appears to diverge from the recorded run, and any change to message interleaving moves where it trips an assertion (discovered while benchmarking #3501, where the broken triples bench reported a bogus −84% "improvement").
Six bench files share the discard pattern and should be audited: advanced_ot_based_ecdsa, advanced_dkg, advanced_robust_ecdsa, advanced_eddsa_frost_sign_v1, advanced_eddsa_frost_sign_v2, ckd.
Acceptance Criteria
- All benches using
run_simulated_protocol fail loudly (e.g. .expect()) when the protocol returns an error, so criterion can never time a truncated run.
- The replay divergence behind the
advanced_ot_based_ecdsa triples and advanced_dkg failures is diagnosed and fixed, and those benches complete successfully.
- The
naive_* benches assert success on their run_protocol results as well.
Background
The
advanced_*benches drive a single real participant against recorded snapshot messages viarun_simulated_protocol, but pass itsResultstraight to criterion, which discards it:A protocol that errors mid-run is simply timed as a fast sample. Adding an
.expect()shows two benches are already failing onmain:advanced_ot_based_ecdsa(triples group):AssertionFailed("q check failed")— the reported ~13.6ms is time-to-error, not triple generation time.advanced_dkg: replay also returnsErr.The replay appears to diverge from the recorded run, and any change to message interleaving moves where it trips an assertion (discovered while benchmarking #3501, where the broken triples bench reported a bogus −84% "improvement").
Six bench files share the discard pattern and should be audited:
advanced_ot_based_ecdsa,advanced_dkg,advanced_robust_ecdsa,advanced_eddsa_frost_sign_v1,advanced_eddsa_frost_sign_v2,ckd.Acceptance Criteria
run_simulated_protocolfail loudly (e.g..expect()) when the protocol returns an error, so criterion can never time a truncated run.advanced_ot_based_ecdsatriples andadvanced_dkgfailures is diagnosed and fixed, and those benches complete successfully.naive_*benches assert success on theirrun_protocolresults as well.