Skip to content

Simulated-replay benches silently measure time-to-error instead of protocol runtime #3516

Description

@gilcu3

Background

The advanced_* benches drive a single real participant against recorded snapshot messages via run_simulated_protocol, but pass its Result straight to criterion, which discards it:

|preps| run_simulated_protocol(preps.participant, preps.protocol, preps.simulator),

A protocol that errors mid-run is simply timed as a fast sample. Adding an .expect() shows two benches are already failing on main:

  • advanced_ot_based_ecdsa (triples group):
    AssertionFailed("q check failed") — the reported ~13.6ms is time-to-error, not triple generation time.
  • advanced_dkg: replay also returns Err.

The replay appears to diverge from the recorded run, and any change to message interleaving moves where it trips an assertion (discovered while benchmarking #3501, where the broken triples bench reported a bogus −84% "improvement").

Six bench files share the discard pattern and should be audited: advanced_ot_based_ecdsa, advanced_dkg, advanced_robust_ecdsa, advanced_eddsa_frost_sign_v1, advanced_eddsa_frost_sign_v2, ckd.

Acceptance Criteria

  • All benches using run_simulated_protocol fail loudly (e.g. .expect()) when the protocol returns an error, so criterion can never time a truncated run.
  • The replay divergence behind the advanced_ot_based_ecdsa triples and advanced_dkg failures is diagnosed and fixed, and those benches complete successfully.
  • The naive_* benches assert success on their run_protocol results as well.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions