Echidna manual seed in debug mode

[Thomas-Malcolm]

Randomly generate (and log) a seed for Echidna to use when fuzzing for new input in debug mode so that behaviour can be reproduced in testing?

i.e. when we invoke Echidna through hybrid-echidna, we should pass the --seed <value> parameter through on the "cli", where <value> is a seed we generate internally, but only in debug mode.

[Boyan-MILANOV]

I'm not quite sure to understand why we would like to log the seed when running in debug mode? If users want to debug, they can also specify the seed through the command line.

[Thomas-Malcolm]

My thought was that when debugging sometimes you'll run into an unexpected (non-deterministic) error, which can be annoying to reproduce. If we have hybrid-echidna explicitly telling the developer what seed was used in echidna to fuzz, then that could be passed as a CLI option by the developer manually in another run to attempt to reproduce the error.

[Boyan-MILANOV]

Mmmh I see. The thing is that unless the user specifies a specific seed to use by Echidna we will use a different seed at every fuzzing iteration, so we'll have to record all of them for reproducibility.

I guess instead of relying on Echidna manually generating a seed we can generate random seeds (if --seed wasn't specified of course) in Optik and put them in the Echidna command line. This way the debug logs will contain all the seeds for all invocations, and that'll be enough for reproducing some bugs.

I'm also wondering wether we should make --seeds accept a list of multiple seeds, one for each fuzzing iteration?

[Thomas-Malcolm]

A potential problem with having --seeds accept a list of multiple seeds is that it's possible the number of seeds required before running may not be known. Perhaps it would be useful for when someone hands you debug output however?

One solution could be that all seeds passed to Echidna are generated by a single input from --seed. Thus the list of seeds generated (however many that may be) will always be the same coming from the single --seed input.