Invariant Fuzzer: Phased improvement plan to close gap with Echidna/Medusa (SCFuzzBench)

[grandizzy]

Context

SCFuzzBench benchmarks smart contract fuzzers (Echidna, Medusa, Foundry, Recon) against real DeFi protocols (Aave v4, Superform, Liquity). Current results show Foundry finding 0-3 bugs vs Echidna's 10+ on the same targets with the same time budget.

Analysis of the Foundry invariant testing engine identified several root causes, ranging from architectural blockers to corpus/coverage quality issues. This issue tracks the phased improvement plan.

Benchmark Baseline (Aave v4, 24h, SCFuzzBench)

Fuzzer	Bugs Found	AUC (norm)	Coverage Proxy
Echidna 2.3.1	10 [9,10]	0.816	456k
Medusa 1.4.1	10 [9,10]	0.754	5.2k
Foundry v1.6.0-rc1	3 [3,3]	0.272	374

---

Phase 1: Architectural Blockers (prerequisite)

These must land first — without them, the fuzzer uses ~4% of its time budget because trivially-findable assertion failures stop exploration immediately.

• fail_on_assert config — Gate assertion-failure detection behind a config flag so users can opt in to treating Solidity assert() panics in target functions as invariant breaks.

  • Merged: 419f787 (#14275)

• continuous_run mode — Continue running after finding an invariant/assertion failure. Record it and keep searching for more failures. Currently, the first assert_canary(0) kills every invariant test in <100 runs.

  • PR: feat(invariant): assert all invariants by default (assert_all) #12587 (open, review required)
  • Impact: 0-3 bugs → 6 bugs (all found in <15s)

• Preflight check in continuous_run mode — When continuous_run = true, the preflight invariant check (which verifies invariants hold before fuzzing starts) should record failures but not abort the campaign. Currently an always-failing canary invariant causes "failed to set up invariant testing environment".

  • Location: crates/evm/evm/src/executors/invariant/mod.rs (prepare_test → preflight check)
  • Should be part of feat(invariant): assert all invariants by default (assert_all) #12587

---

Phase 2: Dictionary Quality

The fuzz dictionary collects values from storage, logs, call results, and push bytes. Currently, all values learned during an invariant run are thrown away at the end of each run via FuzzDictionary::revert(). Values from coverage-producing runs should persist to compound across runs.

• Selective dictionary persistence — When an invariant run produces new coverage, promote collected values to the persistent baseline (advance the revert watermark) instead of discarding them. Bounded at ~4096 values / ~512 addresses to prevent unbounded growth. Same pattern as the existing persistent_values mechanism for sancov trace-cmp.

  • Location: crates/evm/fuzz/src/strategies/state.rs (add promote_ephemeral_values())
  • Location: crates/evm/evm/src/executors/invariant/mod.rs (call in end_run() when new_coverage)

• Collect data from reverted calls (filtered) — Currently only successful calls feed the dictionary (if !call_result.reverted). Reverted calls contain boundary values that help explore nearby valid inputs. Needs filtering to avoid noise from panics — only collect from "soft" reverts (require failures), not assertion panics.

  • Location: crates/evm/evm/src/executors/invariant/mod.rs line ~530

---

Phase 3: Corpus Selection & Mutation

The corpus-guided mutation engine has structural weaknesses that limit exploration depth.

• Weighted corpus selection — Corpus entries are selected uniformly at random. Favored entries (those producing new coverage >30% of the time) should be selected 3-5× more often, not just protected from eviction. Simple weighted sampling.

  • Location: crates/evm/evm/src/executors/corpus.rs (new_inputs(), new_input())

• Add Insert/Delete/Swap mutations — The current mutation set (Splice, Repeat, Interleave, Prefix, Suffix, Abi) is missing critical structural mutations:

  • Insert: add a new random call at a random position
  • Delete/Trim: remove calls to find shorter interesting sequences
  • Swap: reorder two calls in a sequence
  • Location: crates/evm/evm/src/executors/corpus.rs (MutationType enum, new_inputs())

• Adaptive mutation scheduling — All 6 mutation types have equal weight. Track which mutation types produce new coverage and bias toward productive ones.

  • Location: crates/evm/evm/src/executors/corpus.rs (mutation_generator)

• Adaptive random injection rate — Hardcoded 10% chance of replacing a corpus call with a fully random one. Should start higher when corpus is small, decay as quality improves.

  • Location: crates/evm/evm/src/executors/corpus.rs (generate_next_input())

---

Phase 4: Coverage Signal Quality

The coverage map tracks EVM JUMPI edge pairs but misses state-dependent behavior.

• State-sensitive coverage — Hash key storage slot deltas into the coverage map so different state transitions count as different coverage. Highest-impact long-term improvement.

  • Location: crates/evm/evm/src/executors/corpus.rs, crates/evm/evm/src/executors/mod.rs

• Function-pair coverage tracking — Track which pairs/triples of Solidity functions have been called in sequence as an additional coverage dimension.

---

Validation

Use SCFuzzBench for end-to-end validation:

• Target: aave-v4-scfuzzbench (v0.5.6-recon)
• Metric: unique broken invariants at 1h and 24h timeouts
• Baseline: Echidna finds 10+; current Foundry finds 3
• Each phase should be benchmarked independently with multiple seeds

cargo build --release --bin forge
FOUNDRY_INVARIANT_TIMEOUT=300 forge test --mc CryticToFoundry -vv --fuzz-seed 42

[grandizzy]

@0xalpharush @aviggiano @rappie would you mind take a look at this plan, ty!

[rappie]

Additional issues:

#13285 & #13575

• Necessary for proper statistics during the benchmark run

#13177 & #12934

• AFAIK foundry does not fuzz msg.value yet. Essential for many Echidna/Medusa-style harnesses.

#12971

• Should be part of Phase 3, especially sharing the corpus between threads/workers.

For the validation I recommend using multiple different scaffolding architectures instead of running it on one single target.

• https://github.com/perimetersec/public-fuzzing-campaigns-list
• Suggestions:
  • Drips Network: link
  • Origin OUSD: link

[grandizzy]

Further architectural improvement tracked in #14448: track handler assertion failures independently during continuous runs (keep running after all invariants break, deduplicate broken handler assertions).

[aviggiano]

Yes using more than one target would be ideal. Other than that, looks like a pretty solid plan!

[0xalpharush]

Fwiw, much of the below reiterates #8076 (comment)

Re Phase 2: I think we should make the dictionaries per-worker instead of global. Bounding definitely makes sense but I do wonder if we should cycle somehow... Also, I suspect we may miss logs in nested call frames at the moment here but need to double check. Wonder if some of this could be pulled into an revm-inspector 🤔

Re Phase 3: I explored a refactor, additional mutations, and weighted sampling here 0xalpharush#7. As I said here, getting the compatibility sorted will enable more experimentation on this front. For weighted corpus selection aka seed scheduling, I do think the approach here is better, but I haven't evaluated it.

Re Phase 4: I think the state snapshots are promising and worth exploring but I think we should be able get to coverage-parity on scfuzzbench without it.

Should be part of Phase 3, especially sharing the corpus between threads/workers.
From: @rappie

I go back and forth on this and it probably requires an experiment. I think the most promising use of parallel workers is some sort of "swarm" mode which perturbs all the configs like lowering the sequence length, changing dict. freq., changing mutation weight or altogether disabling some mutations (this is more straightforward than adaptive scheduling).

As far as adaptive mechanisms go, I think we could experiment with one or both of the following:

• Sync when no new coverage for X minutes/N runs
• Drop X percentage of the corpus from workers when no new coverage for X minutes/N runs ala https://dl.acm.org/doi/10.1145/3712186

The intuition is that syncing too frequently can turn one worker's local minima into a global minima. IOW, reducing novelty

In addition to these phases, I think adding collision-free edges (configurable), gathering a seed corpus (#10875), and addressing #12044 are promising. I'd place these prior to phase 4 just bc they're "simpler" to implement.

[0xMars42]

Opened #15088 for the Insert/Delete/Swap structural mutations from the Phase 3 list. They're extracted into small unit-tested helpers, mirroring the existing cmp_mutate. I can run a coverage delta from SCFuzzBench or tune the weighting if that'd help before merging.