Hi @EVMBench team,
First, thank you for releasing EVMBench; it's a great contribution to the space, and the harness design is really well thought out.
I wanted to suggest a potential dataset extension that might broaden EVMBench's evaluation scope, in case it's useful for future work.
FORGE-Curated (part of ICSE'26 paper) is an EVM smart contract vulnerability dataset derived from audit reports published by 11 top-tier audit teams (we are actively expanding coverage — currently including teams such as ChainSecurity, Cantina, Trail of Bits, Zellic, Zenith and others) between December 2024 and February 2026.
Some numbers that might be relevant:
- 2,469 vulnerability findings across 252 projects and 29,221 Solidity files
- Findings include CWE hierarchy classification (multi-level), severity, and code location (
file::function#line) alongside full source code
- Pre-built Vulnerability-File Pairs (VFPs) for easy loading — 627 total, 307 at Medium severity and above
- Temporal freshness: all reports are from Dec 2024–Feb 2026, which substantially reduces data leakage concerns for models trained before 2025
Why it might complement EVMBench:
EVMBench's 120 curated challenges are excellent for precise detect/exploit/patch evaluation. FORGE-Curated could potentially serve a different role — evaluating recall (whether an agent catches real-world bugs in project-level scope) and false positive rate (whether it over-reports on large, realistic codebases) at a much larger scale, using vulnerabilities confirmed by professional auditors rather than synthetically constructed ones.
Of course, the two practices serve different purposes, and there are tradeoffs: FORGE-Curated is not primarily focused on exploit scripts (though some findings do include or reference PoC code from the original reports), so it may not fit EVMBench's exploit/patch tracks as directly. That said, it could be a good fit for extending the detection evaluation dimension.
Happy to answer any questions or discuss further. Thanks again for the great work!
Hi @EVMBench team,
First, thank you for releasing EVMBench; it's a great contribution to the space, and the harness design is really well thought out.
I wanted to suggest a potential dataset extension that might broaden EVMBench's evaluation scope, in case it's useful for future work.
FORGE-Curated (part of ICSE'26 paper) is an EVM smart contract vulnerability dataset derived from audit reports published by 11 top-tier audit teams (we are actively expanding coverage — currently including teams such as ChainSecurity, Cantina, Trail of Bits, Zellic, Zenith and others) between December 2024 and February 2026.
Some numbers that might be relevant:
file::function#line) alongside full source codeWhy it might complement EVMBench:
EVMBench's 120 curated challenges are excellent for precise detect/exploit/patch evaluation. FORGE-Curated could potentially serve a different role — evaluating recall (whether an agent catches real-world bugs in project-level scope) and false positive rate (whether it over-reports on large, realistic codebases) at a much larger scale, using vulnerabilities confirmed by professional auditors rather than synthetically constructed ones.
Of course, the two practices serve different purposes, and there are tradeoffs: FORGE-Curated is not primarily focused on exploit scripts (though some findings do include or reference PoC code from the original reports), so it may not fit EVMBench's exploit/patch tracks as directly. That said, it could be a good fit for extending the detection evaluation dimension.
Happy to answer any questions or discuss further. Thanks again for the great work!