Thesis available?

[gustavo-grieco]

Hi!,

I was wondering if the thesis is available to read, there are a few interesting concepts here and there 😄

[forkforkdog]

Same, would like to read the thesis!

[mlbyvn]

@rappie @gustavo-grieco @forkforkdog Hi guys, appreciate your interest!
Given the limited time, I've managed to construct a relatively small benchmark, so the results should be interpreted with caution. Additionaly, in light of the recent paper named The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity by Shojaee et al it turns out that reasoning models like o3 may fail to perform on more complex/obfuscated protocols. It would be fun to test it further, so if you have any ideas on how the benchmark could be extended, feel free to share.

B_Sc_Thesis__Aleksandr_Rybin(1).pdf

[ljz3]

I’m wondering if it’s possible for an LLM to generate additional seed or corpus files on the fly during fuzzing. With access to coverage files, the LLM could create more targeted seeds, leading to deeper exploration. In turn, the fuzzer could benefit from more meaningful mutations based on those inputs.

Additionally, this setup could also improve fuzzing efficiency via the GPU and not strictly reliant on CPU.

[gustavo-grieco]

Thanks a lot @mlbyvn. While you are correct about the small scale of the benchmark, it is indeed a good example of preliminary results to follow (if you need a bit of help on how to go into that direction, please feel free to contact me, I previously helped other msc students in their Echidna-related thesis).
Regarding the Apple research paper, it is a very controversial topic. Some researcher found that some of the problems in the benchmark their proposed are flawed and have no solution at all 😱.

[mlbyvn]

@ljz3 My initial intuition was somewhat similar, I tried feeding the entire corpus (or also parts of it) to the model after a small fuzzing campaign. There are a couple challenges:

1. Maximal token input limit
   In the prompt I included the contract in scope (only the relevant parts, if possible) and the test setup. When additionaly including the whole corpus, I ran into the input limit issue. If only a part of corpus is included, there is a follow-up question: how many separate TXes should that part include to make any meaningful impact?
2. Formatting
   If any of the seeds are formatted incorectly, Echidna seems to silently skip the whole corpus. I havent found a way to get an OpenAI model to generate a properly wrapped seed corpus - there is always some degree of corruption (e.g., missed brackets). Perhaps a custom model would perform better.

Totally agree on leveraging the GPU! Thanks for your thoughts.

[gustavo-grieco]

Some answers to you:

1. For selecting which parts of the code to fuzz are more relevant to show the LLM, check my other tool quimera. It essentially uses some basic static analysis from Slither to find the target contract, show its source code and include all their dependencies. The LLM can also request more source as it is needed, instead of feeding it with all. I don't think the corpus needs to be included in the context, instead you can use the coverage report, so you can about reaching new coverage (e.g. falsified assertions or deeper parts of the code).

2. Your LLM does not need to replicate the Echidna corpus format. It is easier to write a small tool that takes a string that specifies a list transaction (e.g. f(1,2,3)) and then dumps a valid JSON to use as corpus. Take a look to this tool for inspiration.

[rappie]

For 1. you could try lowering the seqLen setting to generate shorter sequences per corpus file and perhaps only provide parts of the corpus to the LLM.

Another helpful tool might be fuzz-utils for modifying or cleaning the corpus:
crytic/fuzz-utils#54