pkg/aflow/tool: bug reproduction/patch testing tool

[dvyukov]

We need a tool that would initially confirm that we can reproduce the given bug on the latest tree we use for patch generation; and a tool that will later confirm the bug is fixed by the patch. These things look closely related functionality-wise, so maybe it should be a single tool, or 2 separate tools, but that reuse the same code.

For now we need just a basic testing (kernel builds/boots, reproducer does not trigger a kernel bug anymore), this issues does not include complex exhaustive verification.

In order to be able to run kernels we need to somehow instruct tools how to create VMs. We probably need to accept some subset of syz-manager config (type/vm) as part of workflow inputs, so that tools have access to the info.

[QGrain]

I used to write a tool for the automatic syzkaller-generated reproducer verification (the oracle judge is panic or not), which uses the api of syzqemuctl and is out of the tree of syzkaller.

Maybe I could migrate it under the subset of syz-manager.

[dvyukov]

Hi @QGrain,

If I understand correctly "reproducer verification (the oracle judge is panic or not)", we already have such tool here:
https://github.com/google/syzkaller/blob/master/pkg/aflow/action/crash/reproduce.go

Though, it probably can be improved (use syzkaller reproducer, run reproducer several times, may be do some flake filtering).

We also need a more comprehensive patch testing step, that would do much more than just running the reproducer.

Maybe I could migrate it under the subset of syz-manager.

Note: pkg/aflow has nothing to do with syz-manager.

[QGrain]

Hi @dvyukov ,

If I understand correctly "reproducer verification (the oracle judge is panic or not)", we already have such tool here:
https://github.com/google/syzkaller/blob/master/pkg/aflow/action/crash/reproduce.go

Thanks for referring. I also noticed about the recently added module pkg/aflow, which serves for syz-agent. Yes, you are right. So, has this issue already been solved by those commits?

We also need a more comprehensive patch testing step, that would do much more than just running the reproducer.

Such as directed fuzzing or ... root cause analysis?

[dvyukov]

Thanks for referring. I also noticed about the recently added module pkg/aflow, which serves for syz-agent. Yes, you are right. So, has this issue already been solved by those commits?

The existing repro probably can be improved (use syzkaller reproducer, run reproducer several times, may be do some flake filtering). We also need a more comprehensive patch testing step, that would do much more than just running the reproducer.

Such as directed fuzzing or ... root cause analysis?

Directed fuzzing, checkpatch, static analysis, executing unit tests, and whatever a kernel maintainer would do to verify a patch.

[QGrain]

The existing repro probably can be improved (use syzkaller reproducer, run reproducer several times, may be do some flake filtering). We also need a more comprehensive patch testing step, that would do much more than just running the reproducer.

Understand. May I join and contribute to these modules for agentic syzkaller? As I am also doing something similar and interesting as well :)

[dvyukov]

Sure. Help is welcome.
But please discuss what you want to work on first on the mailing list so that (1) we don't duplicate work, (2) agree on high level decisions first.
Here are 3 separable work-items:
https://github.com/google/syzkaller/issues?q=state%3Aopen%20label%3A%22AI%20patching%22
But there are more of them: we need a debugger tool, improve prompts, tune temperature, etc.

[QGrain]

But please discuss what you want to work on first on the mailing list so that (1) we don't duplicate work, (2) agree on high level decisions first.

I see.