pkg/manager: list broken seeds + fix one

[a-nogikh]

Fixes #5395.

[dvyukov]

We have a test somewhere that checks parsing of all seeds. How did it happen that it wasn't triggered?
Can we fix the test instead to catch this earlier? There is really no point in committing broken seeds in the first place, so none should reach syz-manager (that's why it did not have proper diagnostics for this case).

[a-nogikh]

I think printing the failed tests is useful anyway -- it e.g. gives more some human-friendly feedback during development and local syz-manager runs, and this can often happen well before one gets to run all presubmit tests.

There is really no point in committing broken seeds in the first place

I agree.

We do check that we can parse the seed programs (and AFAIK in multiple places), but here the seed was violating the expectations of pkg/manager on the maximum number of calls in it.

One of the tests (the most relevant one?) is here:

syzkaller/pkg/runtest/run_test.go

Lines 543 to 551 in 084d817

	func TestParsing(t *testing.T) {
	t.Parallel()
	// Test only one target in race mode (we have gazillion of auto-generated Linux test).
	raceTarget := targets.Get(targets.TestOS, targets.TestArch64)
	for OS, arches := range targets.List {
	if testutil.RaceEnabled && OS != raceTarget.OS {
	continue
	}
	dir := filepath.Join("..", "..", "sys", OS, "test")

But it relies on the checks done in pkg/runtest itself:

syzkaller/pkg/runtest/run.go

Lines 313 to 316 in 084d817

	p, err := target.Deserialize(data, prog.Strict)
	if err != nil {
	return nil, nil, nil, fmt.Errorf("failed to deserialize %v: %w", filename, err)
	}

We can try to reuse manager.LoadProg() here to rely on the same logic that will later be used by syz-manager.

[dvyukov]

Extending syz-manager reporting is fine (I guess it may be useful for some downstream users with custom seeds).
But the most important here is to fix the test to do what prod does. I guess we need to move the loading function from syz-manager to somewhere else and reuse it in the test.

[a-nogikh]

Added a commit that reuses the same seed validation logic. PTAL.

[peiwithhao]

So how to fix it?I found the fuzz program has been stopped because of the failed

[a-nogikh]

The fuzzing cannot have been stopped because of these
a) it's just seeds (syzkaller can run without them)
b) it's 13 out of 100s of seed programs.

@ramosian-glider It looks like we need to prevent syz-manager from trying to parse arm64 seeds on non-arm64 machines.

There's an annotation for that

syzkaller/sys/linux/test/nvme

Line 1 in b499ea6

# requires: arch=amd64

But I'm not sure it's supported by syz-manager now, probably so far only by pkg/runtest.

[ramosian-glider]

We also have annotations like this: # {Threaded:true Repeat:true RepeatTimes:0 Procs:1 Slowdown:10 Sandbox: SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:false KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:false IEEE802154:false Sysctl:false Swap:false UseTmpDir:false HandleSegv:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}} in reproducers generated by syzkaller, perhaps these should be unified?

[a-nogikh]

perhaps these should be unified?

Unified with what exactly?

[ramosian-glider]

So how to fix it?I found the fuzz program has been stopped because of the failed

This should be fixed in #5560