prog: TestSerializeDeserializeRandom failure

[a-nogikh]

ok  	github.com/google/syzkaller/pkg/vminfo	15.323s
--- FAIL: TestSerializeDeserializeRandom (6.55s)
    testutil.go:35: seed=1754053438880905146
    --- FAIL: TestSerializeDeserializeRandom/linux/riscv64 (8.55s)
        encoding_test.go:437: encodingexec: too large program (4732298/4194304)
FAIL
FAIL	github.com/google/syzkaller/prog	37.832s

[a-nogikh]

It's some insanely big ioctl$COMEDI_INSNLIST(r3, 0x8010640b, &(0x7f0000230f40)={0xf, &(0x7f0000230cc0)=[...], 0x5}]}) call.

[a-nogikh]

One more case

--- FAIL: TestSerializeForExecRandom (28.55s)
    testutil.go:35: seed=1756810767636672562
    encodingexec_test.go:27: failed to serialize: encodingexec: too large program (4374162/4194304)
FAIL
FAIL	github.com/google/syzkaller/prog	57.385s

[a-nogikh]

One more 267f56c

--- FAIL: TestSerializeDeserializeRandom (5.18s)
    testutil.go:35: seed=1759363128359195259
    --- FAIL: TestSerializeDeserializeRandom/linux/arm64 (7.56s)
        encoding_test.go:437: encodingexec: too large program (4577536/4194304)
FAIL
FAIL	github.com/google/syzkaller/prog	83.567s

[eaibmz]

One on s390x

ok  	github.com/google/syzkaller/pkg/vminfo	10.609s
--- FAIL: TestSerializeForExecRandom (7.00s)
    testutil.go:35: seed=1762247116897083520
    encodingexec_test.go:27: failed to serialize: encodingexec: too large program (4832751/4194304)

[a-nogikh]

@pimyn-girgis could you please take a look?

The bug should be reproducible via sth like

./tools/syz-env SYZ_SEED=1762247116897083520 go test ./prog -run TestSerializeForExecRandom 

I suppose the reason is that for some of the syscalls we generate an array that is inadequately large considering the length of each array item.

See the code around:

syzkaller/prog/rand.go

Line 882 in 686bf65

count = r.randArrayLen()

But that needs to be confirmed.