syzkaller: fake coverage due to rt_sigaction

[a-nogikh]

https://syzkaller.appspot.com/upstream/graph/fuzzing?Instances=ci-snapshot-upstream-root&Metrics=MaxPCs&Months=1

+200K fake PCs at the moment

There's the following SYZFAIL: too much cover report that hints at the way the coverage is generated.

program:
rt_sigaction(0xd, &(0x7f0000000040)={&(0x7f0000000380)="c441f96ec866400fe2dec441a16dfb46c7045300101000f00fc01e66410f6f150400000052c4637bf02640f5c4b93c10", 0xdc000006, 0x0, {[0x5]}}, 0x0, 0x8, &(0x7f0000000000))
r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
sendmmsg$inet6(r0, &(0x7f0000003c00)=[{{0x0, 0x0, 0x0}}], 0x1, 0x63)
r1 = syz_init_net_socket$nl_rdma(0x10, 0x3, 0x10)
sendmsg$netlink(r1, &(0x7f00000001c0)={0x0, 0x0, &(0x7f0000000000)=[{&(0x7f0000005d00)=ANY=[@ANYBLOB="140100002d00010000000000fcdbdf250401f280080018000bac0f"], 0x114}], 0x1, 0x0, 0x0, 0x1}, 0x0)
mkdir(&(0x7f0000000080)='./file0\x00', 0x10e)

[   81.056528][   T49] Bluetooth: hci0: command tx timeout
[   81.060278][ T1311] ieee802154 phy0 wpan0: encryption failed: -22
[   81.062861][ T1311] ieee802154 phy1 wpan1: encryption failed: -22
[   81.214687][ T5316] SYZFAIL: too much cover
[   81.219992][ T5316] cov=886
[   81.220118][ T5316]  (errno 11: Resource temporarily unavailable)
[   81.316123][ T5301] SYZFAIL: child failed
[   81.323751][ T5301]  (errno 0: Success)
[   83.469609][ T5288] loop exited with status 67

I tried to run that program with syz-crush on a non-shapshot based instance with

--- a/pkg/instance/instance.go
+++ b/pkg/instance/instance.go
@@ -488,7 +488,7 @@ func ExecprogCmd(execprog, executor, OS, arch, vmType string, opts csource.Optio
                })
        }
        return fmt.Sprintf("%v -executor=%v -arch=%v%v -sandbox=%v"+
-               " -procs=%v -repeat=%v -threaded=%v -collide=%v -cover=0%v %v",
+               " -procs=%v -repeat=%v -threaded=%v -collide=%v -cover=1%v %v",
                execprog, executor, arch, osArg, opts.Sandbox,
                opts.Procs, repeatCount, opts.Threaded, opts.Collide,
                optionalArg, progFile)

but it didn't reproduce.

[a-nogikh]

I still haven't been able to reproduce SYZFAIL: too much cover using that program locally.

The only really suspicious part there is rt_sigaction:

rt_sigaction(0xd, &(0x7f0000000040)={&(0x7f0000000380)="c441f96ec866400fe2dec441a16dfb46c7045300101000f00fc01e66410f6f150400000052c4637bf02640f5c4b93c10", 0xdc000006, 0x0, {[0x5]}}, 0x0, 0x8, &(0x7f0000000000))

corresponds to

rt_sigaction(SIGPIPE, {sa_handler=0x200000000380, sa_mask=[HUP QUIT], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_NODEFER|SA_RESETHAND|SA_SIGINFO|SA_NOCLDWAIT, sa_restorer=NULL}, NULL, 8)

The code at 0x200000000380 is as follows

0x200000000380: vmovq   xmm9, r8
0x200000000385: psrad   xmm3, xmm6
0x20000000038a: vpunpckhqdq     xmm15, xmm11, xmm11
0x20000000038f: mov     dword ptr [rbx + r10*2], 0x101000
0x200000000397: lock xadd       byte ptr [rsi], bl
0x20000000039b: movdqa  xmm2, xmmword ptr [rip + 4]
0x2000000003a4: push    rdx
0x2000000003a5: rorx    r12d, dword ptr [rsi], 0x40
0x2000000003ab: cmc

It doesn't seem like something that could rewrite significant parts of memory.

[ramosian-glider]

Perhaps we can add some simple check for early detection of fake coverage?

E.g. if there are no modules, and the difference between the min and the max PC values is greater than the size of the .text section, we can report an error.

[ramosian-glider]

The above program indeed looks relevant.

The corpus on that builder that @a-nogikh shared with me contains 3690 syscalls, and rt_sigaction, sendmmsg$inet6, and socket$inet6_tcp are the most common among those (so most likely they are responsible for this extra coverage):

...
   6704 socket$nl_route
   7038 socket
   7695 socket$nl_generic
   8436 openat
  35398 rt_sigaction
  35924 sendmmsg$inet6
  37497 socket$inet6_tcp

Out of 35398 rt_sigaction calls, 35175 contained non-trivial code blobs mutated from c441f96ec866400fe2dec441a16dfb46c7045300101000f00fc01e66410f6f150400000052c46b7bf036c0f5c42279243a3c10:

...
    382 c441f96ec8c441a16dfb46c70453001010100000410f6f15040000000f1bcbc46b7bf036c0f5c43a3c10
    642 c441f96ec866400fe2dec441a16dfb46c70453001010a800000fc01e66410f6f150400000052c46b7bf036c0f5c42279243a3c10
    994 c441f96ec866400fe2dec441a16dfb46c70453004d101000000fc01e52c4637bf02640f5c4b93e6566420f383375a1
   5645 66400fe2dec441a16dfb46c704531d106a1800000fc01e66410f6f150400000052c66b7bf036c0f5c4227924c462b8f34d0e
   9548 c441f96ec866400fe2dec441a16dfb46c7045300101000f00fc01e66410f6f150400000052c4637bf02640f5c4b93c10
  16774 c441f96ec866400fe2dec441a16dfb46c7045300101000f00fc01e66410f6f150400000052c46b7bf036c0f5c42279243a3c10

There were 89 distinct blobs.

[a-nogikh]

When I run the program in #6015 (comment) under -debug:

[   71.921285][ T5306] [15246ms] exec opts: reqid=0 type=0 procid=0 threaded=1 cover=1 comps=0 dedup=1 signal=1  sandbox=1/0/0/0 timeouts=50/5000/1 kernel_64_bit=1
[   71.922864][ T5307] #0 [15247ms] -> rt_sigaction(
[   71.922886][ T5307] 0xd
[   71.923322][ T5307] , 
[   71.923566][ T5307] 0x200000000040
[   71.923802][ T5307] , 
[   71.924117][ T5307] 0x0
[   71.924352][ T5307] , 
[   71.924640][ T5307] 0x8
[   71.924876][ T5307] , 
[   71.925118][ T5307] 0x200000000000
[   71.925353][ T5307] )
[   71.925894][ T5307] #0 [15250ms] <- rt_sigaction=0x0
[   71.925914][ T5307]  cover=19
[   71.926364][ T5307] 
[   71.930890][ T5307] #0 [15255ms] -> socket$inet6_tcp(
[   71.930914][ T5307] 0xa
[   71.931386][ T5307] , 
[   71.931626][ T5307] 0x1
[   71.931860][ T5307] , 
[   71.932100][ T5307] 0x0
[   71.932334][ T5307] )
[   71.932957][ T5307] #0 [15257ms] <- socket$inet6_tcp=0x3
[   71.932977][ T5307]  cover=413
[   71.933439][ T5307] 
[   71.934293][ T5307] #0 [15259ms] -> sendmmsg$inet6(
[   71.934316][ T5307] 0x3
[   71.934809][ T5307] , 
[   71.935049][ T5307] 0x200000003c00
[   71.935295][ T5307] , 
[   71.935610][ T5307] 0x1
[   71.935843][ T5307] , 
[   71.936082][ T5307] 0x62
[   71.936315][ T5307] )
[   71.937201][ T5307] SIGSEGV on (nil), skipping
[   71.937642][ T5307] SIGSEGV on 0x7f3f965fe000, exiting
[   71.965842][ T5291] SnapshotDone
[   71.966858][ T5291] handle completion: completed=2 output_size=14680048
[   71.970189][ T5291] changing stapshot state Execute -> Executed

so it actually doesn't get past the first two calls, at least on my local setup.

[a-nogikh]

After another debugging session with @ramosian-glider we found out why it was not reproducing locally: we don't have the memory protection keys CPU feature on the ci-snapshot VM on Google Cloud, but locally we do.

The SYZFAIL: too much cover crash becomes locally reproducible after appending pku=off to -cpu in qemu command line settings.

Cc @dvyukov

[a-nogikh]

FTR an example of a program that triggers fake coverage:

rt_sigaction(0xd, &(0x7f0000000040)={&(0x7f0000000380)="c441f96ec866400fe2dec441a16dfb46c7045300101000f00fc01e66410f6f150400000052c4637bf02640f5c4b93c10", 0xdc000006, 0x0, {[0x5]}}, 0x0, 0x8, &(0x7f0000000000))
r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
sendmmsg$inet6(r0, &(0x7f0000003c00)=[{{0x0, 0x0, 0x0}}], 0x1, 0x66)
ioctl$ifreq_SIOCGIFINDEX_batadv_mesh(0xffffffffffffffff, 0x8933, 0x0)
r1 = openat$sw_sync(0xffffffffffffff9c, &(0x7f0000000000), 0x0, 0x0)
ioctl$SW_SYNC_IOC_CREATE_FENCE(r1, 0xc0285700, &(0x7f00000002c0)={0x3, "421ae3753785259249154c944122ad063ff47d3bd7a8a45d6bb4c78a3ab4c981"})
ioctl$SYNC_IOC_MERGE(0xffffffffffffffff, 0xc0303e03, 0x0)
ioctl$SW_SYNC_IOC_INC(r1, 0x40045701, &(0x7f0000000180)=0x8)

Execution trace under -debug with pku=off:

[   71.117339][ T5318] [14359ms] exec opts: reqid=0 type=0 procid=0 threaded=1 cover=1 comps=0 dedup=1 signal=1  sandbox=1/0/0/0 timeouts=50/5000/1 kernel_64_bit=1
[   71.129627][ T5319] #0 [14372ms] -> rt_sigaction(
[   71.129661][ T5319] 0xd
[   71.130197][ T5319] , 
[   71.132104][ T5319] 0x200000000040
[   71.132382][ T5319] , 
[   71.132717][ T5319] 0x0
[   71.132962][ T5319] , 
[   71.133214][ T5319] 0x8
[   71.133456][ T5319] , 
[   71.133706][ T5319] 0x200000000000
[   71.133946][ T5319] )
[   71.134505][ T5319] #0 [14377ms] <- rt_sigaction=0x0
[   71.134527][ T5319]  cover=19
[   71.135002][ T5319] 
[   71.135842][ T5319] #0 [14378ms] -> socket$inet6_tcp(
[   71.135863][ T5319] 0xa
[   71.136324][ T5319] , 
[   71.136569][ T5319] 0x1
[   71.136810][ T5319] , 
[   71.137054][ T5319] 0x0
[   71.137293][ T5319] )
[   71.137954][ T5319] #0 [14380ms] <- socket$inet6_tcp=0x3
[   71.137974][ T5319]  cover=471
[   71.138447][ T5319] 
[   71.142824][ T5319] #0 [14385ms] -> sendmmsg$inet6(
[   71.142848][ T5319] 0x3
[   71.143309][ T5319] , 
[   71.143555][ T5319] 0x200000003c00
[   71.143803][ T5319] , 
[   71.144125][ T5319] 0x1
[   71.144380][ T5319] , 
[   71.144625][ T5319] 0x66
[   71.144863][ T5319] )
[   71.145728][ T5319] SIGSEGV on (nil), skipping
[   71.146144][ T5319] #0 [14388ms] <- sendmmsg$inet6=0xffffffffffffffff
[   71.146164][ T5319]  errno=14
[   71.146721][ T5319]  cover=3731
[   71.147010][ T5319] 
[   71.153235][ T5319] #0 [14395ms] -> ioctl$ifreq_SIOCGIFINDEX_batadv_mesh(
[   71.153259][ T5319] 0xffffffffffffffff
[   71.153860][ T5319] , 
[   71.154209][ T5319] 0x8933
[   71.154446][ T5319] , 
[   71.154727][ T5319] 0x0
[   71.154965][ T5319] )
[   71.155422][ T5319] #0 [14398ms] <- ioctl$ifreq_SIOCGIFINDEX_batadv_mesh=0xffffffffffffffff
[   71.155442][ T5319]  errno=9
[   71.156155][ T5319]  cover=11
[   71.156435][ T5319] 
[   71.156971][ T5319] #0 [14399ms] -> openat$sw_sync(
[   71.156990][ T5319] 0xffffffffffffff9c
[   71.157429][ T5319] , 
[   71.157775][ T5319] 0x200000000000
[   71.158012][ T5319] , 
[   71.158332][ T5319] 0x0
[   71.158569][ T5319] , 
[   71.158813][ T5319] 0x0
[   71.159049][ T5319] )
[   71.159724][ T5319] #0 [14402ms] <- openat$sw_sync=0x4
[   71.159756][ T5319]  cover=1642
[   71.160221][ T5319] 
[   71.163571][ T5319] #0 [14406ms] -> ioctl$SW_SYNC_IOC_CREATE_FENCE(
[   71.163594][ T5319] 0x4
[   71.164181][ T5319] , 
[   71.164449][ T5319] 0xc0285700
[   71.164689][ T5319] , 
[   71.164984][ T5319] 0x2000000002c0
[   71.165222][ T5319] )
[   71.165927][ T5319] #0 [14408ms] <- ioctl$SW_SYNC_IOC_CREATE_FENCE=0x0
[   71.165948][ T5319]  cover=1032
[   71.166521][ T5319] 
[   71.167112][ T5319] #0 [14409ms] -> ioctl$SYNC_IOC_MERGE(
[   71.167132][ T5319] 0xffffffffffffffff
[   71.167614][ T5319] , 
[   71.167971][ T5319] 0xc0303e03
[   71.168207][ T5319] , 
[   71.168501][ T5319] 0x0
[   71.168737][ T5319] )
[   71.169192][ T5319] #0 [14411ms] <- ioctl$SYNC_IOC_MERGE=0xffffffffffffffff
[   71.169211][ T5319]  errno=9
[   71.169805][ T5319]  cover=11
[   71.170085][ T5319] 
[   71.172674][ T5319] #0 [14415ms] -> ioctl$SW_SYNC_IOC_INC(
[   71.172696][ T5319] 0x4
[   71.173192][ T5319] , 
[   71.173437][ T5319] 0x40045701
[   71.173674][ T5319] , 
[   71.173967][ T5319] 0x200000000180
[   71.174204][ T5319] )
[   71.174798][ T5319] #0 [14417ms] <- ioctl$SW_SYNC_IOC_INC=0x0
[   71.174818][ T5319]  cover=836
[   71.175321][ T5319] 
[   71.203714][ T5303] SnapshotDone
[   71.204195][ T5303] handle completion: completed=8 output_size=14680048

Coverage from calls 1 and 2 is good, from 3rd to 8th - only garbage.

call 0 (errno=0x0): cover= ffffffff81641b22 ffffffff81641b67 ffffffff81679049 < ... >
call 1 (errno=0x0): cover= ffffffff8133832e ffffffff81679049 ffffffff816790e3 < ... >
call 2 (errno=0xe): cover= 100fff81279464 100fff81336410 100fff813366ac < ... >
call 3 (errno=0x9): cover= 100fff81679049 100fff816790e3 100fff823736bd < ... >
call 4 (errno=0x0): cover= 100fff8133832e 100fff81679049 100fff816790e3 < ... >
< ... >

[a-nogikh]

Setting -cpu host,pku=on on a machine that does not offer pku does not help:

qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:ECX.pku [bit 3]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:ECX.pku [bit 3]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX [bit 9]

The VM just boots without it.

So I checked ci (Intel Haswell), ci-qemu (Intel Ice Lake), ci-snapshot (Intel Cascade Lake). All of them have

$ cat /proc/cpuinfo | grep pku | wc -l
0

Is the feature supported anywhere on Cloud?

[a-nogikh]

https://cloud.google.com/compute/docs/machine-resource#machine_type_comparison

Checked C3 (Intel Sapphire Rapids), C4 (Intel Emerald Rapids) and N4 (Intel Emerald Rapids). None of them seem to have the pku feature. The base OS images all have CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y.

So I guess we'll have to rethink our pkey-based kcov data protection for qemu-based instances. Or not only qemu-based ones??

[a-nogikh]

With pku=off and -debug is set, SYZFAIL: too much cover reproduces also in the snapshot: false mode. The fake coverage doesn't.

[ramosian-glider]

Random idea: checksum kcov output on the kernel side so that the userspace can validate that it was corrupted during execution.

[a-nogikh]

Another crazy idea: intercept the writes to the mmapped kcov area on the kernel side and only permit the reset of the first word. Cons: we anyway get extra context switches for every syscall executed during fuzzing, so this may not be really faster than just doing mprotect.

[a-nogikh]

Syzkaller has come up with a version that works on all other instances as well.

The program (shared by @pimyn-girgis )

rt_sigaction(0xd, &(0x7f0000000180)={&(0x7f0000000000)="ca00d1c441ef196ec866400fe2de0fae4e0afaf2466ff00fc01ec422e10399c5c1202063df", 0x44000004, 0x0, {[0x2]}}, 0x0, 0x8, &(0x7f0000000300))
r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
r1 = openat$cgroup_ro(0xffffffffffffff9c, &(0x7f0000000080)='blkio.bfq.group_wait_time\x00', 0x275a, 0x0)
write$UHID_CREATE2(r1, &(0x7f0000000340)=ANY=[@ANYRES32, @ANYRES8, @ANYRES8=r1, @ANYRES64=r0], 0x118)
mmap(&(0x7f0000000000/0x3000)=nil, 0x3000, 0x88fd537e5c114b6e, 0x11, r1, 0x0)
r2 = socket$inet6_tcp(0xa, 0x1, 0x0)
ioctl$KVM_X86_SETUP_MCE(0xffffffffffffffff, 0x4008ae9c, &(0x7f0000000000)={0x1c, 0x526d630517582f27, 0x4})
sendmmsg$inet6(r2, &(0x7f0000003c00)=[{{0x0, 0x0, 0x0}}], 0x1, 0x4)
madvise(&(0x7f0000bdc000/0x4000)=nil, 0x86ac726dff2f4713, 0xa)
syz_clone(0x41000000, 0x0, 0x0, 0x0, 0x0, 0x0)