tools/syz-cover: PCs returned by kcov do not have matching coverage callbacks

[6eanut]

I am encountering an issue while generating an HTML coverage report using
syz-cover. I would appreciate some guidance.

My Workflow:
I downloaded the raw coverage (approx. 20k addresses) via:
wget http://localhost:57701/rawcover
I ran syz-cover to generate the HTML report:
./bin/syz-cover -config test.config -debug rawcover

The Error:
The tool produces the following output: "202286 out of 202680
PCs returned by kcov do not have matching coverage callbacks. Check the
discoverModules() code. Use ?force=1 to disable this message."

My Investigation:
I manually verified the addresses in the 'rawcover' file using addr2line
with my vmlinux binary. Every address I tested was successfully resolved
to the correct source file and line number.

Question:
Since addr2line can resolve these addresses, why does syz-cover fail to
match them to coverage callbacks?

Any insights or debugging tips would be greatly appreciated.

[a-nogikh]

If you pass debug=1 when querying /rawcover (or /cover, I assume the problem affects both), it should also print the exact PCs that it failed to match.

[6eanut]

@a-nogikh
Thanks for the suggestion.

I queried /cover with debug=1 as you mentioned and was able to obtain the exact PCs that failed to match. I then took a subset of those PCs and resolved them locally using addr2line against my vmlinux, and all of them were successfully mapped to valid source files and line numbers.

Do you have any insight into why syz-cover fails to match them to coverage callbacks?

Thanks again for your help.

[a-nogikh]

Could you somehow share the vmlinux file and the PCs syzkaller failed to match? What was the arch and the compiler?

Internally, it's not just about addr2line (which will return something for almost all valid PCs), but rather about whether there's a KCOV coverage point at the particular address (and there's just ~1 such point per basic block). Also, not just all KCOV points, but afaik it excludes the comparison operand collection-related ones.

FWIW these are collected here:

syzkaller/pkg/cover/backend/dwarf.go

Lines 127 to 139 in 15f6fd0

	var data []byte
	var coverPoints [2][]uint64
	if _, ok := arches[target.Arch]; !ok {
	coverPoints, err = objdump(target, module)
	} else if module.Name == "" {
	data, err = params.readTextData(module)
	if err != nil {
	return nil, err
	}
	coverPoints, err = readCoverPoints(target, info, data)
	} else {
	coverPoints, err = params.readModuleCoverPoints(target, module, info)
	}

[6eanut]

@a-nogikh
Thanks for the detailed explanation.

One additional thought from my side: my understanding is that the PCs returned by /rawcover should already correspond to KCOV coverage points (i.e. addresses where KCOV instrumentation actually reports execution). Given that, I was a bit surprised that such a large fraction of them do not match the cover points discovered by syz-cover.

This makes me wonder whether there could be a mismatch between how KCOV instrumentation points are generated on riscv64 (or with this toolchain) and how syz-cover discovers expected cover points via DWARF/objdump.

For some more context: I’m running riscv64 QEMU on an x86 host.
The kernel was built with the following compiler:

$ riscv64-unknown-linux-gnu-gcc --version
riscv64-unknown-linux-gnu-gcc (g1b306039a) 15.1.0
Copyright (C) 2025 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I’ve uploaded the vmlinux file, the PCs that syzkaller failed to match, and the compiler to the following link:

https://github.com/6eanut/temp/releases/tag/syzkaller_issues_6551

Thanks again for taking a look.

[a-nogikh]

I've tried to build a coverage report from your vmlinux and PCs, and it worked fine on my side.

Probably due to some env differences?

$ riscv64-linux-gnu-objdump --version
GNU objdump (GNU Binutils for Debian) 2.45

You can try to debug what's happening inside this function:

syzkaller/pkg/cover/backend/dwarf.go

Line 710 in f7cfc62

func objdump(target *targets.Target, mod *vminfo.KernelModule) ([2][]uint64, error) {

[6eanut]

Thanks a lot for taking the time to test this and for verifying it on your side.

Based on further comparison on my end, it does seem more likely that this is caused by some local environment difference rather than an issue in syz-cover itself. My setup uses riscv64-linux-gnu-objdump 2.38; interestingly, the report fails on one machine but works better on another machine with the same objdump version, so the exact environmental factor is still unclear.

I’ll continue investigating potential differences in the toolchain / binutils / DWARF-related environment.
Thanks again for your help and insights!

[6eanut]

@a-nogikh

I've tried to build a coverage report from your vmlinux and PCs, and it worked fine on my side.

Hi, I'd like to ask if this means that all PCs returned by kcov can match coverage callbacks? Because when I generated the report on another machine, the number of unmatched PCS decreased (compared to the initial 202286 out of 202680), but still listened to a large amount.
failed to generate coverage profile: 62330 out of 116776 PCs returned by kcov do not have matching coverage callbacks. Check the discoverModules() code. Use ? force=1 to disable this message.

[a-nogikh]

All PCs returned by kcov should match the coverage callbacks because they are only recorded if those callbacks are hit.

[6eanut]

Could you share the exact command you used to generate the coverage report？

[a-nogikh]

I used tools/syz-cover, but I patched it a bit to accept hex encoding of pc.txt that you shared in https://github.com/6eanut/temp/releases/tag/syzkaller_issues_6551:

diff --git a/tools/syz-cover/syz-cover.go b/tools/syz-cover/syz-cover.go
index 4f78ad29b..cced9248a 100644
--- a/tools/syz-cover/syz-cover.go
+++ b/tools/syz-cover/syz-cover.go
@@ -208,7 +208,7 @@ func readPCs(files []string) ([]uint64, error) {
                        if line == "" {
                                continue
                        }
-                       pc, err := strconv.ParseUint(line, 0, 64)
+                       pc, err := strconv.ParseUint(line, 16, 64)
                        if err != nil {
                                return nil, err
                        }

(it's not necessary if you take the PC list from /rawcover, then they are already prefixed with 0x)

Then I've invoked:

$ ./tools/syz-env make cover
$ ./bin/syz-cover -config riscv.cfg -debug /tmp/pcs.txt 

[6eanut]

Thank you very much. I agree with what you said earlier that it was caused by environmental differences. Later, I will try to find out the specific reasons.

[6eanut]

I think this issue can be closed now.

Based on the previous discussion, it seems reasonable to conclude that this is caused by some local environment differences on my side rather than an issue in syzkaller itself.

Thanks again for the help.