Missing file coverage for Kotlin files with package mismatch

[IgnatBeresnev]

Describe the bug

In Kotlin it is possible to declare a class in a package that does not match file's actual location. For instance,

// File path: src/kover/reproducer/PackageMismatchFile.kt

package kover.reproducer.some.inner.pckg // some.inner.pckg has been added

class PackageMismatchClass {}

It will then result in the following JaCoCo-like XML report:

<package name="org/jetbrains/kover/reproducer/some/inner/pckg">

    <class name="org/jetbrains/kover/reproducer/some/inner/pckg/PackageMismatchClass"
           sourcefilename="PackageMismatchFile.kt">
        ...
    </class>
    <sourcefile name="PackageMismatchFile.kt">
        ...
    </sourcefile>
    ...
</package>

As you can see, the path is taken from the package statement within the file, not file's location.

This leads to codecov ignoring such files altogether. You can observe it in this report - the file should be present in the linked package, but it's not. GitHub link to the package

To Reproduce

See kover-reproducer project and its README.

Expected behavior

Codecov should calculate coverage for files in which package does not match the actual location.

Additional context

The report was generated by Kover: a Gradle plugin that calculates code coverage for Kotlin projects. It is maintained by the Kotlin team at JetBrains and is developed to analyze Kotlin code specifically.

Kover has the ability to generate xml reports that have the same structure as JaCoCo Java reports to enable compatibility with existing tools. At the moment, codecov's coverage for Kover's XML reports is inaccurate, some of the problems are Kover bugs, some are gray areas and some seem to be server-side codecov issues. Umbrella issue for codecov integration issues: Kotlin/kotlinx-kover#16. We're working on making the integration better for our users.

cc @shanshin @qwwdfsad (maintainers of Kover)

[IgnatBeresnev]

As a side note: while the bug reported above could be fixed on your end, there are some gray areas that lead to inaccurate codecov reports, but it's a bug of neither kover nor codecov, and more of a JaCoCo format's limitation.

For instance, in Kotlin you can have multiple files with the same name if they are declared in different source sets or in different Gradle subprojects. This leads to codecov ignoring these files (presumably, it doesn't know which one to choose), but at the moment such files are indistinguishable in the report, so there doesn't seem to be an easy fix for it. For details, see Kotlin/kotlinx-kover#279

We (as in maintainers of Kover) are considering different options, ranging from enrichment of JaCoCo's format with custom tags (to provide an absolute path) to developing our own format altogether. However, for it to work, it would require external tools like codecov to support it.

How open are you to customizing server-side parsers to match Kover's (enriched) reports, and to which extent? :) What would be the best line of communication to coordinate efforts? Email/Slack/GH issues/etc?

[drazisil-codecov]

@IgnatBeresnev I just saw and shared it with our Product team.

[qwwdfsad]

Thanks!

We are keeping track of this issue closely as it would enable codecov integration in official Kotlin libraries, as well as make codecov compatible with the project structure we are advocating for, so feel free to ask any questions

[drazisil-codecov]

@qwwdfsad Absolutely!

The flat structure is something that customers have asked for, but Codecov currently can't support since we have no way to match the files in the reports to files in the repo. This would make a bunch of folks happy.

[drazisil-codecov]

@IgnatBeresnev What are the blockers to outputting absolute paths for files in the existing spec, rather then making changes that would require modifications to server-side and other tools in the community?

[qwwdfsad]

What are the blockers to outputting absolute paths for files in the existing spec

Could you please elaborate on what exact spec do you mean?

Is it correct that you would expect the following output for the example from the original report?

<package name="org/jetbrains/kover/reproducer/some/inner/pckg"> // <- non-patched as it is package, not path?

    <class name="org/jetbrains/kover/reproducer/PackageMismatchClass" // <- patched path
           sourcefilename="PackageMismatchFile.kt">
        ...
    </class>
    <sourcefile name="PackageMismatchFile.kt">
        ...
    </sourcefile>
    ...
</package>

[drazisil-codecov]

Could you please elaborate on what exact spec do you mean?

The Jacoco one https://github.com/jacoco/jacoco/blob/a68effb42f89682c275cc1e26418793191512985/org.jacoco.report/src/org/jacoco/report/xml/report.dtd

Is it correct that you would expect the following output for the example from the original report?

Codecov needs sourcefilename to match one (and only one) file that have been checked into the repository. If there are < 1 or > 1 matches the file will be skipped as we can not make a single match.

[qwwdfsad]

If we got it right, JaCoCo format spec has no notion of path in its data schema.

Let's take a look at the original report's data:

• We have the file PackageMismatchFile.kt with package org.jetbrains.kover.reproducer.some.inner.pckg
• It is located at src/kover/reproducer location. It has some.inner.pckg omitted, and thus codecov is skipping this file. We would like to fix it.

Here's the XML output:

<package name="org/jetbrains/kover/reproducer/some/inner/pckg">

    <class name="org/jetbrains/kover/reproducer/some/inner/pckg/PackageMismatchClass"
           sourcefilename="PackageMismatchFile.kt">
        ...
    </class>
    <sourcefile name="PackageMismatchFile.kt">
        ...
    </sourcefile>
    ...
</package>

Important note that JaCoCo XML format is not codecov-specific. It describes the overall coverage data for the given classes and packages and knows nothing about the actual filesystem layout. Moreover, other 3rd-party tools depend on that format and its properties' semantics, so it would be very much likely to break external tooling when making incompatible changes in such reports.

It basically contains four entities:

• package name. It has to be a fully-qualified package of a class covered (to have a proper grouping by package etc.). We cannot change it to path src/kover/reproducer
• class name. Exactly the same story, it should be a fully qualified classname in terms of JVM (package + name)
• Two entities left are sourcefile name and sourcefilename. It seems like their usage (all over various tools, as well as the JaCoCo implementation) considers this to be file names . It basically means two things:
  • Other tools (e.g. IDEA) may use it as a mask to search for an actual file in the filesystem layout. Making it a path instead of name will break it
  • Other tools (that I'm not aware of, hypothetical) will not expect path separators there (especially on Windows where it's \)

So, TL;DR: we cannot change the meaning of any of the XML properties without breaking other tools' assumptions.
Also, IDEA's viewer, in order to resolve this exact situation, does the following search:

• It optimistically assumes that the file can be located by package name + file name path
• If it's not, it scans the whole folder (building a hashmap-based index in memory to avoid O(n^2)) to locate this file

It would be nice if codecov could do the same. Alternatively, we can supply an additional XML-attribute somewhere with a relative path to a file in case of mismatch, if we will agree on that. Any other XML changes probably would also do the trick.

That's quite a lot of text, so please let me know if it makes sense or if I'm missing or misunderstanding something

[drazisil-codecov]

@qwwdfsad

I believe you are correct. I was thinking that sourcefilename contained paths normally , but it appears it does not.

Looking at our code, it appears we do the same thing as IDEA when it comes to crafting the file path:

    for package in xml.iter("package"):
        base_name = package.attrib["name"]
        for source in package.iter("sourcefile"):
            source_name = "%s/%s" % (base_name, source.attrib["name"])

If it's not, it scans the whole folder (building a hashmap-based index in memory to avoid O(n^2)) to locate this file

I can create a feature request for this, i don't know what resources it would consume , given that we deal with some very massive monorepos and this map would either be huge, or need to be computed on each file.

@aj-codecov, is #913 (comment) clear enough to copy over directly?

[qwwdfsad]

Thanks!

I can create a feature request for this

It would be really nice.

given that we deal with some very massive monorepos

The trick with this index is that it is not required at all if the class can be located at "%s/%s" % (base_name, source.attrib["name"]). So by default, it shouldn't affect any existing projects.

I believe that at some point, if the whole Kotlin ecosystem starts leveraging our "omitted common package prefix" scheme, we can come up with a better solution -- e.g. by adding a relative path to XML attributes.

[drazisil-codecov]

The trick with this index is that it is not required at all if the class can be located at "%s/%s" % (base_name, source.attrib["name"]). So by default, it shouldn't affect any existing projects

This is what we do now. We currently have no issues with Kotlin projects , only those that that already follow your "omitted common package prefix" scheme.

You can view this here https://github.com/codecov/standards, as it runs daily.

Did I misunderstood the ask / issue?

[qwwdfsad]

I mean, the index is not required by default for all Kotlin projects; it only is necessary to build if the file does not exist at %s/%s (-> for projects that has common prefix omitted), addressing "i don't know what resources it would consume , given that we deal with some very massive monorepos and this map would either be huge" -- it's unlikely that those huge monorepos require such index