Skip to content

fix(spdx): Check for the presence of files #11910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oheger-bosch wants to merge 1 commit into from
Closed

fix(spdx): Check for the presence of files #11910

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions plugins/reporters/spdx/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ dependencies {

implementation(jacksonLibs.jacksonDatabind)

testImplementation(libs.mockk)

funTestImplementation(projects.utils.testUtils)
funTestImplementation(projects.plugins.licenseFactProviders.scancodeLicenseFactProvider)
funTestImplementation(testFixtures(projects.plugins.reporters.spdxReporter))
Expand Down
17 changes: 13 additions & 4 deletions plugins/reporters/spdx/src/main/kotlin/Extensions.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,15 +143,23 @@ internal enum class SpdxPackageType(val infix: String, val suffix: String = "")
/**
* Convert this ORT package to an SPDX package. As an ORT package can hold more metadata about its associated artifacts
* and origin than an SPDX package, the [type] is used to specify which kind of SPDX package should be created from the
* respective ORT package metadata.
* respective ORT package metadata. Also, the list of [files] contained in the package can be provided which has an

@fviernau fviernau May 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commit-msg: Where did you find this in the spec?

@oheger-bosch oheger-bosch May 29, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added more details to the commit message.

@fviernau fviernau May 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

I've read the links to the spec twice, but I cannot find any evidence, that package verification code must not be set if files are empty.

Any chance this is a bug in the verification tool?

@oheger-bosch oheger-bosch Jun 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to admit that from reading the spec, I would not necessarily come to the same results as the validation tool; however, the behavior of the tool also does not contradict what I read there. One could argue that the validation tool is the ultimate authority, since it implements the "intention" of the spec in an algorithm.

@oheger-bosch oheger-bosch Jun 2, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fviernau, I am trying to better understand the possible combinations of the affected properties, so that I maybe can file an issue against the validation tool. So, there are the properties hasFiles, filesAnalyzed, and packageVerificationCode. From my experiments, the validation tool seems to correlate all of them.

What is unclear to me is the exact meaning of the hasFiles property, which seems to be deprecated. The implementation in ORT does not list all files of the package here, but IIUC, only files with findings. So, packages without findings will have an empty file list. Why is this restriction made? According to Gemini, the description of the file in the SPDX 2.1 specification is just: "Contains a list of SPDXIDs of files that are contained in the package."

@oheger-bosch oheger-bosch Jun 3, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I belief, if all files were listed instead of only those with findings, this would also fix the validation errors we see: then there would always be files if a package verification code was available, and the properties would be consistent (except in the corner case of an empty package).

@fviernau fviernau Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

H, I have a hunch there could be a misunderstanding. I just found this, which has not been discussed:

  1. The package verification code is computed taking into account all files in the provenance, no matter of
    exclusion state, see 1.
  2. Not all files included in the verification code are actually included as SpdxFile.

For 1 I like that the code remains stable, independent of path excludes or license detections, and that it works regardless of how fileInformationEnabled is set, as mentioned by @tsteenbe .

For 2, I do not recall exactly why not all files are included. But I believe this would blow-up the size
of the report tremendously, and if implemented should have a feature toggle.

So, could the problem be that the verification tool assumes that the packageVerificationCode must match the included SpdxFiles in the report? (And assumption is wrong?)

Footnotes

  1. https://github.com/oss-review-toolkit/ort/blob/9d51987d4c5ebd9d7d9eb03a679746403213f288/plugins/reporters/spdx/src/main/kotlin/Extensions.kt#L232-L240 ↩

@oheger-bosch oheger-bosch Jun 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have found the place in the source code of the SPDX library where the related validations are done. This is not sophisticated, it actually correlates the properties in a way that is enforced by this PR. Do you think it should be different? I would be willing to file an issue, but am not sure what the desired behavior should be.

* impact on some properties.
*/
internal fun Package.toSpdxPackage(
type: SpdxPackageType,
licenseInfoResolver: LicenseInfoResolver,
ortResult: OrtResult
ortResult: OrtResult,
files: List<SpdxFile> = emptyList()
): SpdxPackage {
val packageVerificationCode = ortResult.getPackageVerificationCode(id, type)?.let {
SpdxPackageVerificationCode(packageVerificationCodeValue = it)
val packageVerificationCode = if (files.isNotEmpty()) {

@fviernau fviernau May 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commit: I wonder what kind of packages these are, which do not have an spdx file, but provenance is sccanned. Can you give an example?

@oheger-bosch oheger-bosch May 29, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is from an internal project which uses the SPDXDocument package manager. I don't know their exact setup and what the single packages contain.

@fviernau fviernau May 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, would it be possible to share a minimal reproducer?
I'm trying to understand the scenario.

@oheger-bosch oheger-bosch Jun 3, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, packages are affected for which the scanner does not return any findings. Therefore, the list of files is empty / not present, although the package was scanned.

ortResult.getPackageVerificationCode(id, type)?.let {
SpdxPackageVerificationCode(packageVerificationCodeValue = it)
}
} else {
// A package verification code is only allowed if files are present. Some other properties are derived from
// this as well.
null
}

val resolvedLicenseInfo = licenseInfoResolver.resolveLicenseInfo(id).filterExcluded()
Expand Down Expand Up @@ -181,6 +189,7 @@ internal fun Package.toSpdxPackage(
},
externalRefs = if (type == SpdxPackageType.PROJECT) emptyList() else toSpdxExternalReferences(),
filesAnalyzed = packageVerificationCode != null,
hasFiles = files.map { it.spdxId },
homepage = homepageUrl.nullOrBlankToSpdxNone(),
licenseComments = "effectiveLicense: ${effectiveLicense?.toString().nullOrBlankToSpdxNone()}",
licenseConcluded = when (type) {
Expand Down
15 changes: 9 additions & 6 deletions plugins/reporters/spdx/src/main/kotlin/SpdxDocumentModelMapper.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,9 @@ internal object SpdxDocumentModelMapper {
val spdxProjectPackage = project.toPackage().toSpdxPackage(
SpdxPackageType.PROJECT,
licenseInfoResolver,
ortResult
).copy(hasFiles = filesForProject.map { it.spdxId })
ortResult,
filesForProject
)

ortResult.getDependencies(
id = project.id,
Expand Down Expand Up @@ -138,8 +139,9 @@ internal object SpdxDocumentModelMapper {
val vcsPackage = pkg.toSpdxPackage(
SpdxPackageType.VCS_PACKAGE,
licenseInfoResolver,
ortResult
).copy(hasFiles = filesForPackage.map { it.spdxId })
ortResult,
filesForPackage
)

val vcsPackageRelationShip = SpdxRelationship(
spdxElementId = binaryPackage.spdxId,
Expand All @@ -163,8 +165,9 @@ internal object SpdxDocumentModelMapper {
val sourceArtifactPackage = pkg.toSpdxPackage(
SpdxPackageType.SOURCE_PACKAGE,
licenseInfoResolver,
ortResult
).copy(hasFiles = filesForPackage.map { it.spdxId })
ortResult,
filesForPackage
)

val sourceArtifactPackageRelationship = SpdxRelationship(
spdxElementId = binaryPackage.spdxId,
Expand Down
75 changes: 75 additions & 0 deletions plugins/reporters/spdx/src/test/kotlin/ExtensionsTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
/*
* Copyright (C) 2026 The ORT Project Copyright Holders <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.reporters.spdx

import io.kotest.core.spec.style.WordSpec
import io.kotest.matchers.collections.beEmpty
import io.kotest.matchers.collections.shouldContainExactly
import io.kotest.matchers.nulls.beNull
import io.kotest.matchers.should
import io.kotest.matchers.shouldBe
import io.kotest.matchers.shouldNot

import io.mockk.mockk

import org.ossreviewtoolkit.model.Identifier
import org.ossreviewtoolkit.utils.spdxdocument.model.SpdxFile

class ExtensionsTest : WordSpec({

Check warning on line 35 in plugins/reporters/spdx/src/test/kotlin/ExtensionsTest.kt

View workflow job for this annotation

GitHub Actions / qodana-scan

Unused symbol 

Class "ExtensionsTest" is never used
Comment thread
sschuberth marked this conversation as resolved.
Dismissed
"toSpdxPackage" should {
"set filesAnalyzed to true if there is a package verification code and files are available" {
val id = Identifier("Maven:pkg1-grp:pkg1:0.0.1")
val pkg = requireNotNull(ORT_RESULT.getPackage(id)?.metadata)
val spdxFile = SpdxFile(
spdxId = "SPDXRef-Pkg1-testFile",
filename = "testFile",
checksums = emptyList(),
licenseConcluded = "MIT"
)
val files = listOf(spdxFile)

val spdxPkg = pkg.toSpdxPackage(
type = SpdxPackageType.VCS_PACKAGE,
licenseInfoResolver = mockk(relaxed = true),
ortResult = ORT_RESULT,
files = files
)

spdxPkg.filesAnalyzed shouldBe true
spdxPkg.hasFiles shouldContainExactly listOf(spdxFile.spdxId)
spdxPkg.packageVerificationCode shouldNot beNull()
}

"set filesAnalyzed to false and no verification code if there are no files available" {
val id = Identifier("Maven:pkg1-grp:pkg1:0.0.1")
val pkg = requireNotNull(ORT_RESULT.getPackage(id)?.metadata)

val spdxPkg = pkg.toSpdxPackage(
type = SpdxPackageType.VCS_PACKAGE,
licenseInfoResolver = mockk(relaxed = true),
ortResult = ORT_RESULT
)

spdxPkg.filesAnalyzed shouldBe false
spdxPkg.hasFiles should beEmpty()
spdxPkg.packageVerificationCode should beNull()
}
}
})
Loading