FileArchiver creates unnecessary archives

[oheger-bosch]

When scanning a large repository containing a Maven build with many subprojects, I noticed that the FileArchiver was invoked for each subproject. The repository was then cloned, an archive was created, and the archive was written to the storage.

I have created #11483 to prevent that the repository is checked out again and again for each project. However, from looking at the code in FileArchiver, it seems that this scenario is handled in a rather strange way:

The FileArchiver.archive() function creates an archive from the root of the checked out repository whose exact content may depend on the passed in package ID (the project ID in this case). The archive is then passed to the storage and associated with the provenance. So, when having a provenance with multiple packages, IIUC, the archiver creates multiple archives which override each other in the storage. This is even more problematic since the archives may have different content, depending on the package ID.

I assume, there should be only a single archive per provenance; or when storing archives, the package ID would need to be taken into account.

[oheger-bosch]

Any comments or feedback here? At least for the mentioned project, the creation of file archives in the current form contributes significantly to the execution time of the scanner step; it took about 10 minutes to clone the repository and create the archive file for every subproject. I assume that all larger projects with many subprojects are affected.

[sschuberth]

Note to myself, as I was missing a bit of context here: The (optionally configured) archive of a ScannerConfiguration is used to "archive[s] certain scanned files in an external FileStorage". Those "certain files" are defined by LicenseFilePatterns. So in short, the feature is used to archive license files.

As license files can vary by package within a monorepo, I basically agree that associating the license file archives with only a repository provenance is not enough.

However, I'm not sure if the package ID needs to be taken into account already when storing archives. I could imagine that we have logic in place that picks the right license files for a given package from the corresponding subdirectory in a file archive; so we take the package ID into account when reading files from the archive, I assume.

In any case, if we do provenance-level archives that include all license files for all packages for that provenance, I agree we should not create that archive multiple times per provenance.

Finally, it all sounds to me as if something in this check is not working as expected:

ort/scanner/src/main/kotlin/Scanner.kt

Lines 736 to 739 in 38fba04

	val provenancesWithMissingArchives = controller.getNestedProvenancesByPackage()
	.filterNot { (_, nestedProvenance) -> archiver.hasArchive(nestedProvenance.root) }
	if (provenancesWithMissingArchives.isEmpty()) return

[oheger-bosch]

After recent changes in ad73569, there is no longer a dependency to a package when creating a file archive, and the API of the FileArchiver has been changed accordingly. So, it is clear that only a single archive should be created per provenance. Currently, for provenances containing multiple packages, the (same) archive is created repeatedly for each package. #11483 fixes this.