Http storage management for scan-results.yml

[woznik]

Hello
In the recent release of ORT the toolkit stores scan results in several types like db, http, local.
I am using the http(artifactory) as a storage and the final shape of the stored results looks like this:

Where the folder artifacts contains the cached results for opensource libraries while the folder repository contains cached results of the scanned project.
My Use Case is to share the same scanned results with multiple projects only for opensoure libraries.
Is it possible to configure ORT not to store the repository results or store them in other place than the artifacts scan-results?

[sschuberth]

As an unrelated side note, the directory structure looks wrong to me: Seems like Artifactory is taking the "/" from URLs as directory separators for the structure on the server... @fviernau can this be related to your recent storage-by-provenance changes?

[fviernau]

As an unrelated side note, the directory structure looks wrong to me: Seems like Artifactory is taking the "/" from URLs as directory separators for the structure on the server... @fviernau can this be related to your recent storage-by-provenance changes?

My changes have been done to the storage which was previously only used by FileArchiver.
IIRC the file archiver hasn't ever turned URLs into a hierarchical structure but uses hashes of the storage key as
directory name. This is why I believe my changes are completely unrelated.

But to what storage does the above screenshot actually correspond to? We have as options

1. provenance resolution
2. scan results
3. file archiver
4. file lists

I guess this ticket refers to either 1 or 2 or both.

[mnonnenmacher]

@woznik There is currently no way to filter which scan results are stored or to use separate storage locations for projects and packages.

[fviernau]

Where the folder artifacts contains the cached results for opensource libraries while the folder repository contains cached results of the scanned project.

I guess you're making an assumption which does not hold in general: artifacts contains an entry for a package if the corresponding "sources artifact" has been scanned, while "repository" contains an entry if the corresponding VCS has been scanned. This is not a separation between open source and non-open source packages, in your case it was only by coincidence.

[woznik]

@fviernau Is there a way to skip storing the scan-results.yml file for the main project sources?
Use Case: I want to share the same storage place for multiple projects to save scanning time of opensource libraries but I don't want to share the sensitive data as project source code results with other projects.

[sschuberth]

I want to share the same storage place for multiple projects to save scanning time of opensource libraries but I don't want to share the sensitive data as project source code results with other projects.

Sounds a bit like #6460.

[woznik]

As an unrelated side note, the directory structure looks wrong to me: Seems like Artifactory is taking the "/" from URLs as directory separators for the structure on the server... @fviernau can this be related to your recent storage-by-provenance changes?

My changes have been done to the storage which was previously only used by FileArchiver. IIRC the file archiver hasn't ever turned URLs into a hierarchical structure but uses hashes of the storage key as directory name. This is why I believe my changes are completely unrelated.

But to what storage does the above screenshot actually correspond to? We have as options

1. provenance resolution
2. scan results
3. file archiver
4. file lists

I guess this ticket refers to either 1 or 2 or both.

Hello @fviernau

I want to reply to your comment.
Do you think it is correct that when I use the web storage with artifactory the created directories names are the protocols like on the screen below?

It looks strange to me. I remember that before the provenance storage ORT used to work differently. The structure was more readable and obvious.

Despite this suspicous names of folders in the artifactory structure ORT web Storage works well

[sschuberth]

It looks strange to me.

I agree this looks strange, It probably didn't occur to most of us as we prefer to use a database for storage. However, I do not currently see who's going to have time to look into this.