Empty scan results from failed runs are stored and reused even if the failure cause gets fixed and a new scan is triggered

[alpianon]

Describe the bug

When scancode completely fails to scan a package for whatever reason, no meaningful log message is shown and the processing goes on, until ORT finds that the temporary report.json output file from scancode is not there and throws a FileNotFoundException error, like in:

11:40:01.114 [main] ERROR org.ossreviewtoolkit.scanner.PathScannerWrapper - Failed to scan
RepositoryProvenance(vcsInfo=VcsInfo(type=Git, url=https://github.com/flozz/StackBlur.git, 
revision=1b85fe57ae5c2e7beeff157e1f9f7c7a7082f537, path=), resolvedRevision=1b85fe57ae5c2e7beeff157e1f9f7c7a7082f537)
with path scanner 'ScanCode': FileNotFoundException: /tmp/ort-ScanCode16441962167167250959/result.json (No such file or 
directory)

What is worse, the result with the generic error gets saved in ~/.ort/scanner/artifact/<name>/scan-results.yml, so it gets reused and the corresponding package never gets scanned again even if you fix the problem you had with ScanCode, and you will continue seeing the same error in the next runs of ort scan, unless you delete all the (wrong) cached results.

To Reproduce

Steps to reproduce the behavior:

1. force scancode to fail, f.e. add a wrong configuration entry for scancode in config.yml, like:

ort:
    config:
      ScanCode:
        options:
          commandLine: '--copyright --license --info --strip-root --timeout 600'

(since ORT 52.0.0, the commandLine field should contain parameters separated by comma and not by space; the correct value would be '--copyright,--license,--info,--strip-root,--timeout,600')

2. launch ort scan with the relevant options for your case
3. check the output log and the result file and see the generic error above
4. fix the scancode issue (in the example above, you should correct the commandLine value replacing spaces with commas)
5. re-run ort scan with the same parameters, check the result file and see that the errors are still there
6. delete all cached results with generic errors:

to_delete=$(find ~/.ort/scanner -name *.yml -exec grep -l FileNotFoundException {} \;)
rm $to_delete

7. re-run ort-scan with the same parameters; check the result file and the errors should be gone

Expected behavior

ORT should detect when ScanCode completely fails to run, and it should report the error message it gets from ScanCode instead of ignoring the error and throwing a generic FileNotFoundException when it does not find the report. Otherwise when ScanCode fails for whatever reason, it is impossible to analytically debug the error and one may just guess.

Moreover, in case ScanCode completely fails to run, no result should be cached.

Console / log output

see above

Environment

• ORT version: 54.0.0
• Java version: 21
• OS: Linux

But it should affect any previous version of ORT

[sschuberth]

the result with the generic error gets saved in ~/.ort/scanner/artifact/<name>/scan-results.yml, so it gets reused and the corresponding package never gets scanned again

This bit sounds a bit like eclipse-apoapsis/ort-server#2228, right @mnonnenmacher?

[alpianon]

@sschuberth I've just updated the description while working around my problem "live" :)

[mnonnenmacher]

the result with the generic error gets saved in ~/.ort/scanner/artifact/<name>/scan-results.yml, so it gets reused and the corresponding package never gets scanned again

This bit sounds a bit like eclipse-apoapsis/ort-server#2228, right @mnonnenmacher?

Yes, and it also seems related to #10050.

If ScanCode does not create a result file, it should create an issue and not write anything to the scan result storage. The problem is that the only way for the scanner to report issues is to add them to a scan result, there is no generic issues property. So for the ORT CLI we could probably solve this by not storing the result if ScanCode fails and creating a fake scan result. For the server this would not work, because there the issue would be lost without storing a result, but let's not get too much into detail about server specific issues here.

I see some options to solve this:

• Add a property to the ScanResult class to mark it as failed and do not reuse such results from the storage.
• Add a generic issues property to the ScannerRun class, to be able to report issues independent of scan results.
• Fail the scanner completely if ScanCode does not create a result file.