Cleanup Error on Scan

[Martin-Idel]

Describe the bug

Scanning Excalidraw (https://github.com/excalidraw/excalidraw) results in an error and no ort output.

To Reproduce

Steps to reproduce the behavior:

1. Check out repository for excalidraw at following link 'https://github.com/excalidraw/excalidraw/releases/tag/v0.17.3'
2. Run Analyzer from dockerfile built from ORT master (no issues), via:

docker run -v ~/gitRepos/ort_output/:/home/ort/.ort/config/ -v ~/gitRepos/excalidraw/:/download -v ~/gitRepos/ort_output/ort_excalidraw/analyze/:/analyze ort:latest analyze -i /download -o/analyze

Config is just

ort:
  analyzer:
    allowDynamicVersions: true

4. Finally do run the scanner:

docker run --network=host -e ORT_CONFIG_DIR=/config -v ~/gitRepos/ort_output/ort_excalidraw/:/config/ -v ~/gitRepos/ort_output/ort_excalidraw/analyze/:/analyze -v ~/gitRepos/ort_output/ort_excalidraw/scan/:/scan ort:latest --info scan -i /analyze/analyzer-result.yml -o/scan

6. See error

I also tried adding options:
-e "JAVA_OPTS=-Xms512M -Xmx10g -XX:MaxMetaspaceSize=6g -Dkotlin.daemon.jvm.options='-Xmx16g'"
--memory="32g"
and various combinations of sizes, that seemed to not help.

I also tried using a postgres database which significantly reduced scan time, however the error remains the same.

Expected behavior

No error.

Console / log output

09:19:42.855 [main] INFO org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git checkout 0136e8a05ef864666a6e95b0de8b679b0bc93338' in '/tmp/ort-DefaultWorkingTreeCache18354474298420537781'...
Exception in thread "main" java.nio.file.NoSuchFileException: /tmp/ort-DefaultWorkingTreeCache18354474298420537781/.git/gc.log.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171)
at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1853)
at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)
at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2820)
at kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree(PathUtils.kt:1147)
at kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree(PathUtils.kt:1201)
at kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree$default(PathUtils.kt:1193)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.copyToRecursively(PathRecursiveFunctions.kt:234)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.copyToRecursively(PathRecursiveFunctions.kt:81)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.copyToRecursively$default(PathRecursiveFunctions.kt:72)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadFromVcs$lambda$1(ProvenanceDownloader.kt:117)
at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.use(WorkingTreeCache.kt:65)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadFromVcs(ProvenanceDownloader.kt:104)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.access$downloadFromVcs(ProvenanceDownloader.kt:80)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invokeSuspend(ProvenanceDownloader.kt:96)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invoke(ProvenanceDownloader.kt)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader$download$1.invoke(ProvenanceDownloader.kt)
at org.ossreviewtoolkit.utils.ort.UtilsKt$runBlocking$1.invokeSuspend(Utils.kt:231)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at org.ossreviewtoolkit.utils.ort.UtilsKt.runBlocking(Utils.kt:231)
at org.ossreviewtoolkit.utils.ort.UtilsKt.runBlocking$default(Utils.kt:229)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.download(ProvenanceDownloader.kt:96)
at org.ossreviewtoolkit.scanner.provenance.ProvenanceDownloader$DefaultImpls.downloadRecursively(ProvenanceDownloader.kt:64)
at org.ossreviewtoolkit.scanner.provenance.DefaultProvenanceDownloader.downloadRecursively(ProvenanceDownloader.kt:80)
at org.ossreviewtoolkit.scanner.Scanner.createMissingArchives(Scanner.kt:708)
at org.ossreviewtoolkit.scanner.Scanner.scan(Scanner.kt:179)
at org.ossreviewtoolkit.scanner.Scanner$scan$3.invokeSuspend(Scanner.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:263)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:47)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.runScanners(ScannerCommand.kt:242)
at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.run(ScannerCommand.kt:148)
at com.github.ajalt.clikt.core.CoreCliktCommandKt.parse(CoreCliktCommand.kt:107)
at com.github.ajalt.clikt.core.CoreCliktCommandKt.main(CoreCliktCommand.kt:78)
at com.github.ajalt.clikt.core.CoreCliktCommandKt.main(CoreCliktCommand.kt:90)
at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:88)

Environment

Output of the ort requirements -l commands command:

09:42:44.616 [main] INFO  org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file '/config/config.yml'.
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
09:42:44.936 [main] INFO  org.ossreviewtoolkit.utils.common.EnvironmentVariableFilter - EnvironmentVariableFilter initialized with denySubstrings = [key, pass, pwd, token, user] and allowNames = [CARGO_HTTP_USER_AGENT, COMPOSER_ALLOW_SUPERUSER, CONAN_LOGIN_ENCRYPTION_KEY, CONAN_LOGIN_USERNAME, CONAN_PASSWORD, CONAN_USERNAME, CONAN_USER_HOME, CONAN_USER_HOME_SHORT, DOTNET_CLI_CONTEXT_ANSI_PASS_THRU, GIT_ASKPASS, GIT_HTTP_USER_AGENT, GRADLE_USER_HOME, HACKAGE_USERNAME, HACKAGE_PASSWORD, HACKAGE_KEY, PWD, USER, USERPROFILE].
 ______________________________                                                
/        \_______   \__    ___/ The OSS Review Toolkit, version DOCKER-SNAPSHOT
|    |   | |       _/ |    |    built with JDK 21.0.5+11-LTS, running under Jav
|    |   | |    |   \ |    |    Executing 'requirements' as 'ort' on Linux     
\________/ |____|___/ |____|    with 16 CPUs and a maximum of 10240 MiB of memo
                                                                               
Environment variables:                                                        
ORT_CONFIG_DIR = /config                                                      
ORT_DATA_DIR = /home/ort/.ort                                                 
HOME = /home/ort                                                              
JAVA_HOME = /opt/java/openjdk                                                 
ANDROID_HOME = /opt/android-sdk                                               
                                                                              
Looking for ORT configuration in the following file:
        /config/config.yml

09:42:45.330 [main] INFO  org.reflections.Reflections - Reflections took 270 ms to scan 94 urls, producing 155 keys and 1369 values
Scanners:
        - Askalono: Requires 'askalono' in no specific version. Tool not found.
        - BoyterLc: Requires 'lc' in no specific version. Tool not found.
        - Licensee: Requires 'licensee' in no specific version. Tool not found.
09:42:45.599 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'scancode --version' in '/home/ort'...
        * ScanCode: Requires 'scancode' in version >=30.0.0. Found version 32.3.1.

PackageManagers:
09:42:46.760 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'bazel --version' in '/home/ort'...
        * BazelCommand: Requires 'bazel' in version >=7.0.0. Found version 7.0.1.
09:42:50.771 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'bower --version' in '/home/ort'...
        * BowerCommand: Requires 'bower' in version >=1.8.8. Found version 1.8.14.
09:42:51.138 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'buildozer --version' in '/home/ort'...
        + BuildozerCommand: Requires 'buildozer' in no specific version. Found version redacted.
09:42:51.159 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'cargo --version' in '/home/ort'...
        * CargoCommand: Requires 'cargo' in no specific version. Found version 1.84.0.
09:42:51.243 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'pod --version --allow-root' in '/home/ort'...
        * CocoaPodsCommand: Requires 'pod' in version >=1.11.0. Found version 1.16.2.
09:42:51.915 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'composer --no-ansi --version' in '/home/ort'...
        * ComposerCommand: Requires 'composer' in version >=1.5.0. Found version 2.8.4.
09:42:52.000 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'conan --version' in '/home/ort'...
        * ConanCommand: Requires 'conan' in version >=1.44.0 and <2.0.0. Found version 1.64.1.
09:42:52.517 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'go version' in '/home/ort'...
        * GoCommand: Requires 'go' in version >=1.21.1. Found version 1.23.0.
09:42:52.553 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'npm --version' in '/home/ort'...
        * NpmCommand: Requires 'npm' in version >=6.0.0 and <11.0.0. Found version 10.9.2.
09:42:52.671 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'nuget-inspector --version' in '/home/ort'...
        + NuGetInspector: Requires 'nuget-inspector' in no specific version. Could not determine the version.
09:42:53.047 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'pipenv --version' in '/home/ort'...
        * PipenvCommand: Requires 'pipenv' in version >=2018.10.9. Found version 2023.12.1.
09:42:53.679 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'pnpm --version' in '/home/ort'...
        * PnpmCommand: Requires 'pnpm' in version >=5.0.0 and <10.0.0. Found version 9.9.0.
09:42:54.072 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'poetry --version' in '/home/ort'...
        * PoetryCommand: Requires 'poetry' in no specific version. Found version 1.8.3.
09:42:54.505 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'dart --version' in '/home/ort'...
        * Pub: Requires 'dart' in version >=2.10.0. Found version 2.18.4.
09:42:54.532 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'python-inspector --version' in '/home/ort'...
        * PythonInspector: Requires 'python-inspector' in version >=0.9.2. Found version 0.10.0.
09:42:54.837 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'sbt --version' in '/home/ort'...
        * SbtCommand: Requires 'sbt' in no specific version. Found version copying runtime jar...
sbt version in this project: 1.10.0
sbt script version: 1.10.0.
09:43:13.913 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'stack --version' in '/home/ort'...
        * StackCommand: Requires 'stack' in version >=2.1.1. Found version 3.3.1.
09:43:13.953 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'swift --version' in '/home/ort'...
        * SwiftCommand: Requires 'swift' in no specific version. Found version 5.10.1.
09:43:14.375 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'yarn --version' in '/home/ort'...
        * YarnCommand: Requires 'yarn' in version >=1.3.0 and <1.23.0. Found version 1.22.22.

VersionControlSystems:
09:43:14.581 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'git --version' in '/home/ort'...
        * Git: Requires 'git' in version >=2.29.0. Found version 2.34.1.
09:43:14.595 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'repo --version' in '/home/ort'...
        * GitRepo: Requires 'repo' in no specific version. Found version 2.50 (launcher).
09:43:14.672 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'hg --version' in '/home/ort'...
        * Mercurial: Requires 'hg' in no specific version. Found version 6.9.1.

Prefix legend:
        - The tool was not found in the PATH environment.
        + The tool was found in the PATH environment, but not in the required version.
        * The tool was found in the PATH environment in the required version.

ScanCode license texts found in '/opt/scancode-license-data'.

Not all tools requirements were satisfied:
        ! Some tools were not found in their required versions.

Additional context

Running with --debug it seems that this happens during some cleanup.

[sschuberth]

Good to read from you again @Martin-Idel 😉

Running 'git checkout 0136e8a05ef864666a6e95b0de8b679b0bc93338' in '/tmp/ort-DefaultWorkingTreeCache18354474298420537781'...
Exception in thread "main" java.nio.file.NoSuchFileException: /tmp/ort-DefaultWorkingTreeCache18354474298420537781/.git/gc.log.lock

I assume this could be a permission problem, i.e. the file actually is there but not accessible for whether reason. If so, the problem should appear with any (smaller) project in your specific setup. Is that something you could confirm?

[Martin-Idel]

Thanks @sschuberth !

Sorry for the delay, but it took me a while to run a couple of tests.
I tried it with the mime-types repository (exact same commands) and all is well, so it seems to be a more specific issue.

I have some issues with permissions, but it would be a bit weird in THIS case, because the file is in a temporary folder, why should access be messed up there? I don't have any mount point on that directory and there is no restriction on the docker container to use some space. I also checked df -h and my main partition has 140 GB available so that should also be a non-issue (hopefully :D ).

[sschuberth]

Can you attach the (compressed) ORT analyzer result that you used as the input to the ort scan command to this issue?

[Martin-Idel]

Sorry, I could have thought of that. Here it is:

analyzer-result.zip