[build] drop the whole Scala Native work directory in CI, not just generated/ - #1822
Merged
Conversation
…nerated/ The Native rows failed at link with a run of clang: error: no such file or directory: .../generated/<hash>.ll.o and no diagnostic from any earlier phase. It was deterministic: the same nine hashes on both arches, and shared across kyo-aeron and kyo-actor, because those are the compilation units whose NIR nothing in the build perturbs. The Test / nativeLink hook deleted <module>/native/target/scala-*/native-test/generated after each link to keep ~40 modules inside the runner's disk, but left package2hash, the incremental-codegen state naming those IR files, sitting next to it. Scala Native's build-skip checksum covers classpath mtimes, so a relink whose module had a dependency recompiled (which the test phase does) rebuilds, and then CodeGen.separateIncrementally trusts package2hash, skips regenerating every unchanged unit, and returns the .ll path without checking the file is there. Its only guard is on the parent directory, which emit just recreated. LLVM.needsCompiling then compares two absent files, 0 > 0, skips clang, and returns the object path anyway. Codegen and compilation both log success and the link is the first thing to touch the disk. Prune everything except build-checksum. That file plus the linked binary, which sits one level up outside the work directory, is the entire input to the "Build skipped" check, so a repeat link still short-circuits; dropping the rest wholesale is what keeps the directory self-consistent, and it reclaims more disk than before. scripts/native-relink-selftest.sh guards it: link, assert the work directory holds only build-checksum, bump one classpath mtime so the checksum misses without recompiling anything, relink, assert it succeeds and referenced no object codegen never wrote. Its own --self-test covers the harness against a faked sbt, negative control included. The Native rows run it before the suite, on kyo-data, and reuse the binary it links. Fixes #1821
…appens Every job on every arch has failed in the setup action since 571ecd4, which added runc to the podman apt install. Ubuntu's runc package Conflicts with Docker's containerd.io, so apt resolves it by removing containerd.io and docker-ce: The following packages will be REMOVED: containerd.io docker-ce `docker version`, the last line of the podman start step, then fails under `bash -e` and takes the whole setup down before any target builds. The kyo-pod suite also runs its container cases against docker as well as podman, so the removal costs coverage even where setup survives. The runners already carry a runc binary alongside containerd.io, which is why that package Conflicts in the first place. Install podman and uidmap as before and reach for the runc package only when the runner has none, so the normal path installs nothing and leaves docker alone. Both runtimes have now failed silently here, so the step names what broke: it prints which OCI runtime podman resolved, and an unreachable docker daemon reports the conflict to look for instead of a bare exit code.
DamianReeves
added a commit
to DamianReeves/scala-native-bug-reporting
that referenced
this pull request
Jul 30, 2026
Deleting generated/ alone is the trap: package2hash is its sibling, not its child, so it outlives the IR it names. Keeping only build-checksum, which with the artifact is the whole input to Build.buildCached's skip check, preserves the short-circuit while leaving nothing behind that can describe deleted IR. Verified in getkyo/kyo#1822, where the Scala Native CI rows that were failing on missing .ll.o are green on x64 and arm64. Still a workaround: any other route to a missing generated/ produces the same undiagnosed link failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1821.
What was happening
The Native rows failed at link with a run of
and no diagnostic from any earlier phase. Deterministic: the same nine hashes on both arches, shared across
kyo-aeron(#1818) andkyo-actor(#1809), because those are the compilation units whose NIR nothing in the build perturbs.The issue guessed that clang was failing and its stderr was being discarded. It is not: clang is never started for those units.
Cause
Test / nativeLinkdeleted<module>/native/target/scala-*/native-test/generatedafter each link, to keep ~40 modules inside the runner's disk. It leftpackage2hash, Scala Native's incremental-codegen state naming those IR files, as a sibling of the directory it deleted.Scala Native's build-skip checksum covers classpath mtimes, so a module whose dependency the test phase recompiled relinks rather than short-circuiting. Then:
CodeGen.separateIncrementallyreadspackage2hash, sees the unit unchanged, and returns the.llpath without checking the file exists. Its only guard isassert(ownerDirectory.toFile.exists())on the parent directory, whichemitrecreated a few lines earlier.LLVM.needsCompilingcompares two absent files,0 > 0, skips clang, and returns the object path anyway.Fix
Prune the whole work directory except
build-checksum. That file plus the linked binary (one level up, outside the work directory) is the entire input to Scala Native's "Build skipped" check, so a repeat link still short-circuits. Dropping the rest wholesale is what keeps the directory self-consistent, and it reclaims more disk than before.Guard
scripts/native-relink-selftest.sh: link, assert the work directory holds onlybuild-checksum, bump one classpath mtime so the checksum misses without recompiling anything, relink, assert it succeeds and referenced no object codegen never wrote.--self-testcovers the harness against a faked sbt, negative control included. The Native rows run it before the suite, onkyo-data, and reuse the binary it links.Verification
Reproduced locally on macOS with
kyo-dataNative, hitting the same nine hashes as CI.native-relink-selftest.sh --self-testwork directory not pruned.ll.oBuild skipped, fast path intactactionlint,shellcheck,scalafmtSbtUpstream
The underlying defect is in Scala Native 0.5.12 and reproduces without kyo: https://github.com/DamianReeves/scala-native-bug-reporting
Not yet filed against
scala-native.