Skip to content

[build] drop the whole Scala Native work directory in CI, not just generated/ - #1822

Merged
fwbrasil merged 2 commits into
mainfrom
t3code/fix-issue-1821
Jul 30, 2026
Merged

[build] drop the whole Scala Native work directory in CI, not just generated/#1822
fwbrasil merged 2 commits into
mainfrom
t3code/fix-issue-1821

Conversation

@DamianReeves

Copy link
Copy Markdown
Collaborator

Fixes #1821.

What was happening

The Native rows failed at link with a run of

clang: error: no such file or directory: .../generated/<hash>.ll.o

and no diagnostic from any earlier phase. Deterministic: the same nine hashes on both arches, shared across kyo-aeron (#1818) and kyo-actor (#1809), because those are the compilation units whose NIR nothing in the build perturbs.

The issue guessed that clang was failing and its stderr was being discarded. It is not: clang is never started for those units.

Cause

Test / nativeLink deleted <module>/native/target/scala-*/native-test/generated after each link, to keep ~40 modules inside the runner's disk. It left package2hash, Scala Native's incremental-codegen state naming those IR files, as a sibling of the directory it deleted.

Scala Native's build-skip checksum covers classpath mtimes, so a module whose dependency the test phase recompiled relinks rather than short-circuiting. Then:

  1. CodeGen.separateIncrementally reads package2hash, sees the unit unchanged, and returns the .ll path without checking the file exists. Its only guard is assert(ownerDirectory.toFile.exists()) on the parent directory, which emit recreated a few lines earlier.
  2. LLVM.needsCompiling compares two absent files, 0 > 0, skips clang, and returns the object path anyway.
  3. Both phases log success. The link is the first thing to touch the disk.

Fix

Prune the whole work directory except build-checksum. That file plus the linked binary (one level up, outside the work directory) is the entire input to Scala Native's "Build skipped" check, so a repeat link still short-circuits. Dropping the rest wholesale is what keeps the directory self-consistent, and it reclaims more disk than before.

Guard

scripts/native-relink-selftest.sh: link, assert the work directory holds only build-checksum, bump one classpath mtime so the checksum misses without recompiling anything, relink, assert it succeeds and referenced no object codegen never wrote. --self-test covers the harness against a faked sbt, negative control included. The Native rows run it before the suite, on kyo-data, and reuse the binary it links.

Verification

Reproduced locally on macOS with kyo-dataNative, hitting the same nine hashes as CI.

Check Result
native-relink-selftest.sh --self-test 7/7
guard against the fixed build pass
guard against the old hook fails on work directory not pruned
old hook plus a checksum miss exit 1, nine missing .ll.o
no-change relink after the prune Build skipped, fast path intact
actionlint, shellcheck, scalafmtSbt clean

Upstream

The underlying defect is in Scala Native 0.5.12 and reproduces without kyo: https://github.com/DamianReeves/scala-native-bug-reporting

Not yet filed against scala-native.

…nerated/

The Native rows failed at link with a run of

  clang: error: no such file or directory: .../generated/<hash>.ll.o

and no diagnostic from any earlier phase. It was deterministic: the same nine
hashes on both arches, and shared across kyo-aeron and kyo-actor, because those
are the compilation units whose NIR nothing in the build perturbs.

The Test / nativeLink hook deleted <module>/native/target/scala-*/native-test/generated
after each link to keep ~40 modules inside the runner's disk, but left
package2hash, the incremental-codegen state naming those IR files, sitting next
to it. Scala Native's build-skip checksum covers classpath mtimes, so a relink
whose module had a dependency recompiled (which the test phase does) rebuilds,
and then CodeGen.separateIncrementally trusts package2hash, skips regenerating
every unchanged unit, and returns the .ll path without checking the file is
there. Its only guard is on the parent directory, which emit just recreated.
LLVM.needsCompiling then compares two absent files, 0 > 0, skips clang, and
returns the object path anyway. Codegen and compilation both log success and
the link is the first thing to touch the disk.

Prune everything except build-checksum. That file plus the linked binary, which
sits one level up outside the work directory, is the entire input to the
"Build skipped" check, so a repeat link still short-circuits; dropping the rest
wholesale is what keeps the directory self-consistent, and it reclaims more
disk than before.

scripts/native-relink-selftest.sh guards it: link, assert the work directory
holds only build-checksum, bump one classpath mtime so the checksum misses
without recompiling anything, relink, assert it succeeds and referenced no
object codegen never wrote. Its own --self-test covers the harness against a
faked sbt, negative control included. The Native rows run it before the suite,
on kyo-data, and reuse the binary it links.

Fixes #1821
…appens

Every job on every arch has failed in the setup action since 571ecd4, which
added runc to the podman apt install. Ubuntu's runc package Conflicts with
Docker's containerd.io, so apt resolves it by removing containerd.io and
docker-ce:

  The following packages will be REMOVED:
    containerd.io docker-ce

`docker version`, the last line of the podman start step, then fails under
`bash -e` and takes the whole setup down before any target builds. The kyo-pod
suite also runs its container cases against docker as well as podman, so the
removal costs coverage even where setup survives.

The runners already carry a runc binary alongside containerd.io, which is why
that package Conflicts in the first place. Install podman and uidmap as before
and reach for the runc package only when the runner has none, so the normal path
installs nothing and leaves docker alone.

Both runtimes have now failed silently here, so the step names what broke: it
prints which OCI runtime podman resolved, and an unreachable docker daemon
reports the conflict to look for instead of a bare exit code.
@fwbrasil
fwbrasil merged commit 6241961 into main Jul 30, 2026
14 of 16 checks passed
@fwbrasil
fwbrasil deleted the t3code/fix-issue-1821 branch July 30, 2026 15:43
DamianReeves added a commit to DamianReeves/scala-native-bug-reporting that referenced this pull request Jul 30, 2026
Deleting generated/ alone is the trap: package2hash is its sibling, not its
child, so it outlives the IR it names. Keeping only build-checksum, which with
the artifact is the whole input to Build.buildCached's skip check, preserves the
short-circuit while leaving nothing behind that can describe deleted IR.

Verified in getkyo/kyo#1822, where the Scala Native CI rows that were failing on
missing .ll.o are green on x64 and arm64.

Still a workaround: any other route to a missing generated/ produces the same
undiagnosed link failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Native test link fails on missing .ll.o objects with no clang diagnostic (kyo-aeron, kyo-actor)

2 participants