ci: GHA runner crashes with "No space left on device" on _diag/Worker_*.log during [rhoai] builds

[jiridanek]

Problem

GHA runners crash with System.IO.IOException: No space left on device on the runner's internal diagnostic log (/home/runner/actions-runner/cached/2.334.0/_diag/Worker_*.log), not on build storage. This kills the runner process before the build step completes.

The crash consistently affects [rhoai] hermetic builds (which download ~3GB prefetch deps + subscription setup) but not [odh] builds on the same commit. It affects both PR builds and manually-triggered push builds that include [rhoai].

Evidence

PR builds (symlink PR #3633): Multiple [rhoai] jobs crash with identical stack trace:

• runtime-minimal-ubi9-python-3.12 · linux/amd64 [rhoai]
• runtime-datascience-ubi9-python-3.12 · linux/amd64 [rhoai]
• jupyter-minimal-ubi9-python-3.12 · linux/s390x [rhoai]

Main push builds (manual dispatch): Same crash on run 25937471902:

• jupyter-minimal-ubi9-python-3.12 · linux/ppc64le / rhoai
• jupyter-minimal-ubi9-python-3.12 · linux/amd64 / rhoai

Auto push builds (odh only): No _diag crashes — only unrelated pytest failures.

Disk layout at crash time

From the LVM overlay step logs:

/dev/root                     72G   58G   15G  80% /
/dev/sdb1                     74G  8.1G   62G  12% /mnt
/dev/mapper/buildvg-buildlv   72G  5.8M   70G   1% /home/runner/.local/share/containers

The build storage (buildvg-buildlv) has 70GB free. Root has 15GB free. The _diag/Worker_*.log lives on root — and something fills those 15GB during the build.

The visible step logs for the crashed jobs total only ~150-280KB, so it's not the build stdout filling root. The runner's internal _diag tracing captures more than what's visible in the step logs.

Diagnostic monitoring added

PR #3633 commit 9d99f4c3b adds background monitoring to the build template:

• df -h + _diag directory size every 30s
• fatrace — traces which processes write to root fs
• bpftrace — sums bytes written per process every 30s
• Each long-running step (prefetch, build) tails the monitor inline so output is visible even if the runner crashes

Hypotheses

1. Runner diagnostic tracing bloat: The runner traces all stdout/stderr to _diag/Worker_*.log internally, not just what's shown in step logs. Hermetic prefetch + subscription setup may produce verbose internal tracing that fills root.

2. Container storage leaking to root: Despite the LVM overlay, some container operations (layer extraction, cache) may write to root before the overlay is mounted.

3. Go module cache / tool downloads: go build for bin/buildinputs, uv sync, and other setup steps download to ~/go/, ~/.cache/uv/ etc. on root.

4. QEMU emulation overhead: Cross-arch builds (s390x, ppc64le via qemu-user) may produce additional core dumps or temp files on root.

5. Runner image got larger: If the base runner image grew, 15GB remaining after LVM setup is no longer enough. May need to adjust LVM allocation to leave more space on root.

Potential fixes

• Move _diag to LVM volume: Symlink _diag directory to the 70GB build volume
• Leave more root space in LVM: Adjust gha_lvm_overlay.sh to take less from root
• Truncate _diag logs periodically: Background job that truncates Worker_*.log when it exceeds a threshold
• Reduce runner tracing: Check if ACTIONS_RUNNER_DEBUG is somehow enabled, or if there's a way to reduce internal trace verbosity

Related issues

• actions/runner-images#13719 — Runner provisioned with full disk
• GitHub Community #133486 — Same _diag crash stack trace
• pmem/run_qemu#235 — QEMU project hit identical crash, Dec 2025
• jiridanek/rhoai-in-kind#10 — Similar disk exhaustion on GHA, different symptom

Next steps

• Wait for diagnostic monitor results from PR RHAIENG-5139: symlink Dockerfile.* → Dockerfile.konflux.* for all 18 image variants #3633 CI run
• Identify what's consuming root space from fatrace/bpftrace output
• Apply targeted fix (move _diag, adjust LVM, or reduce tracing)

[jiridanek]

Investigation notes — disk layout and crash pattern

Runner disk layout after LVM setup

The gha_lvm_overlay.sh script creates an LVM volume from root + /mnt free space:

Before LVM:
/dev/root    72G   58G   15G  80% /
/dev/sdb1    74G  4.1G   66G   6% /mnt

After LVM:
/dev/root                     72G   58G   15G  80% /
/dev/sdb1                     74G  8.1G   62G  12% /mnt
/dev/mapper/buildvg-buildlv   72G  5.8M   70G   1% /home/runner/.local/share/containers
Swap:          8.0Gi          0B       8.0Gi

Key: The _diag directory lives at /home/runner/actions-runner/cached/2.334.0/_diag/ which is on /dev/root (15GB free), NOT on the LVM volume.

Crash stack trace (identical across all occurrences)

System.IO.IOException: No space left on device : '/home/runner/actions-runner/cached/2.334.0/_diag/Worker_20260515-*.log'
   at System.IO.RandomAccess.WriteAtOffset(...)
   at System.IO.StreamWriter.Flush(...)
   at GitHub.Runner.Common.HostTraceListener.WriteHeader(...)
   at GitHub.Runner.Worker.Worker.RunAsync(...)

The runner's HostTraceListener writes diagnostic traces for every operation. When root fills up, the .NET runtime throws an unhandled exception and the runner process dies.

Visible log sizes don't explain 15GB consumption

For the crashed runtime-minimal-ubi9-python-3.12 · linux/amd64 [rhoai] job, total visible step log output:

system.txt                  1,337
1_Set up job.txt           40,401
14_Mount lvm overlay.txt    6,543
15_Install Podman.txt      16,074
19_Compute cache args.txt   4,702
... (total ~156KB)

156KB of visible logs can't explain 15GB of root consumption. The runner's internal _diag/Worker_*.log captures much more than what appears in the step logs — including internal API calls, file operations, and environment setup tracing.

The crash happens BEFORE build/prefetch

The job reaches step 19 (Compute CONTAINER_BUILD_CACHE_ARGS) and crashes before even starting the prefetch or build step. So it's not build output filling the disk — it's something during setup.

[jiridanek]

Correlation analysis — which builds crash?

Pattern: only [rhoai] hermetic builds crash

Build type	Includes [rhoai]	_diag crash?
PR builds (auto)	Yes (always runs both odh + rhoai)	Yes — multiple rhoai jobs crash
Push builds (auto, triggered by merge)	No (odh only)	No — only pytest flakes
Push builds (manual dispatch with rhoai=true)	Yes	Yes — same crash

What's different about [rhoai] builds?

The [rhoai] build path adds several steps that don't exist in [odh]:

1. git-crypt unlock — decrypts subscription secrets
2. Subscription setup — adds RHEL subscription credentials
3. --rhds prefetch variant — downloads from cdn.redhat.com (requires subscription auth, different RPM repos)

These steps produce additional runner tracing. The subscription setup in particular involves writing credentials to disk and configuring dnf/yum repos.

Timeline of failures on main

2026-05-14 17:34  RHAIENG-2488 base-images   failure  (build: codeserver, pytorch — NOT _diag)
2026-05-14 18:32  docs: base-images README    failure  (build: codeserver, pytorch — NOT _diag)
2026-05-14 19:38  hermeto version update      failure  (pytest, trustyai build — NOT _diag)
2026-05-15 03:08  scheduled push              failure  (NOT checked in detail)
2026-05-15 11:13  label unification #3629     failure  (pytest only — NOT _diag)
2026-05-15 12:30  byte-identical #3630        failure  (pytest only — NOT _diag)
2026-05-15 12:42  Tekton tags update #3632    failure  (pytest only — NOT _diag)
2026-05-15 16:47  pnpm+hermeto update #3635   failure  (s390x prefetch + pytest — NOT _diag)
2026-05-15 19:34  manual dispatch (rhoai=yes) failure  (_diag crash on ppc64le + amd64 rhoai)

The _diag crash only appears when [rhoai] builds run. All auto push builds (odh-only) fail on different, unrelated issues.

Hermeto version update — coincidence or cause?

PR #3554 updated the hermeto image version. The s390x prefetch failure on the pnpm+hermeto commit was a tornado wheel download failure from packages.redhat.com (CDN issue), not a disk issue. No evidence the hermeto update contributes to the _diag crash.

[jiridanek]

QEMU connection (possibly accidental)

Several of the crashed jobs are cross-architecture builds using QEMU user-mode emulation:

• jupyter-minimal-ubi9-python-3.12 · linux/s390x [rhoai]
• jupyter-minimal-ubi9-python-3.12 · linux/ppc64le [rhoai]

QEMU emulation is known to:

• Generate verbose debug output during instruction translation
• Produce core dumps on segfaults (written to root fs)
• Use significantly more temp space for emulated builds
• Occasionally crash with qemu: uncaught target signal 11 (multiarch/qemu-user-static#72)

However, the crash also affects native amd64 [rhoai] builds:

• runtime-minimal-ubi9-python-3.12 · linux/amd64 [rhoai]
• jupyter-minimal-ubi9-python-3.12 · linux/amd64 [rhoai]

So QEMU is not the sole cause — native builds crash too. The common factor is [rhoai], not the architecture. QEMU may exacerbate disk usage but isn't the root cause.

Related QEMU + disk space issues in the ecosystem

• Travis CI: s390x "no space left on device" — Docker layer extraction fills disk on s390x
• pmem/run_qemu#235 — QEMU project hit identical _diag/Worker_*.log crash on GHA, Dec 2025. Resolved by runner image update.
• multiarch/qemu-user-static#110 — go install fails inside s390x container on x86_64 host via QEMU

[jiridanek]

Related issues and prior art

Identical crash in other projects

Project	Issue	Date	Resolution
actions/runner-images	#13719	Feb 2026	Runner provisioned with full disk — infra issue, no user fix
GitHub Community	#133486	Nov 2025	Same stack trace, went dormant
pmem/run_qemu	#235	Dec 2025	Resolved by runner image update
docker/build-push-action	#968	Sep 2023	QEMU + buildx, disk space
jiridanek/rhoai-in-kind	#10	May 2025	Same GHA disk exhaustion, fix: rm -rf /mnt/* (PR #34)

GHA runner disk space resources

• actions/runner-images#2875 — General "no space left on device" discussion
• GitHub Community #25678 — Disk space cleanup strategies
• hyperenv.com blog — Comprehensive cleanup guide
• eBPF in CI/CD: GitHub Actions — Confirms basic eBPF tracepoints work on GHA with sudo
• Ken Muse: Creating GitHub Checks — Checks API for reporting results

Our notebook repo context

• Our gha_lvm_overlay.sh already does aggressive cleanup (removes dotnet, android SDK, etc.) and creates a 72GB LVM volume for container storage
• The free-up-disk-space action reclaims ~2GB of additional packages
• Despite this, root (/dev/root) ends up at 80% used with only 15GB free
• The _diag directory is on root, not on the LVM volume

Debug mode investigation

ACTIONS_RUNNER_DEBUG and ACTIONS_STEP_DEBUG are not set as repo variables or secrets. Debug mode is not accidentally enabled. The _diag log grows from normal runner tracing, not debug verbosity.

[jiridanek]

Diagnostic monitoring deployed

PR #3633 commit 9d99f4c adds comprehensive monitoring to build-notebooks-TEMPLATE.yaml:

Background monitors (started after LVM setup)

1. Disk monitor — every 30s logs:

   • df -h / /mnt
   • _diag directory size (du -sh)
   • Top 5 space consumers under /home/runner (du -xh --max-depth=2)
   • Uses ::group:: markers for collapsible output in GHA logs

2. fatrace — traces file writes on root fs in real time (cumulative summary every 30s). Shows which processes are writing and how much.

3. bpftrace — sums bytes written per process every 30s via eBPF tracepoint (syscalls:sys_exit_write). May not work if linux-headers-$(uname -r) doesn't match.

Inline tailing

Each long-running step (prefetch, build) runs tail -f /tmp/{disk-monitor,fatrace,bpftrace}.log & so output appears in that step's log. This is critical because when the runner crashes with _diag disk-full, subsequent steps never execute — only output already written to the current step's log survives.

Expected output

When the next CI run hits the crash, we should see in the prefetch/build step log:

• Exactly when root fills up (30s granularity)
• Which directory/files are growing
• Which process is doing the writing (fatrace/bpftrace)

This will tell us whether to:

• Move _diag to LVM
• Leave more root space in LVM allocation
• Reduce whatever is writing to root

[jiridanek]

Root cause found: prefetch writes ~3GB to root, only 4GB free

LVM leaves only 4GB on root

gha_lvm_overlay.sh line 11: root_reserve_mb=4096

The script takes all root free space minus 4GB for the LVM volume:

Root before LVM:  15GB free
LVM takes:        ~11GB from root (via /pv.img fallocate)
Root after LVM:   ~4GB free   ← this is the actual free space

The df -h output showing "15G free" in the issue description is from before the LVM fallocate runs. After LVM setup, root has only ~4GB.

Prefetch writes ~3GB to root

cachi2/output/ is created in the git checkout directory at /home/runner/work/notebooks/notebooks/ which is on /dev/root, not on the LVM volume (/home/runner/.local/share/containers).

Measured prefetch sizes for jupyter-minimal (one of the smaller images):

68M   cachi2/output/deps/generic/
175M  cachi2/output/deps/pip/
2.9G  cachi2/output/deps/rpm/

So: 4GB free - 3GB prefetch ≈ 1GB remaining for _diag, go cache, pip cache, uv cache, and everything else on root. The _diag/Worker_*.log fills the last ~1GB and the runner crashes.

Why [odh] builds don't crash

ODH builds either:

• Don't use hermetic prefetch (the prefetch step is skipped)
• Or use smaller lockfiles (no subscription RPMs from cdn.redhat.com)

Fix options

1. Increase root_reserve_mb from 4096 to ~8192 — simple but reduces build volume by 4GB
2. Move cachi2/output/ to the LVM volume — symlink or bind-mount before prefetch
3. Move _diag to LVM — symlink the runner's _diag directory to the build volume
4. Both: increase root reserve AND move prefetch output to LVM

Option 2 is cleanest — the prefetch deps are consumed by podman build --volume anyway, so having them on the build volume is natural.

[jiridanek]

Monitoring script for reuse

Added in PR #3633 commit 9d99f4c. Can be dropped into any GHA workflow step for disk diagnostics.

Background monitor setup (add as a step after disk/LVM setup)

- name: Start root-fs disk monitor (background)
  run: |
    set +x
    DIAG_DIR=$(find /home/runner -path '*/_diag' -type d 2>/dev/null | head -1)
    echo "Runner _diag directory: ${DIAG_DIR:-not found}"

    sudo apt-get install -y -qq fatrace 2>/dev/null || true
    sudo apt-get install -y -qq bpfcc-tools linux-headers-"$(uname -r)" 2>/dev/null || true

    # Disk monitor: df, _diag size, top space consumers — every 30s
    cat > /tmp/disk-monitor.sh << 'MONITOR'
    #!/bin/bash
    DIAG_DIR="${1:-}"
    while true; do
      echo "::group::disk-monitor $(date -u '+%H:%M:%S')"
      df -h / /mnt 2>/dev/null | grep -v tmpfs
      if [ -n "$DIAG_DIR" ]; then
        echo "_diag: $(du -sh "$DIAG_DIR" 2>/dev/null | cut -f1)"
        ls -lhS "$DIAG_DIR"/*.log 2>/dev/null | head -3
      fi
      du -xh --max-depth=2 /home/runner 2>/dev/null | sort -rh | head -5
      echo "::endgroup::"
      sleep 30
    done
    MONITOR
    chmod +x /tmp/disk-monitor.sh
    /tmp/disk-monitor.sh "$DIAG_DIR" >> /tmp/disk-monitor.log 2>&1 &
    echo $! > /tmp/disk-monitor.pid

    # fatrace: cumulative write summary every 30s
    if command -v fatrace &>/dev/null; then
      (while true; do
        echo "::group::fatrace top writers $(date -u '+%H:%M:%S')"
        sudo timeout 30 fatrace -f W -t -c 2>/dev/null || true
        echo "::endgroup::"
      done) >> /tmp/fatrace.log 2>&1 &
      echo $! > /tmp/fatrace.pid
      echo "fatrace started"
    fi

    # bpftrace: sum bytes written per process every 30s
    if command -v bpftrace &>/dev/null; then
      sudo bpftrace -e '
        tracepoint:syscalls:sys_exit_write /args.ret > 0/ {
          @bytes[comm] = sum(args.ret);
        }
        interval:s:30 {
          printf("::group::bpftrace write bytes\n");
          print(@bytes);
          clear(@bytes);
          printf("::endgroup::\n");
        }
      ' >> /tmp/bpftrace.log 2>&1 &
      echo $! > /tmp/bpftrace.pid
      echo "bpftrace started"
    fi

Inline tailing (add at the top of any step that might crash)

# Tail disk monitors inline — output visible even if runner crashes
for f in /tmp/disk-monitor.log /tmp/fatrace.log /tmp/bpftrace.log; do
  [ -f "$f" ] && tail -f "$f" &
done

Stop monitors (add as a cleanup step with if: !cancelled())

- name: Stop disk monitors
  if: "${{ !cancelled() }}"
  run: |
    for pidfile in /tmp/disk-monitor.pid /tmp/fatrace.pid /tmp/bpftrace.pid; do
      [ -f "$pidfile" ] && kill "$(cat "$pidfile")" 2>/dev/null || true
    done
    df -h / /mnt "${HOME}/.local/share/containers" 2>/dev/null || df -h
    DIAG_DIR=$(find /home/runner -path '*/_diag' -type d 2>/dev/null | head -1)
    [ -n "$DIAG_DIR" ] && echo "_diag final: $(du -sh "$DIAG_DIR" 2>/dev/null | cut -f1)"

First results (from successful [odh] amd64 build, PR #3633 run 25938039621)

/dev/root       145G   36G  109G  25% /
_diag: 972K
-rw-r--r-- 1 runner runner 478K  Worker_*.log
6.1G  /home/runner
2.1G  /home/runner/work

This was a single-disk runner (root = 145G, no /mnt), so the LVM script skipped LVM entirely and root had 109GB free. The _diag stayed tiny (972K). The crash only happens on dual-disk runners where gha_lvm_overlay.sh takes ~11GB from root, leaving only ~4GB.

[jiridanek]

Key finding: single-disk vs dual-disk runners

GHA provides two runner configurations:

• Single-disk: /dev/root is 145GB, no /mnt — LVM script skips, everything on root
• Dual-disk: /dev/root is 72GB (58GB used = 15GB free), /dev/sdb1 at /mnt is 74GB

Our gha_lvm_overlay.sh detects this and only creates LVM on dual-disk runners. On dual-disk:

root_reserve_mb=4096   # only 4GB kept on root!

Root starts with 15GB free, LVM takes ~11GB → 4GB left on root.

Then cachi2/output/ (hermetic prefetch deps) writes ~3GB to root (it's in the git checkout at /home/runner/work/notebooks/notebooks/).

4GB - 3GB = 1GB for _diag + go cache + uv cache + pip cache + everything else on root → crash.

On single-disk runners, there's 109GB free — no problem.

Corrected root cause summary

The issue is the combination of:

1. Dual-disk runner allocation leaving only 4GB on root
2. Hermetic prefetch writing ~3GB of deps to root (not to the LVM build volume)
3. Runner's _diag log, caches, and other root-fs consumers filling the remaining ~1GB

[jiridanek]

Disk timeline from successful [rhoai] build (run 25935304977, job 76239410552)

This is cuda-jupyter-tensorflow-ubi9-python-3.12 · linux/amd64 [rhoai] — a successful rhoai hermetic build with our diagnostic monitor.

Timeline

18:49:46  /dev/root  72G  61G  12G  84%  ← raw state, before cleanup
18:52:48  /dev/root  72G  36G  37G  50%  ← after free-up-disk-space + LVM (single-disk: LVM skipped)
18:53:06  mount --bind containers/tmpdir → /var/tmp
18:53:16  go build bin/buildinputs
18:53:23  uv sync (notebooks project install)
18:53:35  prefetch starts (jupyter/tensorflow cuda)
18:59:11  subscription cert setup for RPM downloads
19:07:02  /dev/root  72G  54G  19G  75%  ← after build complete

Key observation

This was a single-disk runner (72G root, but LVM was skipped — Skipping compsize: not a btrfs mountpoint). So root had 37GB free after cleanup, plenty of room. The build consumed ~18GB on root (from 37G → 19G free):

• cachi2/output/deps/ — ~3GB (RPMs, pip wheels, generic artifacts)
• Container image layers during podman build — ~15GB (CUDA tensorflow is huge)

Why dual-disk runners crash

On dual-disk runners, gha_lvm_overlay.sh creates LVM and takes ~11GB from root:

root_reserve_mb=4096   →  only 4GB kept on root

After cleanup on dual-disk: root has ~4GB + whatever free-up-disk-space frees.
But the build needs ~18GB on root (prefetch + build layers).

4GB vs 18GB needed — the runner crashes immediately during prefetch.

The build needs ~18GB on root even with LVM

Even though container storage is mounted on LVM (/home/runner/.local/share/containers), the build still writes heavily to root:

1. cachi2/output/ — in git checkout on root (~3GB)
2. podman build intermediate layers — goes through container storage (LVM) BUT the Dockerfile COPY operations first read from root
3. Go module cache at /home/runner/go/pkg/mod and /home/runner/.cache/go-build — on root
4. uv cache and venv — on root
5. Git submodule checkout (code-server + vscode) — ~1GB on root

[jiridanek]

UPDATED ROOT CAUSE: sandbox.py duplicates the entire source tree including cachi2/output/

From the failed cuda-jupyter-tensorflow [rhoai] build log (run 25935304977, job 76239410552):

Error: [('/source/cachi2/output/deps/pip/tensorflow-2.21.0+redhat-9-cp312-cp312-linux_x86_64.whl',
  '/tmpye80dfoo.hermeto-source-copy/cachi2/output/deps/pip/tensorflow-2.21.0+redhat-9-cp312-cp312-linux_x86_64.whl',
  "[Errno 28] No space left on device")]

sandbox.py copies the entire git checkout — including cachi2/output/ (~3GB+ of prefetched deps) — into a temporary directory (/tmp*hermeto-source-copy/). This doubles the disk usage:

cachi2/output/ on root:     ~3GB (prefetched deps)
sandbox.py tmp copy:        ~3GB (duplicate)
source tree copy:           ~1GB
go cache, uv cache, etc:    ~1GB
                            -------
Total on root:              ~8GB+

On a dual-disk runner with 4GB free after LVM, or even a single-disk with 19GB free after prefetch of a large image like tensorflow-cuda, this overflows.

The fix

sandbox.py's setup_sandbox() function should exclude cachi2/ from the source copy. The cachi2/output/ directory is volume-mounted into the container anyway (--volume cachi2/output:/cachi2/output:Z), so it doesn't need to be in the build context copy.

Alternatively, sandbox.py could use the original source directory in place and only copy the minimal set of files that buildinputs identifies as prerequisites.

[jiridanek]

CORRECTION: It's hermeto doing the copy, not sandbox.py

The /source/ → /tmp*hermeto-source-copy/ paths are inside the hermeto container during the prefetch step, not in sandbox.py during the build step. The job never got to the build step.

What happens during prefetch

prefetch-all.sh runs multiple steps sequentially:

1. Generic artifacts → cachi2/output/deps/generic/ (~68MB)
2. Pip wheels → cachi2/output/deps/pip/ (~175MB for minimal, much more for tensorflow-cuda)
3. RPMs via hermeto → cachi2/output/deps/rpm/ (~2.9GB)
4. Go modules → cachi2/output/deps/gomod/

The hermeto container (step 3) copies the entire source tree into its working directory (/tmp*hermeto-source-copy/). By step 3, cachi2/output/deps/pip/ already has the pip wheels from step 2. For cuda-jupyter-tensorflow, this includes tensorflow-2.21.0+redhat-9-*.whl and hundreds of other wheels.

So hermeto duplicates all of:

• The source code (~1GB with submodules)
• The pip wheels already downloaded in step 2
• The generic artifacts from step 1

The fix should target how we invoke hermeto

Either:

1. Exclude cachi2/ from hermeto's source mount — hermeto only needs rpms.lock.yaml, not the pip wheels
2. Move cachi2/output/ to LVM before hermeto runs — so the duplicate doesn't consume root space
3. Run hermeto first (before pip) — but this changes the step order

Option 1 is cleanest. The hermeto invocation in prefetch-all.sh likely mounts the entire repo as /source. It should mount only the lockfile or exclude cachi2/.

[jiridanek]

FINAL ROOT CAUSE: hermeto Go-modules step copies entire source including prior prefetch output

From the failed build log:

Sequence of events in prefetch-all.sh

[1/5] Generic artifacts  → cachi2/output/deps/generic/  (~68MB)   ✅
[2/5] Pip wheels          → cachi2/output/deps/pip/      (~2GB for tensorflow-cuda)  ✅
[3/5] NPM (skipped)
[4/5] RPMs via hermeto    → cachi2/output/deps/rpm/      (~3GB)   ✅  ← hermeto copies /source but cachi2/output only has generic+pip here
[5/5] Go modules via hermeto                                       ❌  ← hermeto copies /source AGAIN, now cachi2/output has generic+pip+rpm (~5GB)

At step 5, hermeto runs resolve_packages() which calls copy_directory(original_source_dir, ...) — copying everything under /source/ into /tmp*hermeto-source-copy/. By this point, /source/cachi2/output/ already has ~5GB of previously downloaded deps.

The hermeto source copy code (hermeto/core/resolver.py:48):

with TemporaryDirectory(f".{APP_NAME}-source-copy", dir=".") as temp_dir:
    source_backup = copy_directory(original_source_dir.path, Path(temp_dir))

It copies the entire source tree with no exclusions. On a single-disk runner with 37GB free, this means: source (~1GB) + pip (~2GB) + RPMs (~3GB) + hermeto copy of all that (~6GB) = ~12GB consumed, leaving ~25GB — tight but workable. For tensorflow-cuda, the pip deps alone are much larger, pushing it over.

The error

hermeto/core/utils.py:198 → shutil.copytree(/source/, /tmp*hermeto-source-copy/)
  [Errno 28] No space left on device: tensorflow-2.21.0+redhat-9-cp312-cp312-linux_x86_64.whl

Also note the EXDEV (cross-device link) fallback — hermeto first tries os.copy_file_range() which fails with EXDEV because /source/ and /tmp*/ may be on different filesystems (container overlay). Then it falls back to shutil.copy2() which does a full copy — doubling disk usage.

Fix options

1. Reorder steps: Run Go modules (step 5) before pip (step 2) and RPMs (step 4) — so cachi2/output/ is empty when hermeto copies
2. Stash cachi2/output/: mv cachi2/output /tmp/cachi2-stash before hermeto Go, restore after — prevents the copy from including prior deps
3. Upstream fix: File hermeto issue to support .hermetoignore or --exclude for the source copy
4. Mount cachi2/output/ as a separate volume in the hermeto container invocation — excluded from the source mount

Option 1 is the simplest immediate fix with no hermeto changes needed. Option 2 is a workaround. Option 3 is the proper upstream fix.