fix(build): resolve podman at /usr/local/bin - #4880
Conversation
GitHub's ubuntu-24.04 runner image 20260804.265 replaced the apt podman package with a static bundle installed to /usr/local/bin/podman. The Justfile only probed /usr/bin/podman, so it silently fell back to docker. Docker's overlay2 driver caps stacked lower dirs at 128, and silverblue-main:44 has 259 layers, so every Latest build died with "failed to register layer: max depth exceeded" before running a single build script. Podman has no such limit and builds the same base fine. Probe /usr/local/bin/podman before falling back to docker. Existing resolution order and the PODMAN env override are unchanged. Assisted-by: Claude Opus 5 via GitHub Copilot CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dosubot
Bot
added
size:XS
Up to standards ✅🟢 Issues
|
There was a problem hiding this comment.
Pull request overview
Updates Bluefin’s build orchestration (Justfile) to correctly prefer Podman on newer GitHub-hosted runners where Podman is shipped as a static binary under /usr/local/bin, preventing unintended fallback to Docker (and avoiding Docker’s overlay2 layer-depth limitation when building Fedora 44–based images).
Changes:
- Extend the Podman detection logic to probe
/usr/local/bin/podmanbefore falling back to Docker.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export SUDOIF := if `id -u` == "0" { "" } else { "sudo" } | ||
| export PODMAN := if path_exists("/usr/bin/podman") == "true" { env("PODMAN", "/usr/bin/podman") } else if path_exists("/usr/bin/docker") == "true" { env("PODMAN", "docker") } else { env("PODMAN", "exit 1 ; ") } | ||
| export PODMAN := if path_exists("/usr/bin/podman") == "true" { env("PODMAN", "/usr/bin/podman") } else if path_exists("/usr/local/bin/podman") == "true" { env("PODMAN", "/usr/local/bin/podman") } else if path_exists("/usr/bin/docker") == "true" { env("PODMAN", "docker") } else { env("PODMAN", "exit 1 ; ") } | ||
| export PULL_POLICY := if PODMAN =~ "docker" { "missing" } else { "newer" } |
|
Superseded by #4881, which moved builds to ubuntu-26.04 where podman is installed from apt at /usr/bin/podman. The /usr/local/bin probe was only needed for the ubuntu-24.04 static bundle layout, so this is now speculative. Closing as redundant. |
Problem
Every
Latest Imagesbuild has been failing:Root cause
GitHub's
ubuntu-24.04runner image bumped from20260720.247.2to20260804.265.1. Perinstall-container-tools.sh, podman is no longer installed via apt — it now ships as a static bundle at/usr/local/bin/podman(4.9.3 → 5.8.4). Onlybuildah,skopeoanduidmapstill come from apt, and none of them pull in podman, so/usr/bin/podmanno longer exists.The Justfile probed
/usr/bin/podmanonly, so it silently fell through todocker.Docker's overlay2 driver caps stacked lower dirs at 128. Layer counts:
silverblue-main:43silverblue-main:44F43 squeaked under the cap, so the fallback went unnoticed. F44 does not, and the build dies at the
FROMline before running a single build script.Evidence that podman is fine with the identical base — the passing Stable job on the older runner:
vs. the failing Latest job on the new runner:
Same base image, same recipe — only the builder differs.
Fix
Probe
/usr/local/bin/podmanbefore falling back to docker. Line 19 of the Justfile is the only place in the repo hardcoding these paths.Verification
Exercised every branch against a sandboxed path tree with
just --evaluate:PULL_POLICY/usr/bin(old runner)/usr/bin/podmannewer/usr/local/bin(new runner)/usr/local/bin/podmannewerdockermissingexit 1 ;PODMAN=/custom/podman/custom/podmanOnly the second row changes behavior; everything else is byte-identical to before.
just checkandpre-commitpass.Notes
Not adopting aurora's buildah + BTRFS-loopback pipeline here — that repo is green on the same 257-layer F44 base, but its approach changes storage, caching and build semantics. Bluefin's existing podman build handles the base fine once it's actually found.
Follow-ups worth considering separately (all pre-existing):
max depth exceededan hour later.PULL_POLICY := if PODMAN =~ "docker"is a substring test; a path like/opt/docker-tools/podmanwould be misclassified.Assisted-by: Claude Opus 5 via GitHub Copilot CLI