rechunk recipe (Chunkah) crashes with "Argument list too long" on base images chunked with the older rechunker (e.g. Bluefin)

[Danathar]

I've been making my own little utility to help people create their own images (Atomic Image Builder). My utility bundles the latest version of this template into its code for use. When I tried to use Chunkah, it failed for a rather interesting reason.

Apologies if I got any of this wrong. I had to ask Claude a bunch of questions about WHY this occurred, so hopefully it didn't take me down a mirage rabbit hole! Some of the investigation rests on answers it gave me. A lot of this is me talking, and some of it is Claude, edited by me. All that being said, the fix worked when I patched it on my side in my utility.

The rechunk recipe in the bundled Justfile (Justfile#L129-L147) fails immediately, before Chunkah even starts, when run against a base image that was chunked upstream with the older rechunker. The error:

podman: Argument list too long
error: recipe `rechunk` failed with exit code 126

My utility tried to build on Bluefin GNOME, which is chunked upstream with the older hhd-dev/rechunk tool. That tool writes a huge package-manifest label onto the image: a blob listing every installed package and its version. When I tested my utility against Silverblue and Kinoite, it didn't fail; so they don't carry that label. I haven't tried it against Aurora yet, but I'd bet it would work, because Aurora switched to Chunkah for its own builds, and Chunkah doesn't work the same way. It doesn't leave that metadata behind.

In the Justfile:

export CHUNKAH_CONFIG_STR=$(podman inspect "${target_image}")
podman run --rm --mount=type=image,src="${target_image}",target=/chunkah \
-e CHUNKAH_CONFIG_STR quay.io/coreos/chunkah:latest \
...

The podman inspect and the variable assignment both succeed; bash has no problem holding it. The problem is the next line: when podman run -e CHUNKAH_CONFIG_STR ... tries to spawn the podman process, the kernel has to copy the whole environment into the new process, and no single environment variable can exceed 128 KiB. When the target image is Bluefin, podman inspect against it produces ~307 KiB of output, so CHUNKAH_CONFIG_STR ends up that big too (mostly that package-manifest label), and the kernel refuses to create the process at all. podman never gets to run, which is why the error comes from bash.

Claude recommended passing the data via a file instead of an environment variable, which avoids the problem entirely:

Chunkah documents a second, equivalent way to provide the exact same data: a file, via --config <path>, instead of an environment variable (README, "Customizing the OCI image config and annotations"). This patch would be for the rechunk recipe in the Justfile:

-    export CHUNKAH_CONFIG_STR=$(podman inspect "${target_image}")
-    podman run --rm --mount=type=image,src="${target_image}",target=/chunkah \
-    -e CHUNKAH_CONFIG_STR quay.io/coreos/chunkah:latest \
+    CHUNKAH_CONFIG_FILE=$(mktemp)
+    podman inspect "${target_image}" > "${CHUNKAH_CONFIG_FILE}"
+    podman run --rm --mount=type=image,src="${target_image}",target=/chunkah \
+    -v "${CHUNKAH_CONFIG_FILE}:/chunkah-config.json:ro,Z" quay.io/coreos/chunkah:latest \
     build \
     --verbose \
     --compressed \
     --max-layers 128 \
     --prune /sysroot/ \
     --label ostree.commit- --label ostree.final-diffid- \
+    --config /chunkah-config.json \
     --tag "${target_image}:${tag}" | podman load
+    rm -f "${CHUNKAH_CONFIG_FILE}"

Apologies for the long explanation, I do that to explain it to myself as well! ;) Put simply: that variable can't hold the manifest, so you need to pass that info via a file.

The fix was verified when I used it: the unpatched recipe reproduces the exact error against a real ghcr.io/ublue-os/bluefin:stable pull, and the patched one completes a full Chunkah build.

About the Z in the mount: the relabel flag is needed when running locally on SELinux-enforcing hosts (any Fedora/Atomic machine). Since the Justfile is also the local-build path, that covers a lot of this template's audience. On the Ubuntu GitHub runners it's not needed (AppArmour instead of SELinux?), so it's harmless to include everywhere probably.

If you agree with this fix, I can submit a PR for it.

[renner0e]

That would be fine. It is definitely a weird edge case. Although I think the rm command should cleanup the temporary file no matter if chunkah failed or not.

To avoid this you can use trap shell built-in.

The selinux thing seems fine as well for me, it doesn't seem to blow anything up.

[Danathar]

That would be fine. It is definitely a weird edge case. Although I think the rm command should cleanup the temporary file no matter if chunkah failed or not.

To avoid this you can use trap shell built-in.

The selinux thing seems fine as well for me, it doesn't seem to blow anything up.

Thanks! I'll look into that and integrate your suggestions in the patch. Honestly, I was debating if I even should file this bug, if @castrojo gets around to using Chunkah on Bluefin instead of the old chunker, this issue would go away probably, but I figured since somebody may want to use this template with Bluefin and enable chunkah, it may be a good idea to do it anyway.

@castrojo Do you plan on switching Bluefin's build to Chunkah anytime soon? Just an FYI: the image-template repo's Chunkah rechunk step currently breaks if you enable and use Bluefin as the base image, since Bluefin is still built with legacy-rechunk. If its soon I can just hold off on this patch as it only affects people who enable chunkah here.

[benblasco]

I have also put a fix together with the help of Cursor, and will submit it as a PR shortly. It will be interesting to compare the two, as I dealt with multiple issues along the way and documented them all. Much of the text below was written by cursor, but I have made edits and added commentary where needed.

Background

The stock rechunk recipe followed chunkah's documented pattern: export podman inspect output as CHUNKAH_CONFIG_STR, pass it to chunkah via -e, and pipe the OCI archive to podman load.

This works for smaller base images (e.g. plain fedora-bootc), but fails for Universal Blue images like Bluefin-DX, which have large pre-chunk

Problems encountered and fixes

1. Argument list too long (exit 126)

Symptom:

/run/user/1001/just/.../rechunk: line 139: /usr/bin/podman: Argument list too long
error: recipe `rechunk` failed with exit code 126

Cause: The full podman inspect JSON was exported as CHUNKAH_CONFIG_STR. For Bluefin-DX-based images this exceeds Linux's 128 KiB per-environment-variable limit (MAX_ARG_STRLEN). The host shell could not even exec podman.

Fix: Write inspect output to a temp file and pass it to chunkah via --config instead of an environment variable as per the solution described by @Danathar

2. No space left on device during podman load (exit 125)

Symptom:

oci-archive: ... processing tar file(write /usr/share/homebrew.tar.zst: no space left on device)
error: recipe `rechunk` failed with exit code 125

Cause: Piping chunkah's OCI archive to podman load required the runner to hold:

• The original built image in podman storage
• A temp OCI archive in /var/tmp
• The new rechunked image being extracted

For a full Bluefin-DX image this can exceed available GitHub runner disk (~30 GB after cleanup).

Fix: Avoid the | podman load pipe entirely:

1. Write OCI directory layout with --output oci:PATH (no tar roundtrip)
2. Delete the original image with podman rmi to free storage
3. Import the rechunked image with a skopeo container. I had the option of installing skopeo but used the same pattern as chunkah, which is running in a container too.

3. Output path already exists (exit 1)

Symptom:

DEBUG starting chunkah version="0.6.0"
Error: output path already exists: /output

Cause: The temp output directory was bind-mounted directly at /output, then chunkah was told to write to --output oci:/output. chunkah requires the output path to not already exist.

Fix: Mount a parent directory and write to a child path that chunkah creates, per chunkah output options docs:

• Mount ${CHUNKAH_OUTPUT_DIR} at /run/out
• Use --output oci:/run/out/chunked
• skopeo reads from ${CHUNKAH_OUTPUT_DIR}/chunked

All looks straightforward to me.

4. skopeo permission denied on overlay remount (exit 1)

Symptom:

level=fatal msg="Invalid destination name containers-storage:bluefin-template:latest:
  overlay: failed to make mount private: remount /var/lib/containers/storage/overlay,
  flags: 0x40000: permission denied"

Cause: The skopeo container was writing to the host's root podman storage at /var/lib/containers without sufficient privileges to manipulate overlay filesystem mounts.

Fix: Add --privileged to the skopeo podman run, matching the existing ostree-rechunk recipe in the same Justfile.

I don't know of another way to give the container the appropriate access than to give --privileged access

[renner0e]

oci-archive: ... processing tar file(write /usr/share/homebrew.tar.zst: no space left on device)
error: recipe rechunk failed with exit code 125

@benblasco can you link your action run that triggered this? Github's disk configuration varies from 72G with 20G free to 145G with 90GB free.

[benblasco]

@renner0e I have only just had an opportunity to look at this now. Do you still want me to find the run and send it to you? I am building in a private repo.