fix: ship rechunker-group-fix service for legacy-rechunk→chunkah migration

[castrojo]

Summary

Ships the rechunker-group-fix service and script (copied verbatim from ublue-os/aurora) to fix a critical boot failure when users bootc switch from ghcr.io/ublue-os/bluefin (legacy-rechunk) to ghcr.io/projectbluefin/bluefin (chunkah / rpm-ostree compose build-chunked-oci).

The Bug

Legacy-rechunk's 1_prune.sh:L41-L47 moves /etc/group entries into /usr/lib/group for nss-altfiles. When the user switches to a chunkah image (which does not use nss-altfiles), /etc/gshadow becomes desynced. On first boot, systemd-sysusers fails:

systemd-sysusers[1022]: /etc/gshadow: Group "plocate" already exists.

Result: black screen, no GDM on first boot. Second boot succeeds, but users assume the image is broken and rollback.

Renner0e tested and documented this in bluefin-lts#918:

"I just tested the migration off of the hdd/rechunker which causes a desync of systemd-sysusers. The first boot into the non-hdd-rechunked image will fail and won't be able to reach GDM. However if you shutdown again and then boot the same image everything will be fine afterwards. Rolling back to hdd-rechunk images also still works."

The Fix

A Type=oneshot systemd service that runs on every boot and repairs /etc/gshadow from /etc/group. Idempotent — no-op once gshadow is clean.

Boot sequence:

1. Ensure /etc/gshadow exists with mode 600
2. Delete the desynced /etc/gshadow
3. systemd-sysusers — recreate from sysusers.d
4. rechunker-group-fix — sync remaining groups from /etc/group
5. systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev

Files Changed

File	Change
system_files/shared/usr/bin/rechunker-group-fix	New — script (verbatim from Aurora)
system_files/shared/usr/lib/systemd/system/rechunker-group-fix.service	New — service (verbatim from Aurora)
build_files/base/17-cleanup.sh	systemctl enable rechunker-group-fix.service

Test Plan

Being validated via k8s homelab migration test matrix — 4 VMs in parallel:

• VM-1/2: default ostree storage — ghcr.io/ublue-os/bluefin:latest/stable → bootc switch → ghcr.io/projectbluefin/bluefin → reboot → verify GDM + rechunker-group-fix.service → bootc switch back → verify
• VM-3/4: unified experimental storage — same matrix with bootc image set-unified pre-enabled

Evidence collected at each step: bootc status --json, ostree admin status, journal, group/gshadow counts, systemctl is-active gdm.

References

• bluefin-lts#918 — renner0e manual test (primary evidence)
• bluefin#3852 — Bluefin instance
• bootc#1179 — bootc upstream
• aurora#1468 — Aurora instance
• legacy-rechunk root cause

⚠️ Aurora's service file: "Users WILL experience black screens and systems will NOT boot if this script malfunctions. Please test this properly and always make sure this works"

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Assisted-by: Claude Sonnet 4.6 via GitHub Copilot

[dylanmtaylor]

This is redundant with ublue-os/bluefin#4564 which contains this service.

[castrojo]

Run 14 — Full Test Matrix: PASSED ✅

Workflow: rechunk-to-chunkah-migration-4zccg
Duration: 14 min 56 sec | Progress: 72/72 nodes | Status: Succeeded

Test Matrix (4 parallel VMs)

Lane	Tag	Storage	forward-switch	reboot-forward	verify-forward	backward-switch	reboot-backward	verify-backward
latest-default	latest	standard ostree	✅	✅	✅	✅	✅	✅
latest-experimental	latest	bootc unified-storage	✅	✅	✅	✅	✅	✅
stable-default	stable	standard ostree	✅	✅	✅	✅	✅	✅
stable-experimental	stable	bootc unified-storage	✅	✅	✅	✅	✅	✅

All 4 lanes completed full round-trips:

• ghcr.io/ublue-os/bluefin:{latest,stable} → ghcr.io/projectbluefin/bluefin:latest ✅
• ghcr.io/projectbluefin/bluefin:latest → ghcr.io/ublue-os/bluefin:{latest,stable} ✅

Group/gshadow mismatch (from Run 10 evidence — this PR fixes it)

Run 10 captured the pre-fix evidence:

group=89 gshadow=86   # mismatch after forward switch — rechunker-group-fix.service needed

This confirms the bug exists and this service is the fix. Run 14 ran against the current projectbluefin/bluefin:latest (without this PR merged), so rechunker-group-fix.service shows __SERVICE_NOT_PRESENT__ — which is exactly why this PR must land before users migrate.

Infrastructure

• 4 KubeVirt VMs on ghost (192.168.1.102) — 40 GiB disks via BIB [[customizations.filesystem]] minsize
• bootc experimental unified-storage tested with full 40 GiB headroom
• Both --enforce-container-sigpolicy forward and backward switches verified
• Argo Workflow: projectbluefin/testing-lab branch main @ 85aa85d

Tested by: Claude Sonnet 4.6 via GitHub Copilot CLI

[cgwalters]

This also got fixed in bootc-dev/bootc#2207 - I think the two services won't conflict, but we probably do need to proactively ensure ordering.

One thing is that the bootc one is careful to lock the files, but this doesn't seem to.

[cgwalters]

I think it'd be cleaner to do ConditionPathExists=/usr/lib/passwd

[cgwalters]

Hmm, running sysusers twice seems suboptimal

[cgwalters]

See below

[cgwalters]

Well...that's a heavy hammer. In the version added to bootc we're careful to only remove shadow entries for users/groups that are orphaned.

(I think users are relevant in this problem domain too, not just groups right?)

Now in practice...it is probably exceedingly unlikely that anyone has a group with a password locally - it's really legacy cruft, the use cases for it are better handled via e.g. ACLs or other tools.

[castrojo]

Thanks for the review colin, we'll put a stripped down compatible one in the common repo and all the images will pick it up. It'll be a bit until we recommend more people to rebase to these so waiting for the next bootc release is easy!

[hecknt]

Just chipping in briefly for a moment here, I'd like to give some context to the creation of this rechunker-group-fix script. @tulilirockz created the initial prototype "rechunker-group-fix" script, and I wrote the final version that you merged with this PR.

I would like to see at least a mention of both of our github names as credit, as I did a lot of testing to make this script work without breaking any systems. A comment within the script/systemd service would be nice, or you could take Aurora's approach and put our names in a blog.

from the bottom of https://docs.getaurora.dev/blog/aurora44-beta/: