feat(install): all-plain layout + self-sizing first boot, signed origin, auto-updates

[dx4homelab]

Scope widened. This started as the partition-layout fix and now also makes the raw image a complete starting point for seeding similar servers, not just repairing one host. Added to this PR rather than stacked behind it: the new iso/disk.toml change builds directly on this PR's content, and a stacked PR already cost a mis-merge once tonight (#4 was auto-closed when its base branch was deleted on merging #3).

Part 2 — self-sizing first boot, signed origin, auto-updates on

server4home-firstboot-grow.service

Grows root into the real drive on first boot, then hands the remainder to an LVM PV + empty vg4data.

Root stays 20 GiB in the image on purpose. The raw is sparse (~5 GB actual for a 22 GiB apparent file) but dd writes the apparent size, so a baked 250 GiB root would push a quarter-terabyte of mostly zeros onto every install. Growing on the hardware is also the only way one image fits any boot drive size.

Size comes from /etc/server4home/root-size (default 250G; the literal max = whole disk, no VG). Too-small drive, or leftover under 50 GiB → root takes everything and no VG is created.

It does the step that is easy to miss and silently fatal: sgdisk -e. A small image on a big drive leaves the GPT describing the image's size, so growpart is a no-op until the backup header is relocated. That is exactly the state z170asrock was found in:

last usable sector is 44455902      <-- ~21 GB, on a 4 TB drive
Total free space is 0 sectors

Root's own partition entry is never hand-rewritten — the unit creates the app-data partition where root should stop, bounding the gap, then lets growpart fill it exactly.

Verified against z170asrock's real geometry:

target	root result	aligned	VG gets
250G	exactly 250G	yes	3.4 T
512G	exactly 512G	yes	3.2 T
1T	exactly 1.0T	yes	2.7 T
8T	too big → whole disk	—	none
max	whole disk	—	none

Guarded for unattended server use: resolves root by filesystem LABEL not /dev/nvme0n1 (NVMe enumeration is not stable), refuses non-XFS, grows /sysroot rather than / (which is a composefs overlay and not growable), treats growpart exit 2 as no-op, and keeps an attempt marker so a partial failure surfaces instead of looping partition edits unattended.

build-raw-ghcr

bib bakes whatever reference it was given as the installed ostree origin. Building from localhost/server4home yields ostree-image-signed:docker://localhost/server4home:stable — an origin no timer can ever update, because there is no such registry to pull from. That is z170asrock's current state. Building from the ghcr reference makes the host self-updating from first boot and deletes the post-install rpm-ostree rebase step.

rpm-ostreed-automatic.timer enabled

Inactive on every host, which is why one box sat 8 days behind a healthy image pipeline. Paired with the existing AutomaticUpdates=stage this stages but lets the operator pick the activation moment — deliberately not bootc-fetch-apply-updates.timer, which fetches and reboots, which a storage server should not do unattended. (The k3s variant does enable that one; left alone as out of scope.)

Beszel deliberately NOT pre-enabled

beszel.container gates on /etc/beszel/hub.enabled precisely so one host runs the hub — baking that marker would make every server built from this image start its own. The agent gates on /etc/beszel/agent.env, a secret that must not ship in an image. Both quadlets and their tmpfiles are already baked, so the gates are all that remain, and they are correctly manual.

VM verification of the first-boot grow — and the bug it caught

Booted against a 4 TiB qcow2 COW overlay over the real disk.raw, which reproduces exactly what a real dd produces: a small image on a big drive whose GPT still describes the image's size and whose backup header is not at the end. Emulated NVMe so the device is nvme0n1 and the label-based discovery is exercised as it will be on hardware. The overlay means the artifact itself is never modified.

Run 1 failed, and it was worth finding:

firstboot-grow: growing the xfs filesystem at /sysroot
xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Read-only file system
server4home-firstboot-grow.service: Failed with result 'exit-code'

All the partition work had succeeded — sgdisk -e, the p5 placement, growpart reporting CHANGED: ... new: size=524288000 end=526798847. The unit avoided / (composefs, not growable) and targeted /sysroot, which is the XFS — but ostree mounts /sysroot read-only. Result was the worst possible shape: a 250 GiB partition containing a still-20 GiB filesystem, and no PV at all because the script exited before pvcreate.

Note this is invisible from the partition table alone — an independent qemu-nbd read showed p4 250.0 GiB xfs and p5 3.8 TiB vg4data, which looks like success. Only p5 lacking LVM2_member gave it away.

Fixed by growing via /var, a bind of the same filesystem that is mounted rw; growing is a superblock operation so it grows the whole XFS either way (xfs_growfs -n reports an identical superblock through both paths). That exposed a second bug in the same block: findmnt reports /dev/nvme0n1p4[/ostree/deploy/fedora-coreos/var], so the exact-match device comparison would never have matched /var even as a candidate.

Run 2, clean:

firstboot-grow: root=/dev/nvme0n1p4 disk=/dev/nvme0n1 partnum=4 mountpoint=/var
firstboot-grow: relocating backup GPT header to the end of /dev/nvme0n1
firstboot-grow: creating vg4data partition at sector 526798848 (bounds root to ~250G)
CHANGED: partition=4 start=2510848 old: size=41945055 new: size=524288000 end=526798847
firstboot-grow: growing the xfs filesystem at /var
data blocks changed from 5243130 to 65536000
Physical volume "/dev/nvme0n1p5" successfully created.
Volume group "vg4data" successfully created
firstboot-grow: complete
Finished server4home-firstboot-grow.service ... res=success

Independently verified from the overlay rather than trusting the guest's report:

check	method	result
partition table	sgdisk -p via qemu-nbd	p4 250.0 GiB, p5 3.8 TiB vg4data, last usable sector 8589934558
filesystem actually grown	xfs_db -c 'sb 0' -p dblocks	65536000 × 4096 = exactly 250 GiB (5243130 was the stuck value)
PV + VG	on-disk LVM label	LVM2 001 + vg4data present
one-shot	stamp file	/var/lib/server4home/firstboot-grow.done written
artifact safety	inode + mtime	disk.raw unchanged by the test

Boot reached multi-user.target and a login prompt with no emergency shell. Only failing unit is zincati (no Ignition config in a bare VM; uCore uses rpm-ostreed-automatic regardless) — the same benign failure the earlier layout verification saw.

Not exercised: the remount-/sysroot-rw fallback, since /var was writable as expected. It exists for a host where no writable mountpoint of the root filesystem is present.

Also note CI's Build and push image failure here is infrastructure, not content: OCI runtime error: crun: unknown version specified in the ZFS smoke test step.

---

Supersedes #4, which GitHub auto-closed when its base branch feat/nvme-apst-karg-mechanism was deleted on merging #3. A closed PR cannot be retargeted or reopened, so this is a fresh PR with identical content. The branch has been rebased onto main with the already-merged karg commit dropped — diff is now purely Justfile + iso/disk.toml.

Two changes

1. An explicit all-plain partition layout in iso/disk.toml, verified by booting it in a VM.
2. A Justfile fix so a failed mv can no longer leave a previous build's disk.raw in place.

Why there is no LVM in the boot path

FCOS/uCore cannot carry a boot-critical filesystem on LVM. Three attempts, all of which built cleanly and passed loop-mount inspection:

1 — root on an LV. The initramfs has no LVM support whatsoever:

dracut modules present : crypt, dm      <-- no lvm
lvm files in initramfs : none

FCOS locates root by LABEL; with no way to activate the VG the label never appears:

dev-disk-by-label-root.device: Job ... failed with result 'timeout'
Started emergency.service - Emergency Shell

rd.lvm.lv= does not help — there is no LVM tooling in the initramfs to act on the karg.

2 — root plain, /boot omitted. bib only auto-creates /boot "when the root partition is on an LVM logical volume", and customizations.disk gives you exactly what you declare:

error: grub_search_label:527: no such device: boot.
error: grub_loader_boot:196: you need to load the kernel first.
grub>

3 — root plain + /boot + /var on an LV. Boots into the real root, then:

Timed out waiting for device dev-...-<var LV fs UUID>
Dependency failed for var.mount - /var

The image ships lvm2-monitor, lvm2-lvmpolld, lvm-devices-import and coreos-populate-lvmdevices, but no activation unit — no lvm2-activation*, no lvm2-pvscan@. And coreos-populate-lvmdevices cannot help: it declares RequiresMountsFor=/var/lib, so it needs /var mounted, which is the very thing blocked. Circular.

The layout that ships

1M bios_grub | 200M ESP | 1G xfs /boot (label boot) | 20G xfs / (label root)

Mirrors what already runs on z170asrock. Root is the last partition, so it grows with a plain growpart <disk> <n> && xfs_growfs /. /boot is declared explicitly because the baked grub.cfg does search --label boot.

VM verification

qemu + OVMF, emulated NVMe (so the device is nvme0n1, matching the server), COW overlay over the real disk.raw grown to 4 TiB so the GPT-alt-header-not-at-end condition matches a real dd:

GRUB loaded the kernel from /boot
initramfs found /dev/disk/by-label/root; XFS (nvme0n1p4) mounted
ostree-prepare-root: composefs mounted successfully
reached "localhost login:"  — no emergency mode
nvme_core.default_ps_max_latency_us=0 present in the running cmdline

Two benign messages: failed to create symbolic link '/var/log' (present in every boot, including working ones) and zincati.service failing (no Ignition config in a bare VM; uCore uses rpm-ostreed-automatic anyway).

Justfile: bib raw output lands in output/image, not output/raw

The post-build step cleared output/${type} before mv -f. bib's subdir name does not track --type:

--type	actual dir
iso	bootiso (already special-cased)
raw	image ← the bug
qcow2	qcow2

So output/image was never cleared, mv failed with cannot overwrite 'output/image': Directory not empty, and the recipe exited 1 while leaving the previous build's disk.raw in place. A July 21 image survived a July 29 build this way and was nearly verified as if it were new. Replaced with an explicit type→dir mapping.

LVM is not abandoned — just moved out of the boot path

Per-app LVs created after install, mounted under /var/lib/<app> with nofail,x-systemd.device-timeout=30 so a problem degrades instead of hanging a headless box. That is all the snapshot→ZFS workflow needs:

xfs_freeze -f /var/lib/<app>
lvcreate -s -L <cow> -n <app>-snap vg4data/<app>
xfs_freeze -u /var/lib/<app>
mount -o ro,nouuid /dev/vg4data/<app>-snap /mnt/snap
rsync -aHAX --delete /mnt/snap/ /var/mnt/<pool>/backup/<app>/
zfs snapshot <pool>/backup/<app>@<stamp>
umount /mnt/snap && lvremove -f vg4data/<app>-snap

Standing rule added to the file

Always VM-boot a change to disk.toml before writing it to hardware. All three failures above passed loop-mount inspection. Byte-correct on disk is not the same as bootable.

🤖 Generated with Claude Code