Skip to content

fix(disk): explicit all-plain partition layout (VM-verified) + bib raw output dir fix - #4

Closed
dx4homelab wants to merge 2 commits into
feat/nvme-apst-karg-mechanismfrom
feat/lvm-disk-layout
Closed

fix(disk): explicit all-plain partition layout (VM-verified) + bib raw output dir fix#4
dx4homelab wants to merge 2 commits into
feat/nvme-apst-karg-mechanismfrom
feat/lvm-disk-layout

Conversation

@dx4homelab

@dx4homelab dx4homelab commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Stacked on #3. Merge #3 first; this retargets to main automatically.

This PR replaced its own original approach. It started as an LVM-root layout. That was built, loop-mounted, confirmed byte-correct — and does not boot. Three variants were tried; the history below is kept because the failure modes are non-obvious and each one costs a reflash to rediscover.

Two changes

  1. An explicit all-plain partition layout in iso/disk.toml, verified by booting it in a VM.
  2. A Justfile fix so a failed mv can no longer leave a previous build's disk.raw in place.

Why there is no LVM in the boot path

FCOS/uCore cannot carry a boot-critical filesystem on LVM. Three attempts, all of which built cleanly and passed loop-mount inspection:

1 — root on an LV. The initramfs has no LVM support whatsoever:

dracut modules present : crypt, dm      <-- no lvm
lvm files in initramfs : none

FCOS locates root by LABEL; with no way to activate the VG the label never appears:

dev-disk-by-label-root.device: Job ... failed with result 'timeout'
Started emergency.service - Emergency Shell

rd.lvm.lv= does not help — there is no LVM tooling in the initramfs to act on the karg.

2 — root plain, /boot omitted. bib only auto-creates /boot "when the root partition is on an LVM logical volume", and customizations.disk gives you exactly what you declare:

error: grub_search_label:527: no such device: boot.
error: grub_loader_boot:196: you need to load the kernel first.
grub>

3 — root plain + /boot + /var on an LV. Boots into the real root, then:

Timed out waiting for device dev-...-<var LV fs UUID>
Dependency failed for var.mount - /var

The image ships lvm2-monitor, lvm2-lvmpolld, lvm-devices-import and coreos-populate-lvmdevices, but no activation unit — no lvm2-activation*, no lvm2-pvscan@. And coreos-populate-lvmdevices cannot help: it declares RequiresMountsFor=/var/lib, so it needs /var mounted, which is the very thing blocked. Circular.

The layout that ships

1M bios_grub | 200M ESP | 1G xfs /boot (label boot) | 20G xfs / (label root)

Mirrors what already runs on z170asrock. Root is the last partition, so it grows with a plain growpart <disk> <n> && xfs_growfs /. /boot is declared explicitly because the baked grub.cfg does search --label boot.

VM verification

qemu + OVMF, emulated NVMe (so the device is nvme0n1, matching the server), COW overlay over the real disk.raw grown to 4 TiB so the GPT-alt-header-not-at-end condition matches a real dd:

GRUB loaded the kernel from /boot
initramfs found /dev/disk/by-label/root; XFS (nvme0n1p4) mounted
ostree-prepare-root: composefs mounted successfully
reached "localhost login:"  — no emergency mode
nvme_core.default_ps_max_latency_us=0 present in the running cmdline

Two benign messages: failed to create symbolic link '/var/log' (present in every boot, including working ones) and zincati.service failing (no Ignition config in a bare VM; uCore uses rpm-ostreed-automatic anyway).

Justfile: bib raw output lands in output/image, not output/raw

The post-build step cleared output/${type} before mv -f. bib's subdir name does not track --type:

--type actual dir
iso bootiso (already special-cased)
raw image ← the bug
qcow2 qcow2

So output/image was never cleared, mv failed with cannot overwrite 'output/image': Directory not empty, and the recipe exited 1 while leaving the previous build's disk.raw in place. A July 21 image survived a July 29 build this way and was nearly verified as if it were new. Replaced with an explicit type→dir mapping.

LVM is not abandoned — just moved out of the boot path

Per-app LVs created after install, mounted under /var/lib/<app> with nofail,x-systemd.device-timeout=30 so a problem degrades instead of hanging a headless box. That is all the snapshot→ZFS workflow needs:

xfs_freeze -f /var/lib/<app>
lvcreate -s -L <cow> -n <app>-snap vg4data/<app>
xfs_freeze -u /var/lib/<app>
mount -o ro,nouuid /dev/vg4data/<app>-snap /mnt/snap
rsync -aHAX --delete /mnt/snap/ /var/mnt/<pool>/backup/<app>/
zfs snapshot <pool>/backup/<app>@<stamp>
umount /mnt/snap && lvremove -f vg4data/<app>-snap

Standing rule added to the file

Always VM-boot a change to disk.toml before writing it to hardware. All three failures above passed loop-mount inspection. Byte-correct on disk is not the same as bootable.

🤖 Generated with Claude Code

dx4homelab and others added 2 commits July 29, 2026 09:38
…t dir

Two changes, the second discovered while validating the first.

LVM layout (iso/disk.toml)
--------------------------
Replaces the single [[customizations.filesystem]] entry — the simple and
advanced (customizations.disk) forms are alternatives, not additive.

z170asrock is swapping its boot NVMe from 931 GB to 4 TB. Measured live
2026-07-29, that host has NO LVM today: one 20 GiB xfs partition carries
/sysroot + /var + /etc, and ~911 GB of the drive is unallocated because the dd
install was never grown. (docs/hardware-inventory.md described a vg4base +
vg4home layout that has not existed since the uCore migration.)

vg4base now holds root (16 GiB) and var (10 GiB). Splitting /var out of root is
the one structural change and it earns its keep on an unattended server: if
container images, logs or Postgres fill a shared filesystem you cannot stage an
ostree upgrade or cleanly log in to fix it. FCOS supports a separate /var as a
first-class configuration.

Nothing is needed in the image for this — lvm2, device-mapper, the dracut 70lvm
module and coreos-populate-lvmdevices.service are already present, and lsinitrd
confirms LVM is already in the initramfs.

bootc-image-builder minsize semantics
-------------------------------------
Documented in the file because it is not in the upstream docs and cost two
builds to pin down. `minsize` on the lvm partition is REQUIRED (omitting it
fails with "minsize is required"), and it is NOT the size of the volume group:
it is applied to the ROOT lv, root's own minsize is ignored, other LVs get
their own minsize, and the VG becomes the sum.

  VG 26 GiB + root 16 GiB + var 10 GiB -> root=26G var=10G VG=36G  raw 37.2 GiB
  VG 16 GiB + root 16 GiB + var 10 GiB -> root=16G var=10G VG=26G  raw 27.2 GiB

Not just image size: XFS cannot shrink, so an oversized root LV is space that
can never be handed back to var.

Justfile: bib raw output lands in output/image, not output/raw
--------------------------------------------------------------
The post-build step cleared output/${type} before `mv -f`. bib's subdir name
does not track --type: iso -> bootiso (already special-cased), raw -> image,
qcow2 -> qcow2. So for raw builds output/image was never cleared, mv failed
with "cannot overwrite 'output/image': Directory not empty", and the recipe
exited 1 while leaving the PREVIOUS build's disk.raw in place. A green-looking
build that silently ships a stale artifact. Replaced with an explicit
type -> dir mapping.

Verified by building three times and loop-mounting each result:
  partition table  1M bios_grub / 200M EFI / 1G xfs /boot / 26G LVM2_member
  /boot and EFI    outside LVM, as required by the bootloader
  vg4base          root 16.00g xfs label=root (holds boot, ostree, var)
                   var  10.00g xfs label=var
  APST karg        present in /loader.1/entries/ostree-1.conf, so a dd'd
                   system is protected from its first boot
  raw              29208084480 bytes, staged and sha256-verified to
                   /mnt/pny2xxG/server4homelab/july-29-2026/

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverses course on LVM-on-root/-var. Three layouts were built, loop-mounted and
confirmed byte-correct, and all three failed to boot. FCOS/uCore cannot carry a
boot-critical filesystem on LVM:

1. root on an LV — the initramfs has NO LVM support at all:
     dracut modules present : crypt, dm      (no lvm)
     lvm files in initramfs : none
   FCOS finds root by LABEL, nothing can activate the VG, so:
     dev-disk-by-label-root.device: Job ... failed with result 'timeout'
     Started emergency.service - Emergency Shell
   rd.lvm.lv= does not help; there is no LVM tooling to act on it.

2. root plain, /boot omitted — bib only auto-creates /boot when root is on an
   LV, and customizations.disk gives exactly what you declare:
     error: grub_search_label:527: no such device: boot.
     error: grub_loader_boot:196: you need to load the kernel first.

3. root plain + /boot + /var on an LV — boots into the real root, then:
     Timed out waiting for device dev-...-<var LV fs UUID>
     Dependency failed for var.mount - /var
   The image ships lvm2-monitor, lvm2-lvmpolld, lvm-devices-import and
   coreos-populate-lvmdevices but NO activation unit (no lvm2-activation*, no
   lvm2-pvscan@). And coreos-populate-lvmdevices declares
   RequiresMountsFor=/var/lib — it needs /var mounted, which is the very thing
   blocked. Circular, so it can never help here.

This layout is all-plain and mirrors what already runs on z170asrock:
  1M bios_grub | 200M ESP | 1G xfs /boot (label boot) | 20G xfs / (label root)
Root is last, so it grows with plain `growpart <disk> <n> && xfs_growfs /`.

/boot is declared explicitly because the baked grub.cfg does
`search --label boot` and bib will not create it for a plain root.

VM-verified (qemu, OVMF, emulated NVMe, COW overlay grown to 4T so the GPT
alt-header condition matches a real dd):
  GRUB loaded the kernel from /boot
  initramfs found /dev/disk/by-label/root; XFS (nvme0n1p4) mounted
  ostree-prepare-root: composefs mounted successfully
  reached "localhost login:" — no emergency mode
  nvme_core.default_ps_max_latency_us=0 present in the running cmdline

LVM is not abandoned, just moved out of the boot path: per-app LVs created
AFTER install and mounted under /var/lib/<app> with nofail and a device
timeout. That is all the snapshot-to-ZFS workflow needs. Documented in the file.

A loop-mount proves the bytes are right, not that it boots — all three failures
passed loop-mount inspection. Always VM-boot a change to this file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dx4homelab dx4homelab changed the title feat(disk): LVM layout for the 4 TB boot drive, and fix bib raw output dir fix(disk): explicit all-plain partition layout (VM-verified) + bib raw output dir fix Jul 29, 2026
@dx4homelab
dx4homelab deleted the branch feat/nvme-apst-karg-mechanism July 30, 2026 02:04
@dx4homelab dx4homelab closed this Jul 30, 2026
@dx4homelab

Copy link
Copy Markdown
Owner Author

Superseded by #5 — this PR was auto-closed by GitHub when its base branch feat/nvme-apst-karg-mechanism was deleted while merging #3. Closed PRs cannot be retargeted or reopened, so the work continues in #5 with identical content, rebased onto main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant