feat(nvme): add karg mechanism for INNOGRIT IG5236 APST defect

[dx4homelab]

server4home had no way to carry a kernel argument — no kargs.d, no setup hooks, nothing (verified against the built image). Any hardware quirk had to be fixed by remembering to run rpm-ostree kargs by hand after every install, and it would not survive a rebuild. This adds a mechanism, and uses it for the IG5236 defect.

Background

IG5236 "RainierPC" drives (HP SSD FX900 Pro and rebadges) enter their deepest APST power state a few seconds after going idle and stop responding on the PCIe bus:

nvme nvmeN: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
nvme nvmeN: Does your device have a faulty power saving mode enabled?

On a root-filesystem drive an unrecovered reset takes the filesystem read-only. SMART stays clean throughout (0% used, 0 media errors, PASSED) — so neither smartd nor Beszel will warn you. No firmware fix exists: LVFS has nothing in stable or testing, Biwin's updater is Windows-only, and 3.A.J.CR looks current across the whole line.

Observed 10 times on fury4dx over one week, escalating to twice daily, until it took the box down. minis4dx carries the same drive and was fixed the same way — the equivalent hook there applied the karg unattended and self-rebooted, confirmed working.

Relevant here because the z170asrock boot-drive swap may use an FX900 Pro, and that host is headless, unattended, and will hold the ZFS pool keyfile.

Two layers

1. Install-time — iso/disk.toml gains:

[customizations.kernel]
append = "nvme_core.default_ps_max_latency_us=0"

A freshly dd'd system is protected from its very first boot. The runtime unit cannot help there — it only runs after the machine has already booted once on the affected drive, which is exactly the window where an unattended install could go read-only.

Harmless on hosts without an IG5236 (it only disables NVMe autonomous power state transitions, costing a little idle power on an always-on box). The block is commented so it can be dropped if the boot drive ends up being something else.

2. Runtime — server4home-nvme-apst.service detects the controller by PCI ID (1dbe:5236) via sysfs, stages the karg with rpm-ostree kargs --append-if-missing, and reboots. Covers hosts installed before this change and drives swapped in later. Gated on the PCI ID, so machines without the silicon keep their NVMe power savings.

Implementation note

The unit ships enabled via a multi-user.target.wants symlink in the overlay rather than systemctl enable in 10-build.sh — Containerfile line 107 copies build/files/ / after that script runs, so the unit file does not exist yet at enable time.

Reboot safety

A server that reboot-loops is worse than one with APST enabled. A stamp at /var/lib/server4home/nvme-apst.attempted guards it: if the karg was staged on a previous boot and is still not active, the script refuses and exits non-zero rather than rebooting again. A failed rpm-ostree call clears the stamp so the next boot retries.

Verification

All six branches exercised with faked /proc/cmdline, a faked PCI tree, and stubbed rpm-ostree/systemctl (nothing could reboot the test host):

scenario	expected	result
karg already active	no-op, no calls	✅
no IG5236 present	no-op, no stamp	✅
IG5236 + karg missing	stage karg, exactly one reboot, stamp written	✅
stamp + karg still missing	refuse, no second reboot	✅
rpm-ostree fails	no reboot, stamp cleared for retry	✅
karg active after reboot	stale stamp cleaned up	✅

bash -n clean; unit syntax verified.

Note on LVM (asked alongside this)

Worth recording: LVM needs nothing added to this image. lvm2 2.03.38, device-mapper, cryptsetup, the dracut 70lvm module and coreos-populate-lvmdevices.service are all present, and lsinitrd confirms LVM is already in the initramfs for both kernel trees. bootc-image-builder supports LVM root layouts via customizations.disk. So LVM-on-root is purely an iso/disk.toml change, not an image change — deliberately left out of this PR to keep it focused.

🤖 Generated with Claude Code