OpenZFS compatibility patch for kernel 6.8: copy_file_range

[robobenklein]

Pop shipped zfs 2.2.3 which only officially supported up to 6.7, but since kernel 6.8 many of the applications I'm running with data on a ZFS pool have broken. (Proton is one such easily tested example, but it affects my work and research projects as well.)

For now, could the immediate patch be brought in to resolve the current incompatibility issue? openzfs/zfs#15931

Additionally, I am somewhat worried about future similar updates that could cause data loss or break existing applications. Could there be some method of preventing incompatible kernel updates? I've had DKMS problems in the past with ZFS on Pop as well, and now seeing a silent bug crop up is making me paranoid / worried.

[meonkeys]

I'm getting a different ZFS error with Linux 6.8.0 but the root cause might be the same incompatibility issue you reference. My computer was working up through Linux 6.6.10. I'm using Ubuntu 22.04 LTS but again, might be relevant.

I haven't changed any ZFS filesystems on my computer (a System76 Lemur Pro), and both zfs-dkms and zfs-initramfs packages are still installed. dkms status says zfs/2.2.3 is installed for the 6.8.0 kernel. When I try to boot 6.8.0, it gets stuck at initramfs. First I see

Loading Linux 6.8.0-76060800daily20240311-generic ...
Loading initial ramdisk ...

Then I briefly see a graphical System 76 logo. This is where I usually see a prompt to decrypt the root ZFS filesystem ("cryptsetup keystore-rpool"). Instead, it drops to a text console, and I see

Key load error: Failed to open key material file: No such file or directory

Command: mount -o zfsutil -t zfs rpool/ROOT/ubuntu_2bfkjw /root//
Message: zfs_mount_at() failed: encryption key not loadedzfs_mount_at() failed: encryption key not loadedmount: mounting rpool/ROOT/ubuntu_2bfkjw on /root// failed: Permission denied
Error: 2

Failed to mount rpool/ROOT/ubuntu_2bfkjw on /root//.
Manually mount the filesystem and exit.


BusyBox v1.30.1 (Ubuntu 1:1.30.1-7ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

I tried running update-initramfs manually and it made a new initramfs, but that one had the same decryption issue at boot.

[robobenklein]

That doesn't sound like copy_file_range but a different 6.8 compatibility problem.

If I were running ZFS-on-root I would be in a very problematic situation, I'm just using ZFS on secondary drives.

In your case, since your system isn't bootable, do you have the option to boot an older kernel? I still had a 6.6 kernel initrd & vmlinux in /boot.

Perhaps you have a prior or backup kernel version as well that can be overridden during boot? And then after boot pin the older kernel package to prevent autoremoval until we get a fixed ZFS version? (If it were still bootable I would suggest using kernelstub to update the target initrd and kernel files, but that could potentially be done with a Pop!_OS ISO with a 6.7 or 6.6 kernel and chooting into the ZFS root FS.)

[meonkeys]

That doesn't sound like copy_file_range but a different 6.8 compatibility problem.

I agree, but I'm still wondering if these are both just symptoms of related underlying compatibility problem(s). If not, should I file another issue? I'm not using Pop!_OS so I wasn't sure if this would be appropriate.

do you have the option to boot an older kernel?

Thankfully, yes!

I should for sure pin the older working kernel and make it the default to boot with. I forget exactly how to do that so I'll need to dig around a bit. Luckily I rarely need to reboot this laptop (still going strong!), so I've just been manually selecting the old kernel during boot.

Should I be using kernelstub instead of update-initramfs?