Zpool 2.0 Upgrade not possible on RHEL 8.4

[fffmmm]

Hello,

After upgrading from ZFS 0.8.6 to 2.0.5 (DMKS) via EPEL on RHEL 8.4, I see that that the zpool can't be updated:

[root@grendel ~]# uname -r
4.18.0-305.7.1.el8_4.x86_64
[root@grendel ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)
[root@grendel ~]# rpm -qa | grep zfs
libzfs4-2.0.5-1.el8.x86_64
zfs-dkms-2.0.5-1.el8.noarch
zfs-2.0.5-1.el8.x86_64
zfs-release-1-8.4.noarch

[root@grendel ~]# zpool status
  pool: nupool
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 01:13:10 with 0 errors on Mon Jul 12 05:19:26 2021

Attempting to upgrade fails:

[root@grendel ~]# zpool upgrade nupool
This system supports ZFS pool feature flags.

cannot set property for 'nupool': invalid feature 'redaction_bookmarks'

This looks weird: why is kmod (but I'm using DKMS - and have done so for quite a while!) the old version?

[root@grendel ~]# zfs --version
zfs-2.0.5-1
zfs-kmod-0.8.6-1

I have tried removing zfs, rebooting, and re-installing, yet still this old kmod shows up.

Your help would be appreciated.

Cheers

[rincebrain]

Well, what does modinfo zfs say? What does dkms install -m zfs/2.0.5 do? (Note that the latter is from memory, and may require adjustment.)

If the latter doesn't error, what does modinfo zfs say after, and what does find /lib/modules/$(uname -r)/ -iname zfs.ko say?

The two guesses in my mind are "updated modules didn't install/failed to build" and "for some reason they installed in a different path, there's two copies, and the older ones are being picked".

[fffmmm]

Hi rincebrain,

these seem to look ok:

[root@grendel ~]# modinfo zfs
filename:       /lib/modules/4.18.0-305.7.1.el8_4.x86_64/extra/zfs.ko.xz
version:        2.0.5-1
license:        CDDL
author:         OpenZFS
description:    ZFS
alias:          devname:zfs
alias:          char-major-10-249
rhelversion:    8.4
srcversion:     877055B790B94EF5357DAC3
depends:        spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl
name:           zfs
vermagic:       4.18.0-305.7.1.el8_4.x86_64 SMP mod_unload modversions
parm:           zvol_inhibit_dev:Do not create zvol device nodes (uint)
parm:           zvol_major:Major number for zvol device (uint)
parm:           zvol_threads:Max number of threads to handle I/O requests (uint)
parm:           zvol_request_sync:Synchronously handle bio requests (uint)
parm:           zvol_max_discard_blocks:Max number of blocks to discard (ulong)
parm:           zvol_prefetch_bytes:Prefetch N bytes at zvol start+end (uint)
parm:           zvol_volmode:Default volmode property value (uint)
[...etc...]

[root@grendel ~]# dkms install -m zfs/2.0.5
Module zfs/2.0.5 already installed on kernel 4.18.0-305.7.1.el8_4.x86_64/x86_64

[root@grendel ~]#
[root@grendel ~]# modinfo zfs
filename:       /lib/modules/4.18.0-305.7.1.el8_4.x86_64/extra/zfs.ko.xz
version:        2.0.5-1
license:        CDDL
author:         OpenZFS
description:    ZFS
alias:          devname:zfs
alias:          char-major-10-249
rhelversion:    8.4
srcversion:     877055B790B94EF5357DAC3
depends:        spl,znvpair,icp,zlua,zzstd,zunicode,zcommon,zavl
name:           zfs
vermagic:       4.18.0-305.7.1.el8_4.x86_64 SMP mod_unload modversions
parm:           zvol_inhibit_dev:Do not create zvol device nodes (uint)
parm:           zvol_major:Major number for zvol device (uint)
parm:           zvol_threads:Max number of threads to handle I/O requests (uint)
parm:           zvol_request_sync:Synchronously handle bio requests (uint)
parm:           zvol_max_discard_blocks:Max number of blocks to discard (ulong)
parm:           zvol_prefetch_bytes:Prefetch N bytes at zvol start+end (uint)
parm:           zvol_volmode:Default volmode property value (uint)
[...etc...]

[root@grendel ~]# find /lib/modules/$(uname -r)/ -iname zfs.ko
[root@grendel ~]# 
[root@grendel ~]# find /lib/modules/$(uname -r)/ | grep -i zfs
/lib/modules/4.18.0-305.7.1.el8_4.x86_64/weak-updates/zfs.ko.xz
/lib/modules/4.18.0-305.7.1.el8_4.x86_64/extra/zfs.ko.xz
[root@grendel ~]#

Why the ancient kmod is being loaded is a mystery...

[rincebrain]

I'd bet a nickel that modinfo /lib/modules/4.18.0-305.7.1.el8_4.x86_64/weak-updates/zfs.ko.xz says 0.8.x.

My other guess if it doesn't is that the initrd has the old version and you should regenerate that.

[fffmmm]

You're a genius!

Indeed, the initramfs was old. Doing a dnf upgrade zfs is not enough, a manual
dracut -f
was required and a reboot.

Now it looks good:

[root@grendel ~]# zfs --version
zfs-2.0.5-1
zfs-kmod-2.0.5-1

[root@grendel ~]# zpool upgrade -a
This system supports ZFS pool feature flags.

Enabled the following features on 'nupool':
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress

Enabled the following features on 'pool':
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress

Thank you for your help. Is there a button where I can give you a gold star or something? =)

[rincebrain]

You're quite welcome.

I'm not aware of one, but I've not paid attention to most of the features on this site, so one might well exist.