centos stream 9 to rocky 9 migration failure with cannot install the best update candidate for package kernel-devel-matched-5.14.0-474.el9.x86_64

[akrherz]

FWIW, I am migrating my various centos stream 9 hosts to rocky 9 and have hit this error twice today

...
Complete!
Last metadata expiration check: 0:02:15 ago on Wed Jul 10 12:54:37 2024.
No package fwupdate* installed.
Error: 
 Problem 1: cannot install the best update candidate for package kernel-devel-matched-5.14.0-474.el9.x86_64
  - nothing provides kernel-core = 5.14.0-427.24.1.el9_4 needed by kernel-devel-matched-5.14.0-427.24.1.el9_4.x86_64 from appstream
 Problem 2: problem with installed package kernel-devel-matched-5.14.0-474.el9.x86_64
  - kernel-devel-matched-5.14.0-474.el9.x86_64 from @System  does not belong to a distupgrade repository
  - nothing provides kernel-core = 5.14.0-427.24.1.el9_4 needed by kernel-devel-matched-5.14.0-427.24.1.el9_4.x86_64 from appstream
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Error during distro-sync.

An error occurred while we were attempting to convert your system to Rocky Linux. Your system may be unstable. Script will now exit to prevent possible damage.

[pajamian]

Can you show your full migrate2rocky.log file please?

[akrherz]

Sure, here it is.
migrate2rocky.log.txt

[pajamian]

Do you have kernel* excluded? This appears to be the root of the problem.

Other items of note from the log:

centos-stream-release seems to be mis-detected as centos-release-automotive. Ultimately this does not appear to be causing any issues but it's something to look into and fix.

distro-sync appears to complete and only then errors out. This can be problematic.

migrate2rocky9 is looking for a package that matches fwupdate*. This is a relic from migrate2rocky and doesn't appear to be causing any harm, but that should be removed because there are no fwupdate* packages in EL9.

[akrherz]

Do you have kernel* excluded? This appears to be the root of the problem.

I do not believe so. I should mention that for the two systems that have failed, they both are using openzfs. I did have 1 openzfs system succeed though. I have completed about 15 systems so far, so only two failures isn't too bad :) I have got a couple of ZFS hosts yet to go, so I'll see if I can increase the debugging on the failing dnf command and get you something more to go on here.

[akrherz]

I tried another ZFS CS9 box now and got the same failure. I instrumented the distro-sync dnf commands with a -v, but the output does not look that exciting to me :(

migrate2rocky.log.txt

[akrherz]

This seems like a problem to me, running from the host broken with the last comment

# dnf whatprovides  kernel-devel-matched
Last metadata expiration check: 0:20:14 ago on Thu 11 Jul 2024 08:23:47 AM CDT.
kernel-devel-matched-5.14.0-427.22.1.el9_4.x86_64 : Meta package to install matching core and devel packages for a given kernel
Repo        : appstream
Matched from:
Provide    : kernel-devel-matched = 5.14.0-427.22.1.el9_4

kernel-devel-matched-5.14.0-474.el9.x86_64 : Meta package to install matching core and devel packages for a given kernel
Repo        : @System
Matched from:
Provide    : kernel-devel-matched = 5.14.0-474.el9

So there's no 5.14.0-427.24.1 there, but only 5.14.0-427.22.1 ??? but on a host I manually repaired

dnf list kernel-devel-matched-5.14.0-427.24.1.el9_4.x86_64
Last metadata expiration check: 1:10:46 ago on Thu 11 Jul 2024 07:27:19 AM CDT.
Installed Packages
kernel-devel-matched.x86_64    5.14.0-427.24.1.el9_4   @appstream

[akrherz]

Sorry to microblog here, so I run dnf clean all and now 24 appears available

# dnf clean all
182 files removed
# dnf whatprovides  kernel-devel-matched
dell-system-update_independent                                                                                                                             293 kB/s | 142 kB     00:00    
dell-system-update_dependent                                                                                                                               406 kB/s | 272 kB     00:00    
Extra Packages for Enterprise Linux 9 - x86_64                                                                                                              21 MB/s |  22 MB     00:01    
Extra Packages for Enterprise Linux 9 openh264 (From Cisco) - x86_64                                                                                       4.5 kB/s | 2.5 kB     00:00    
Rocky Linux 9 - BaseOS                                                                                                                                     4.8 MB/s | 2.3 MB     00:00    
Rocky Linux 9 - AppStream                                                                                                                                   12 MB/s | 7.9 MB     00:00    
Rocky Linux 9 - CRB                                                                                                                                        5.2 MB/s | 2.4 MB     00:00    
Rocky Linux 9 - Extras                                                                                                                                      51 kB/s |  15 kB     00:00    
ZFS on Linux for EL9 - dkms                                                                                                                                191 kB/s | 107 kB     00:00    
kernel-devel-matched-5.14.0-427.24.1.el9_4.x86_64 : Meta package to install matching core and devel packages for a given kernel
Repo        : appstream
Matched from:
Provide    : kernel-devel-matched = 5.14.0-427.24.1.el9_4

kernel-devel-matched-5.14.0-474.el9.x86_64 : Meta package to install matching core and devel packages for a given kernel
Repo        : @System
Matched from:
Provide    : kernel-devel-matched = 5.14.0-474.el9

[akrherz]

So ZFS may be a red herring, the next system (sans ZFS) I tried worked without issue and got kernel 5.14.0-427.24.1, but this system does not have kernel-devel installed. I may just be getting unlucky here with mirrors not being fully synced.