[ppc64le] testiso pxes tests failing with GRUB error

[jbtrystram]

PPC64LE fails to boot from PXE in pxe-offline-install.4k.ppcfw :

  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php


Trying to load:  from: /pci@800000020000000/scsi@3 ... 
E3404: Not a bootable device!
Trying to load:  from: /pci@800000020000000/ethernet@2 ... 
 Initializing NIC

  Reading MAC address from device: 52:54:00:12:34:56
  Requesting information via DHCP:     ���010����done
  Using IPv4 address: 192.168.76.9
  Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2
  Receiving data:  0 KBytes��������335 KBytes
  TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (335 KBytes)
  Successfully loaded
�[30m�[47mWelcome to GRUB!


�[37m�[40merror: ../../grub-core/kern/device.c:43:variable `root' isn't set.

Entering rescue mode...

grub rescue> 

I see grub was updated:

[2025-03-10T13:55:44.458Z] Upgraded:
[2025-03-10T13:55:44.459Z]   grub2-common 1:2.12-15.fc41 -> 1:2.12-19.fc41
[2025-03-10T13:55:44.459Z]   grub2-ppc64le 1:2.12-15.fc41 -> 1:2.12-19.fc41
[2025-03-10T13:55:44.459Z]   grub2-ppc64le-modules 1:2.12-15.fc41 -> 1:2.12-19.fc41
[2025-03-10T13:55:44.459Z]   grub2-tools 1:2.12-15.fc41 -> 1:2.12-19.fc41
[2025-03-10T13:55:44.459Z]   grub2-tools-minimal 1:2.12-15.fc41 -> 1:2.12-19.fc41

[marta-lewandowska]

Does this new version work for you? https://bodhi.fedoraproject.org/updates/FEDORA-2025-ecbe75c42b

[jbtrystram]

@marta-lewandowska no it does not fix it :

  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php


Trying to load:  from: /pci@800000020000000/scsi@3 ... 
E3404: Not a bootable device!
Trying to load:  from: /pci@800000020000000/ethernet@2 ... 
 Initializing NIC
  Reading MAC address from device: 52:54:00:12:34:56
  Requesting information via DHCP: done
  Using IPv4 address: 192.168.76.9
  Requesting file "/boot/grub2/powerpc-ieee1275/core.elf" via TFTP from 192.168.76.2
  Receiving data:  335 KBytes
  TFTP: Received /boot/grub2/powerpc-ieee1275/core.elf (335 KBytes)
  Successfully loaded
Welcome to GRUB!
                                                                                                                                                                                   
error: ../../grub-core/kern/device.c:43:variable `root' isn't set.                                                                                                                 
Entering rescue mode...                                                                                                                                                            
grub rescue>          

[jbtrystram]

@marta-lewandowska c9s and rhel9.6 are also affected

[jbtrystram]

Xref openshift/os#1767

[dustymabe]

It looks like this is a regression in GRUB but it is observed when we run our tests because we use GRUB (installed in COSA) in our tests to set up and execute PXE tests. This means the GRUB inside the payload (CoreOS) doesn't matter. It's the GRUB inside COSA that we needed to change in order to observe a failure or not. For the record we tried with grub2-2.12-20.fc41 but it didn't help.

For now we'll pin GRUB in COSA (PR incoming). I added the older GRUB2 version to our f41-coreos-continuous tag to enable this pin to happen:

[dustymabe@media ~]$ koji add-pkg --owner dustymabe f41-coreos-continuous grub2
Adding 1 packages to tag f41-coreos-continuous
[dustymabe@media ~]$ koji tag-build f41-coreos-continuous grub2-2.12-15.fc41 
Created task 130230776
Watching tasks (this may be safely interrupted)...
130230776 tagBuild (noarch): free
130230776 tagBuild (noarch): free -> assigned (buildvm-s390x-06.s390.fedoraproject.org)
130230776 tagBuild (noarch): assigned (buildvm-s390x-06.s390.fedoraproject.org) -> closed
  0 free  0 open  1 done  0 failed

130230776 tagBuild (noarch) completed successfully

[marmijo]

It should be noted that both pxe-offline-install.4k.ppcfw and pxe-online-install.rootfs-appended.ppcfw are failing this way.