ci: Update for Fedora 44 branching from rawhide

[jeckersb]

Fedora 44 has branched from rawhide, so update the CI matrix
accordingly:

• Point fedora-44 images to :44 instead of :rawhide
• Add fedora-45 as the new rawhide
• Add fedora-44 to integration test matrix (BZ 2429501 is fixed)
• Add fedora-44 to Packit COPR and TMT test targets
• Re-enable rawhide Packit TMT tests (BZ 2429501 is fixed)

Closes: #1985
Assisted-by: OpenCode (claude-opus-4-6)
Signed-off-by: John Eckersberg jeckersb@redhat.com

[gemini-code-assist]

Code Review

This pull request updates the CI configuration for the Fedora 44 branching. The changes correctly point fedora-44 to its own release tag instead of rawhide, introduce fedora-45 as the new rawhide, and update the Packit and integration test matrices. The re-enabling of rawhide tests is also correctly implemented. The changes are consistent and align perfectly with the pull request description. I see no issues.

[cgwalters]

I'm OK with this, but a definite downside is again increasing the test matrix run on every PR. We may need to move some testing to be postsubmit.

[cgwalters]

Looks like GRUB is still busted in f44

[jeckersb]

Rebased, we'll see if grub is still busted...

[jeckersb]

Hm not sure offhand what this is about in c10s (Edit: also f44)...

content: STEP 1/2: FROM localhost/bootc
content: STEP 2/2: RUN echo 'unified-storage-test-marker' > /usr/share/unified-storage-test.txt
content: error: Subprocess failed: ExitStatus(unix_wait_status(32512))
content: /bin/sh: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
content: Error: building at STEP "RUN echo 'unified-storage-test-marker' > /usr/share/unified-storage-test.txt": while running runtime: exit status 127

[jeckersb]

Hm not sure offhand what this is about in c10s (Edit: also f44)...

content: STEP 1/2: FROM localhost/bootc
content: STEP 2/2: RUN echo 'unified-storage-test-marker' > /usr/share/unified-storage-test.txt
content: error: Subprocess failed: ExitStatus(unix_wait_status(32512))
content: /bin/sh: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
content: Error: building at STEP "RUN echo 'unified-storage-test-marker' > /usr/share/unified-storage-test.txt": while running runtime: exit status 127

Looks like https://bugzilla.redhat.com/show_bug.cgi?id=2436646 is related.

[jeckersb]

I can't reproduce the CI failure by running through the unified storage test manually. I'll force one more CI run just in case it got fixed between last night and now...

[jeckersb]

Ok still failing. I did manage to reproduce it locally, but only by running through the just test-tmt 31-switch-to-unified flow. Here's the audit log:

kernel: audit: type=1400 audit(1771525274.279:7): avc:  denied  { read } for  pid=1778 comm="sh" 
path="/usr/lib64/libc.so.6" dev="vda3" ino=22639696 scontext=system_u:system_r:container_t:s0:c387,c763
tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=0

So libc is unlabeled_t...

[jeckersb]

A bit more detail, running the test with --preserve-vm and then dropping in to run the failing bit manually:

[root@ibm-p8-kvm-03-guest-02 tmp.M3sxK1Uemp]# RUST_LOG=trace bootc image cmd build -t localhost/bootc-unified-derived .
TRACE starting bootc
DEBUG argv0="bootc"
TRACE Verified uid 0 with CAP_SYS_ADMIN
DEBUG Re-executing current process for _ostree_unshared
TRACE starting bootc
DEBUG argv0="bootc"
TRACE Verified uid 0 with CAP_SYS_ADMIN
DEBUG Already in a mount namespace
DEBUG Current security context is unconfined_u:system_r:install_t:s0-s0:c0.c1023
DEBUG We have install_t
DEBUG Target . is a mountpoint, remounting rw
TRACE loading sepolicy from booted ostree deployment
TRACE sepolicy in get_ensure_imgstore: Some(SePolicy { inner: TypedObjectRef { inner: 0x561630cd10a0, type: OstreeSePolicy } })
TRACE Opening container image store
TRACE exec: REGISTRY_AUTH_FILE="/proc/self/fd/26" "podman" "--root" "/run/bootc/storage" "--runroot" "/proc/self/fd/3" "image" "build" "-t" "localhost/bootc-unified-derived" "."
STEP 1/2: FROM localhost/bootc
STEP 2/2: RUN echo 'unified-storage-test-marker' > /usr/share/unified-storage-test.txt
error: Subprocess failed: ExitStatus(unix_wait_status(32512))
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
Error: building at STEP "RUN echo 'unified-storage-test-marker' > /usr/share/unified-storage-test.txt": while running runtime: exit status 127

[jeckersb]

Rebased 🏄 to see if picking up #2014 will move this along...

[jeckersb]

Ok grub appears to still be busted on F44. Probably going to start being a problem for us sooner than later since we're now in beta freeze...

[henrywang]

Our bootc 1.13.0 fedora 44 and 45 gating tests have same failure.

• https://src.fedoraproject.org/rpms/bootc/pull-request/80
• https://src.fedoraproject.org/rpms/bootc/pull-request/79

[henrywang]

Okey, AI told me:

Fedora 44's xfsprogs creates XFS filesystems with newer features (CRC, bigtime, finobt, reflink) that CentOS Stream 9's      grub2 doesn't fully support. When:
  1. Fedora 44's mkfs.xfs creates the filesystem
  2. CentOS Stream 9's grub2-install (via bwrap) tries to install the bootloader
  3. CS9's grub2 fails with "unknown filesystem" because it doesn't recognize the newer XFS metadata format

I think we can use same target_image as the host OS. This fix works in my local run.

diff --git i/tmt/tests/booted/tap.nu w/tmt/tests/booted/tap.nu
index ae1576da..0100fcbd 100644
--- i/tmt/tests/booted/tap.nu
+++ w/tmt/tests/booted/tap.nu
@@ -18,3 +18,62 @@ export def is_composefs [] {
     let st = bootc status --json | from json
     $st.status.booted.composefs? != null
 }
+
+# Get the target image for install tests based on the running OS
+# This ensures the target image matches the host OS to avoid version mismatches
+# (e.g., XFS features created by newer mkfs.xfs not recognized by older grub2)
+export def get_target_image [] {
+    # Parse os-release to get ID and VERSION_ID
+    let os = open /usr/lib/os-release
+        | lines
+        | filter {|l| $l != "" and not ($l | str starts-with "#") }
+        | parse "{key}={value}"
+        | reduce -f {} {|it, acc|
+            $acc | upsert $it.key ($it.value | str trim -c '"')
+        }
+
+    let id = $os.ID
+    let version_id = $os.VERSION_ID
+
+    # Construct the key for os-image-map.json
+    let key = if $id == "centos" {
+        # CentOS uses "centos-9" or "centos-10" format
+        $"centos-($version_id)"
+    } else if $id == "fedora" {
+        $"fedora-($version_id)"
+    } else if $id == "rhel" {
+        # RHEL uses "rhel-9.8" or "rhel-10.2" format
+        $"rhel-($version_id)"
+    } else {
+        # Fallback to centos-9 for unknown distros
+        "centos-9"
+    }
+
+    # Load the os-image-map.json - try multiple possible locations
+    let possible_paths = [
+        "hack/os-image-map.json",
+        "../../../hack/os-image-map.json",
+        "/var/home/bootc/hack/os-image-map.json"
+    ]
+
+    mut image_map = null
+    for p in $possible_paths {
+        if ($p | path exists) {
+            $image_map = (open $p)
+            break
+        }
+    }
+
+    # If map not found, use default centos-9 image
+    if ($image_map == null) {
+        return "docker://quay.io/centos-bootc/centos-bootc:stream9"
+    }
+
+    let image = $image_map.base | get -i $key
+    if ($image | is-empty) {
+        # Fallback to centos-9 if key not found
+        $"docker://($image_map.base.centos-9)"
+    } else {
+        $"docker://($image)"
+    }
+}
diff --git i/tmt/tests/booted/test-install-outside-container.nu w/tmt/tests/booted/test-install-outside-container.nu
index 312c5715..b9c6831e 100644
--- i/tmt/tests/booted/test-install-outside-container.nu
+++ w/tmt/tests/booted/test-install-outside-container.nu
@@ -6,11 +6,9 @@
 use std assert
 use tap.nu

-# In this test we install a generic image mainly because it keeps
-# this test in theory independent of starting from a bootc host,
-# but also because it's useful to test "skew" between the bootc binary
-# doing the install and the target image.
-let target_image = "docker://quay.io/centos-bootc/centos-bootc:stream9"
+# Get the target image based on the running OS to avoid version mismatches
+# (e.g., XFS features created by newer mkfs.xfs not recognized by older grub2)
+let target_image = (tap get_target_image)

 # setup filesystem
 mkdir /var/mnt
diff --git i/tmt/tests/booted/test-install-unified-flag.nu w/tmt/tests/booted/test-install-unified-flag.nu
index 00620b6e..71a8e2aa 100644
--- i/tmt/tests/booted/test-install-unified-flag.nu
+++ w/tmt/tests/booted/test-install-unified-flag.nu
@@ -12,9 +12,9 @@
 use std assert
 use tap.nu

-# Use a generic target image to test skew between the bootc binary doing
-# the install and the target image
-let target_image = "docker://quay.io/centos-bootc/centos-bootc:stream9"
+# Get the target image based on the running OS to avoid version mismatches
+# (e.g., XFS features created by newer mkfs.xfs not recognized by older grub2)
+let target_image = (tap get_target_image)

 def main [] {
     tap begin "install with experimental unified storage flag"

[cgwalters]

I think we can use same target_image as the host OS. This fix works in my local run.

Right, didn't you already land that same fix in another PR?

[cgwalters]

Offhand I suspect this one just needs a rebase

[jeckersb]

Rebased 🏄

[henrywang]

Right, didn't you already land that same fix in another PR?

Not in test-install-outside-container.nu. test-install-outside-container.nu runs in both ostree and composefs tests. In composefs test, the localhost/bootc is UKI image and has a UKI at /boot/EFI/Linux/*.efi. --bootloader=systemd will not work.
The test needs some updates that should cover both sealed image and traditional ostree/BLS image without UKI in this test. Including this update in #2029 is not related.

[cgwalters]

I think we can use same target_image as the host OS.

OK right, sorry I misunderstood this before.

Mmmm....IIRC i mentioned this before but you know what would make this all SO much cleaner is for us to add e.g. BOOTC_IMAGE=<pull spec> into /usr/lib/os-release.

---

A different possibility for install-outside-container.nu is for us to drop it and instead add testing via anaconda/bootc-image-builder as part of PRs which would cover this case.

For test-install-unified-flag.nu...I think we should actually make that a toplevel matrix entry (yes, at which point we'll have WAY too many tests to feasibly run on each PR). But really with unified storage e.g. we do want to test upgrades and we're missing that here.

[cgwalters]

Hmm wait a second this PR is starting to accumulate some nontrivial changes to core code, is that really related to fedora 44?

[henrywang]

Hmm wait a second this PR is starting to accumulate some nontrivial changes to core code, is that really related to fedora 44?

I tried a lot for support container-storage, but failed. I got regression issues for each change. Rollback to update test only. Still have a lot to learn about bootc. Time to land this PR.

[cgwalters]

I think it'd be cleaner to just copy os-image-map.json into e.g. /usr/lib/bootc/os-image-map.json as part of the container build.