On s390x, test_kernel_stat and test_meminfo fail

[musicinmybrain]

I originally saw this while working on an update for the rust-procfs package in Fedora; the package build still happens in a mock chroot, but on real s390x hardware rather than under qemu-user-static emulation. I’ve tried to reproduce it in emulation, below, since I suspect that may be more useful to you.

---

Working in a Fedora 41 mock chroot with git, cargo, and rust installed, and with network access enabled:

On x86_64, skipping:

• process::tests::test_proc_fd_count_runsinglethread because of On at least Fedora 41, test_proc_fd_count_runsinglethread fails #322
• sys::kernel::random::tests::test_write_wakeup_threshold, process::tests::test_proc_auxv, and process::tests::test_proc_status_for_kthreadd because they fail in a mock chroot for various reasons

$ uname -srvmo
Linux 6.11.7-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov  8 19:23:10 UTC 2024 x86_64 GNU/Linux
$ git clone https://github.com/eminence/procfs.git
$ cd procfs
$ git checkout v0.17.0
$ RUST_BACKTRACE=1 cargo test -- --skip-exact \
      --skip=process::tests::test_proc_fd_count_runsinglethread \
      --skip sys::kernel::random::tests::test_write_wakeup_threshold \
      --skip process::tests::test_proc_auxv \
      --skip=process::tests::test_proc_status_for_kthreadd
(all tests pass or are skipped/ignored)

This is our “control” build.

Now, repeating in an aarch64 chroot, emulated via qemu-user-static:

$ RUST_BACKTRACE=1 cargo test -- --skip-exact […]
[…]
failures:

---- process::tests::test_proc_exe stdout ----
thread 'process::tests::test_proc_exe' panicked at procfs/src/process/tests.rs:275:5:
assertion `left == right` failed
  left: "/usr/bin/qemu-aarch64-static"
 right: "/procfs/target/debug/deps/procfs-daeabd3c8934810d"
stack backtrace:
   0: rust_begin_unwind
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:367:5
   4: procfs::process::tests::test_proc_exe
             at ./src/process/tests.rs:275:5
   5: procfs::process::tests::test_proc_exe::{{closure}}
             at ./src/process/tests.rs:271:19
   6: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- tests::test_cpuinfo stdout ----
None
thread 'tests::test_cpuinfo' panicked at procfs/src/lib.rs:684:33:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0: rust_begin_unwind   
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
   2: core::panicking::panic
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:148:5
   3: core::option::unwrap_failed
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/option.rs:2012:5
   4: core::option::Option<T>::unwrap
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/option.rs:972:21
   5: procfs::tests::test_cpuinfo
             at ./src/lib.rs:684:13
   6: procfs::tests::test_cpuinfo::{{closure}}
             at ./src/lib.rs:679:22
   7: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

---- process::tests::test_procinfo stdout ----
thread 'process::tests::test_procinfo' panicked at procfs/src/process/tests.rs:506:52:
called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidInput, error: "unable to parse input: Position(Space, [10])" }
stack backtrace:
   0: rust_begin_unwind   
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
   2: core::result::unwrap_failed
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/result.rs:1700:5
   3: core::result::Result<T,E>::unwrap
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/result.rs:1104:23
   4: procfs::process::tests::test_procinfo
             at ./src/process/tests.rs:506:25
   5: procfs::process::tests::test_procinfo::{{closure}}
             at ./src/process/tests.rs:494:19
   6: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


failures:
    process::tests::test_proc_exe
    process::tests::test_procinfo
    tests::test_cpuinfo   

test result: FAILED. 91 passed; 3 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1.27s

Okay, the tests::test_cpuinfo failure is already reported as #323, the process::tests::test_proc_exe failure seems to be specific to running in emulation, and I don’t know what to say about process::tests::test_procinfo. I didn’t see it in a “real” Fedora build, and none of these are what I am trying to report here.

Repeating the above on ppc64le gives the same result, except that tests::test_cpuinfo doesn’t fail – although it did fail on real ppc64le hardware.

---

Now, for the subject of this report, on s390x:

[…]
---- tests::test_kernel_stat stdout ----
KernelStats {
    total: CpuTime {
        user: 83030541,
        nice: 1456170471,
        system: 13524974,
        idle: 3154050570,
        iowait: Some(
            1543050,
        ),
        irq: Some(
            9041546,
        ),
        softirq: Some(
            866113,
        ),
        steal: Some(
            0,
        ),
        guest: Some(
            0,
        ),
        guest_nice: Some(
            0,
        ),
        tps: 100,
    },
    cpu_time: [
        CpuTime {
            user: 5197507,
            nice: 90828479,
            system: 836449,
            idle: 197307712,
            iowait: Some(
                74260,
            ),
            irq: Some(
                558475,
            ),
            softirq: Some(
                82892,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 4596389,
            nice: 91197844,
            system: 674628,
            idle: 197775509,
            iowait: Some(
                62124,
            ),
            irq: Some(
                548304,
            ),
            softirq: Some(
                44540,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 3530071,
            nice: 90147038,
            system: 742351,
            idle: 199879912,
            iowait: Some(
                41213,
            ),
            irq: Some(
                526190,
            ),
            softirq: Some(
                42161,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 8423397,
            nice: 91541602,
            system: 1349167,
            idle: 192570622,
            iowait: Some(
                268915,
            ),
            irq: Some(
                620041,
            ),
            softirq: Some(
                54695,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 4052862,
            nice: 90933503,
            system: 615937,
            idle: 198685170,
            iowait: Some(
                50074,
            ),
            irq: Some(
                536112,
            ),
            softirq: Some(
                38925,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 3940753,
            nice: 90860186,
            system: 586909,
            idle: 198930525,
            iowait: Some(
                26126,
            ),
            irq: Some(
                537879,
            ),
            softirq: Some(
                36786,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 4180428,
            nice: 91013747,
            system: 615564,
            idle: 198459494,
            iowait: Some(
                45197,
            ),
            irq: Some(
                556439,
            ),
            softirq: Some(
                38756,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 8331391,
            nice: 91488962,
            system: 1236338,
            idle: 192896853,
            iowait: Some(
                224505,
            ),
            irq: Some(
                612317,
            ),
            softirq: Some(
                53209,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 5169403,
            nice: 90916745,
            system: 843821,
            idle: 197260698,
            iowait: Some(
                75680,
            ),
            irq: Some(
                564525,
            ),
            softirq: Some(
                49986,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 10122500,
            nice: 91362720,
            system: 1804961,
            idle: 190432249,
            iowait: Some(
                359176,
            ),
            irq: Some(
                652213,
            ),
            softirq: Some(
                95041,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 4263470,
            nice: 91060333,
            system: 718679,
            idle: 198125533,
            iowait: Some(
                74775,
            ),
            irq: Some(
                565820,
            ),
            softirq: Some(
                87917,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 5419741,
            nice: 91275798,
            system: 862684,
            idle: 196556631,
            iowait: Some(
                95307,
            ),
            irq: Some(
                590100,
            ),
            softirq: Some(
                67471,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 3962060,
            nice: 90859637,
            system: 655854,
            idle: 198777343,
            iowait: Some(
                44810,
            ),
            irq: Some(
                547991,
            ),
            softirq: Some(
                62077,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 3457406,
            nice: 90381010,
            system: 744833,
            idle: 199720763,
            iowait: Some(
                38593,
            ),
            irq: Some(
                530356,
            ),
            softirq: Some(
                37793,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 4528047,
            nice: 91232485,
            system: 645171,
            idle: 197878344,
            iowait: Some(
                32209,
            ),
            irq: Some(
                554304,
            ),
            softirq: Some(
                37576,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 3855107,
            nice: 91070372,
            system: 591621,
            idle: 198793205,
            iowait: Some(
                30079,
            ),
            irq: Some(
                540472,
            ),
            softirq: Some(
                36280,
            ),
            steal: Some(
                0,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
    ],
    ctxt: 3912665812,
    btime: 1731761843,
    processes: 7953721,
    procs_running: Some(
        4,
    ),
    procs_blocked: Some(
        0,
    ),
}
thread 'tests::test_kernel_stat' panicked at procfs/src/lib.rs:577:9:
assertion `left == right` failed 
  left: 1
 right: 16
stack backtrace:
   0: rust_begin_unwind
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt 
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:367:5
   4: procfs::tests::test_kernel_stat
             at ./src/lib.rs:577:9
   5: procfs::tests::test_kernel_stat::{{closure}}
             at ./src/lib.rs:566:26
   6: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
   7: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[…]



failures:
    process::tests::test_proc_exe
    process::tests::test_procinfo
    tests::test_cpuinfo
    tests::test_kernel_stat

test result: FAILED. 90 passed; 4 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1.18s

The failures in process::tests::test_proc_exe, process::tests::test_procinfo, and tests::test_cpuinfo are consistent with those on aarch64 and pp664le; tests::test_kernel_stat is one of the failures I was trying to report in this issue. The output on real hardware is similar:

---- tests::test_cpuinfo stdout ----
None
thread 'tests::test_cpuinfo' panicked at src/lib.rs:683:34:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- tests::test_kernel_stat stdout ----
KernelStats {
    total: CpuTime {
        user: 16790930,
        nice: 3522,
        system: 3384651,
        idle: 233015124,
        iowait: Some(
            326403,
        ),
        irq: Some(
            107465,
        ),
        softirq: Some(
            185327,
        ),
        steal: Some(
            173507,
        ),
        guest: Some(
            0,
        ),
        guest_nice: Some(
            0,
        ),
        tps: 100,
    },
    cpu_time: [
        CpuTime {
            user: 5522045,
            nice: 917,
            system: 1138502,
            idle: 77678704,
            iowait: Some(
                103183,
            ),
            irq: Some(
                53849,
            ),
            softirq: Some(
                111186,
            ),
            steal: Some(
                58703,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 5627933,
            nice: 1041,
            system: 1124884,
            idle: 77670809,
            iowait: Some(
                113340,
            ),
            irq: Some(
                27253,
            ),
            softirq: Some(
                37545,
            ),
            steal: Some(
                57362,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
        CpuTime {
            user: 5640952,
            nice: 1563,
            system: 1121263,
            idle: 77665610,
            iowait: Some(
                109879,
            ),
            irq: Some(
                26363,
            ),
            softirq: Some(
                36595,
            ),
            steal: Some(
                57441,
            ),
            guest: Some(
                0,
            ),
            guest_nice: Some(
                0,
            ),
            tps: 100,
        },
    ],
    ctxt: 555939405,
    btime: 1733854463,
    processes: 13904974,
    procs_running: Some(
        3,
    ),
    procs_blocked: Some(
        0,
    ),
}
thread 'tests::test_kernel_stat' panicked at src/lib.rs:577:9:
assertion `left == right` failed
  left: 1
 right: 3

Unfortunately, I wasn’t able to reproduce the test_meminfo failure in emulation, but it looks like this on real hardware:

---- tests::test_meminfo stdout ----
Meminfo {
    mem_total: 26335961088,
    mem_free: 4957818880,
    mem_available: Some(
        25220423680,
    ),
    buffers: 1196032,
    cached: 19969449984,
    swap_cached: 16384,
    active: 5222580224,
    inactive: 14961217536,
    active_anon: Some(
        169172992,
    ),
    inactive_anon: Some(
        44261376,
    ),
    active_file: Some(
        5053407232,
    ),
    inactive_file: Some(
        14916956160,
    ),
    unevictable: Some(
        0,
    ),
    mlocked: Some(
        0,
    ),
    high_total: None,
    high_free: None,
    low_total: None,
    low_free: None,
    mmap_copy: None,
    swap_total: 8589930496,
    swap_free: 8589144064,
    dirty: 666693632,
    writeback: 0,
    anon_pages: Some(
        213184512,
    ),
    mapped: 278470656,
    shmem: Some(
        278528,
    ),
    slab: 830554112,
    s_reclaimable: Some(
        554332160,
    ),
    s_unreclaim: Some(
        276221952,
    ),
    kernel_stack: Some(
        15982592,
    ),
    page_tables: Some(
        44687360,
    ),
    secondary_page_tables: Some(
        0,
    ),
    quicklists: None,
    nfs_unstable: Some(
        0,
    ),
    bounce: Some(
        0,
    ),
    writeback_tmp: Some(
        0,
    ),
    commit_limit: Some(
        21757911040,
    ),
    committed_as: 579280896,
    vmalloc_total: 547608330240,
    vmalloc_used: 64446464,
    vmalloc_chunk: 0,
    hardware_corrupted: None,
    anon_hugepages: None,
    shmem_hugepages: None,
    shmem_pmd_mapped: None,
    cma_total: Some(
        0,
    ),
    cma_free: Some(
        0,
    ),
    hugepages_total: Some(
        0,
    ),
    hugepages_free: Some(
        0,
    ),
    hugepages_rsvd: Some(
        0,
    ),
    hugepages_surp: Some(
        0,
    ),
    hugepagesize: Some(
        1048576,
    ),
    direct_map_4k: Some(
        7832862720,
    ),
    direct_map_4M: None,
    direct_map_2M: None,
    direct_map_1G: None,
    hugetlb: Some(
        0,
    ),
    per_cpu: Some(
        20029440,
    ),
    k_reclaimable: Some(
        554332160,
    ),
    file_pmd_mapped: None,
    file_huge_pages: None,
    z_swap: Some(
        0,
    ),
    z_swapped: Some(
        0,
    ),
}
thread 'tests::test_meminfo' panicked at src/lib.rs:848:13:
assertion failed: meminfo.hardware_corrupted.is_some()

[musicinmybrain]

… and I don’t know what to say about process::tests::test_procinfo. I didn’t see it in a “real” Fedora build, and none of these are what I am trying to report here.

Actually, this is not quite right. We don’t have a rust-procinfo package in Fedora, so we patch out this test. It’s possible that it would fail in a “real” build after all if we had the necessary dependency.

[eminence]

I don't have access to an s390x environment (though if you have any suggestions about how to easily emulate it, let me know), so I'll need some help here.

For the test_cpuinfo failure, seems like this assertion failed:

assert_eq!(cpuinfo.num_cores(), stat.cpu_time.len());  

So can you please share your /proc/cpuinfo file and your /proc/stat file?

For the test_meminfo failure, it seems like this assertion failed:

assert!(meminfo.hardware_corrupted.is_some()); 

So can you please let me know if /proc/kpagecgroup exists and if you have CONFIG_MEMORY_FAILURE set in your kernel?

[musicinmybrain]

The following are from a “scratch” build in Fedora infrastructure on real s390x hardware:

/proc/cpuinfo

vendor_id       : IBM/S390
# processors    : 2
bogomips per cpu: 3331.00
max thread id   : 0
features	: esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt vxp2 nnpa sie 
facilities      : 0 1 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 47 48 49 50 51 52 53 54 57 58 59 60 61 64 65 69 71 72 73 74 75 76 77 78 80 81 82 129 130 131 133 134 135 138 139 140 146 147 148 150 151 152 155 156 165 192 193 194 196 197
cache0          : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1          : level=1 type=Instruction scope=Private size=128K line_size=256 associativity=8
cache2          : level=2 type=Unified scope=Private size=32768K line_size=256 associativity=16
cache3          : level=3 type=Unified scope=Shared size=262144K line_size=256 associativity=128
processor 0: version = FF,  identification = 525FA8,  machine = 3931
processor 1: version = FF,  identification = 525FA8,  machine = 3931
cpu number      : 0
physical id     : 0
core id         : 0
book id         : 0
drawer id       : 0
dedicated       : 0
address         : 0
siblings        : 1
cpu cores       : 1
version         : FF
identification  : 525FA8
machine         : 3931
cpu MHz dynamic : 5200
cpu MHz static  : 5200
cpu number      : 1
physical id     : 1
core id         : 1
book id         : 0
drawer id       : 0
dedicated       : 0
address         : 1
siblings        : 1
cpu cores       : 1
version         : FF
identification  : 525FA8
machine         : 3931
cpu MHz dynamic : 5200
cpu MHz static  : 5200

/proc/stat

cpu  8599144 3329 2041924 194091752 529229 91582 184450 106340 0 0
cpu0 4313747 2173 1032878 96965488 269437 56344 135127 52583 0 0
cpu1 4285397 1156 1009045 97126263 259791 35238 49322 53757 0 0
intr 195557018 0 0 0
ctxt 264408362
btime 1733857828
processes 5727211
procs_running 2
procs_blocked 0
softirq 234950034 0 31759716 5300 52874123 21733160 0 633316 51258070 9404 76676945

/proc/kpagecgroup

Exists,

-r--------. 1 root root 0 Dec 22 16:45 /proc/kpagecgroup

CONFIG_MEMORY_FAILURE

I’m not sure how to verify this at runtime, but it looks like we do have this enabled.

[musicinmybrain]

For those working on Fedora, installing mock and qemu-user-static is an easy way to emulate other architectures. I can test a package like:

$ fedpkg mockbuild --root fedora-rawhide-s390x

and I can do arbitrary interactive debugging in the chroot with:

$ mock -r fedora-rawhide-s390x -i git cargo rust vim-enhanced
$ mock -r fedora-rawhide-s390x --shell --enable-network

Using mock directly like this doesn’t require me to have an RPM .spec file or have a Fedora account, so this is a straightforward option if you have a machine or VM running Fedora.

Alternatively, you can use qemu-user-system to run a full s390x OS image under emulation. You can get a Fedora Server ISO for s390x at https://alt.fedoraproject.org/alt/. I think virt-manager also supports qemu-user-system emulation.

[QuLogic]

Podman also supports this (not sure if Docker does as well) so long as there's a container image available:

$ podman run --rm -it --arch s390x fedora:rawhide 
[root@79913ff0e543 /]# cat /proc/cpuinfo 
vendor_id       : IBM/S390
...

Unfortunately, being an emulated container, I think some of this information may be less-truthful than a real one. I get permission denied for /proc/kpagecgroup, but this is a rootless container, so that's probably expected.

[musicinmybrain]

I’m piggybacking on this issue to mention that, as of procfs 0.18.0, process::task::tests::test_task_runsinglethread fails on s390x in EPEL10 and EPEL9 flakily:

     Running `/builddir/build/BUILD/procfs-0.18.0/target/rpm/deps/procfs-25f48554f1a5b8cf _runsinglethread --test-threads 1`

running 2 tests
test process::task::tests::test_task_runsinglethread ... FAILED
test process::tests::test_proc_fd_count_runsinglethread ... ok

failures:

---- process::task::tests::test_task_runsinglethread stdout ----

thread 'process::task::tests::test_task_runsinglethread' (1098) panicked at src/process/task.rs:223:17:
Running
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    process::task::tests::test_task_runsinglethread

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 94 filtered out; finished in 0.52s

It does not fail in Fedora. The failure is flaky (and frequent, at perhaps roughly 25-50%) in both Fedora and EPEL releases.

The panic is coming from this line:

procfs/procfs/src/process/task.rs

Line 223 in 7d3a0cd

assert!(matches!(syscall, Syscall::Blocked { .. }), "{:?}", syscall);

With export RUST_BACKTRACE=1:

thread 'process::task::tests::test_task_runsinglethread' (1098) panicked at src/process/task.rs:223:17:
Running
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /builddir/build/BUILD/rustc-1.91.0-src/library/std/src/panicking.rs:698:5
   1: core::panicking::panic_fmt
             at /builddir/build/BUILD/rustc-1.91.0-src/library/core/src/panicking.rs:75:14
   2: procfs::process::task::tests::test_task_runsinglethread
   3: procfs::process::task::tests::test_task_runsinglethread::{{closure}}
             at ./src/process/task.rs:134:35
   4: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rustc-1.91.0-src/library/core/src/ops/function.rs:250:5
   5: core::ops::function::FnOnce::call_once
             at /builddir/build/BUILD/rustc-1.91.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I am not planning to investigate this further; I’ll just skip this test on these releases s390x. I can do experiments if someone wants more information, though.

[musicinmybrain]

I updated the preceding comment to note that the new failure is actually flaky rather than EPEL-specific.