`test_non_ostree` failure with 1.1.6 on f40

[jeckersb]

From the packit PR -> aarch64 build log

failures:
---- test_non_ostree stdout ----
thread 'test_non_ostree' panicked at ostree-ext/tests/it/main.rs:1337:5:
assertion `left == right` failed
  left: Digest { algorithm: Sha256, value: "sha256:898c9219d1a94659edb77a7fc8cfb9c1b23dcab3bd2f4c74c9b618d9dd979d2f", split: 6 }
 right: Digest { algorithm: Sha256, value: "sha256:3d56a2320917879cfa94425fbba9120bad584b07ecf4722be03652865f6cf6cc", split: 6 }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
    test_non_ostree
test result: FAILED. 25 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.75s
error: test failed, to rerun pass `-p ostree-ext --test it`

[jeckersb]

Not going to dig much more into this now, but as a sanity check this at least seems to work running the tests inside of a f40 container on my laptop:

running 1 test
test test_non_ostree ... ok

[cgwalters]

I thought this was because we only had an old ostree in f40 which predated ostreedev/ostree#3346 but now I see we do have 2025.1 in that run so I'm unsure.

[cgwalters]

It'd be cool to figure this out but, in the end F40 is going to go EOL on Tue 2025-05-13 which isn't far away.

[jeckersb]

It'd be cool to figure this out but, in the end F40 is going to go EOL on Tue 2025-05-13 which isn't far away.

Yeah that's mostly why I didn't invest much time in it, but I wanted to note it incase it ends up being part of some bigger problem. Things are fine on F41/F42 though so I suspect this is just something we can ignore.

[cgwalters]

Hmm though this issue also just happened on a new PR to git main targeting centos stream 9: https://download.copr.fedorainfracloud.org/results/packit/containers-bootc-1178/centos-stream-9-x86_64/08739956-bootc/builder-live.log.gz

I wonder if the real problem is there's some race condition here...it's certainly possible. I fixed one example of that over in coreos/rpm-ostree@9df317a (umask == global state == evil)

[jeckersb]

I did try one more thing yesterday that I didn't write down here but I should have. But it was weird, and may be indicative of a race somewhere.

I tried doing a local build with the packit PR patch applied on the f40 branch via fedpkg mockbuild and that worked fine. Then I did fedpkg build --scratch --srpm to do the same thing but in koji, and that failed on x86_64. Which really those two operations should be mostly equivalent.

Koji task: https://koji.fedoraproject.org/koji/taskinfo?taskID=129905919

[cgwalters]

Now that I actually look slightly more closely, it's not an ostree checksum, it's a container manifest digest.

So probably the thing to do here is add some debug logic in the code that diffs the old/new manifest.