GLib-CRITICAL logs on update

[p5]

Today, when updating, I started receiving what appear to be error logs in the "Deploying" stage of an update.
The update is still successful, so no functionality has been broken.

I'm using bootc v1.1.2 on Fedora 41.

❯ sudo bootc update 
layers already present: 42; layers needed: 29 (1.8 GB)
Fetched layers: 1.68 GiB in 13 minutes (2.28 MiB/s)                                                                                                                                                                                          ⠁ Deploying                                                                                                                                                                                                                                  
(bootc:26124): GLib-CRITICAL **: 12:56:48.149: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.158: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.158: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.158: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.164: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.164: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.164: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.165: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
⠁ Deploying                                                                                                                                                                                                                                  
(bootc:26124): GLib-CRITICAL **: 12:56:48.304: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.305: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.313: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
⠉ Deploying                                                                                                                                                                                                                                  
(bootc:26124): GLib-CRITICAL **: 12:56:48.525: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.526: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.526: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.547: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.549: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
⠙ Deploying                                                                                                                                                                                                                                  
(bootc:26124): GLib-CRITICAL **: 12:56:48.565: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.567: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
(bootc:26124): GLib-CRITICAL **: 12:56:48.568: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
  Deploying: done (11 seconds)                                                                                                                                                                                                               Pruned images: 0 (layers: 29, objsize: 431.8 MB)
Queued for next boot: ostree-image-signed:docker://ghcr.io/rsturla/eternal-linux/lumina:stable-nvidia-open
  Version: 250205
  Digest: sha256:31fd1bb124c34b90e8b5675c4a24a68d87f6ffcdf2468d6139d3cba485476949
Total new layers: 71    Size: 4.3 GB
Removed layers:   78    Size: 5.1 GB
Added layers:     70    Size: 4.3 GB

Edit:
I noticed a new version coming down in todays update - v1.1.4-6, so will try updating with this version and report back if it's still happening

[p5]

Unfortunately with v1.1.4, the same logs are being shown.

[cgwalters]

It's probably something else that changed, either ostree or glib2

[cgwalters]

Something similar to this came up in ostreedev/ostree#3303 - could your base image also be pulling in that "hardened allocator"?

[p5]

Will try and figure out!

My image doesn't use Universal Blue as a base, and I've not explicitly tried to pull in the hardened allocator. But that's not to say a dependency hasn't decided to pull this.

In terms of base images, Bazzite uses quay.io/fedora-atomic-desktops/ but I'm pulling the quay.io/fedora/fedora-silverblue images

[p5]

I'll be honest, I have no idea what a "hardened allocator" is.

But I've done a quick check of any "malloc" packages are installed on the system, and can see "jemalloc" is present. This is not present on the fedora-bootc or centos-bootc images which haven't reported the problem.

This "jemalloc" package is being pulled through inside the base quay.io/fedora/fedora-silverblue image.

❯ rpm -qa | grep malloc
jemalloc-5.3.0-7.fc41.x86_64

~ 
❯ podman run --rm -it quay.io/centos-bootc/centos-bootc:stream10 /bin/bash
bash-5.2# rpm -qa | grep malloc
bash-5.2# 

---

This may be completely irrelevant. But it sounded similar.

[RoyalOughtness]

@p5 He's referring to hardened_malloc, which secureblue uses.

We ran into this (and still run into this) when rebasing to a rechunked image (or an image that uses a rechunked base image) while using hardened_malloc. The issue @cgwalters linked links to secureblue/secureblue#369

Our mitigation has been and still is to simply not rechunk, but the underlying memory bug is still present, as your error logs demonstrate. The difference being that when hardened_malloc is in use, the ostree memory bug (arguably correctly) is treated as fatal. The bug is masked by most allocators which permit write-after-free, among other things:

Aug 12 15:49:46 localhost rpm-ostree[13779]: g_atomic_ref_count_dec: assertion 'old_value > 0' failed
Aug 12 15:49:46 localhost rpm-ostree[13779]: fatal allocator error: detected write after free