Dropped fedora-coreos.stream annotation stalls upgrades for zincati < 0.0.31

[prestist]

Upgrades from older builds (Oct 2025) to recent builds (Feb 19+ 2026) are failing across all architectures in testing-devel stream with zincati errors.

kola-upgrade test failures:

• testing-devel s390x: 43.20251024.2.0 → 43.20260219.20.1 build
• testing-devel x86_64: 43.20251024.2.0 → 43.20260219.20.1 build
• testing-devel ppc64le: 43.20251024.2.0 → 43.20260219.20.1 build
• testing-devel aarch64: 43.20251024.2.0 → 43.20260219.20.1 build

All fail with timeout during upgrade - system never completes the staged reboot (stuck at "Waiting for reboot").

I believe this is a window of incompatibility caused by the migration from fedora-coreos.stream to com.coreos.stream metadata:

1. coreos-assembler PR #4440 (merged Feb 17, 2026) dropped the fedora-coreos.stream annotation from OCI manifests
2. zincati 0.0.30 (released Mar 17, 2025) introduced OCI manifest reading but ONLY supports fedora-coreos.stream
3. zincati 0.0.32 (released Feb 2, 2026) added backward compatibility to support BOTH keys

Window of incompatibility

Builds with zincati 0.0.30 or 0.0.31 (March 2025 - January 2026) cannot upgrade to builds from Feb 17+ 2026 that only have com.coreos.stream.

.ok_or_else(|| {
    anyhow!("Missing `fedora-coreos.stream` in base image manifest annotations")
})?

Related PRs

• coreos-assembler#4440: Dropped fedora-coreos.stream - drop uses of fedora-coreos.stream coreos-assembler#4440
• fedora-coreos-config#4015: Related config change - tests/upgrade: pick up stream from com.coreos.stream label fedora-coreos-config#4015
• zincati#1241: Introduced OCI manifest reading (zincati 0.0.30) - update_agent: support rebase to OCI pullspec zincati#1241
• zincati#1321: Added com.coreos.stream support (zincati 0.0.32) - rpm_ostree/cli_status: Support com.coreos.stream OCI config label for stream extraction zincati#1321

[prestist]

I think we need to create a barrier release in the update graph at the last build before the metadata change (e.g., 43.20260217.20.1) to prevent direct upgrades from zincati 0.0.30/0.0.31 builds.

[dustymabe]

Haven't looked into this deeply yet, but just wanting to note this week's testing release (43.20260217.2.1) had this transition:

zincati 0.0.30-4.fc43.x86_64 → 0.0.32-1.fc43.x86_64 

As a result of coreos/fedora-coreos-config@2d6a6e0 (initiated by me).

Also wanted to state that stable (43.20260202.3.1) still has zincati-0.0.30-4.fc43.x86_64 and also was built prior to coreos/coreos-assembler#4440 merging. So any potentials issues here I would not be introduced into stable.

[dustymabe]

ok I see the error now:

[  423.098370] zincati[1115]: [ERROR zincati::update_agent::actor] failed to stage deployment: Missing `fedora-coreos.stream` in base image manifest annotations

We should:

1. pause the rollout of testing. stable should be fine.
2. force a new build of testing and next using quay.io/coreos-assembler/coreos-assembler:v43.20260202.3.1 (see stable: new release on 2026-02-17 (43.20260202.3.1) fedora-coreos-streams#1194 (comment)). We could also revert drop uses of fedora-coreos.stream coreos-assembler#4440 but since we have a quay tag we might as well just use it so we don't have to wait on a build of the COSA container.
   • For the new build set the version manually to a .2 of the previous .1
3. do the whole release process again :(

I'm hoping since peoples systems are failing to upgrade they'll just pick up the new build when it's available and we don't require any manual intervention.

[dustymabe]

Another thing we will need to do (as @prestist touched on earlier); I see now we will need a barrier release for this.

So these testing and next releases will be a barrier and then the next stable will be too.

[jbtrystram]

Nice analysis @prestist. not sure how this one went through the upgrade tests without us catching it !

[dustymabe]

Nice analysis @prestist. not sure how this one went through the upgrade tests without us catching it !

It's because for production streams the upgrade tests run before we actually do any release (i.e. we want to test the upgrade is possible before releasing anything).

Some context in coreos/fedora-coreos-pipeline@49e05be and also we end up here in the code for the upgrade tests today.

A potential improvement here is that we re-run some upgrade tests after we merge the rollouts PR just to actually check there aren't any zincati issues. Preferably these would complete before the rollout starts.