update_agent: support rebase to OCI pullspec by jbtrystram · Pull Request #1241 · coreos/zincati

jbtrystram · 2024-12-04T14:04:10Z

Add a configuration knob in update to optionnaly use OCI pullspec instead of ostree checksums.

This will default to false.
Requires coreos/fedora-coreos-cincinnati#99 and coreos/rpm-ostree#5120

travier

Very quick look but looks good 👍🏻

src/rpm_ostree/cli_deploy.rs

tests/fixtures/00-config-sample.toml

jlebon

Let's cross-link to coreos/fedora-coreos-tracker#1823 in the commit message for the larger context.

tests/fixtures/00-config-sample.toml

src/cincinnati/mod.rs

src/rpm_ostree/cli_deploy.rs

jbtrystram

I updated this to use ostree-remote-image so we validate the ostree signatures, as suggested in coreos/fedora-coreos-tracker#1823 (comment)

I think this is ready for review now. I will clean up and sqash commits once the design is agreed

src/rpm_ostree/cli_status.rs

src/rpm_ostree/cli_deploy.rs

src/rpm_ostree/cli_status.rs

jbtrystram · 2025-01-13T14:36:54Z

OK, had some more discussion with Jlebon and travier today and there are some more things to flesh out

src/rpm_ostree/cli_status.rs

src/rpm_ostree/mod.rs

jbtrystram · 2025-01-15T17:07:39Z

Allright, I think this is ready for review now. I highlighted a couple of things I am unsure of.
I will squash the commits after addressing review comments

Switch boot images to use OCI for updates. This is a step towards bootable containers support and bootc rebase. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates Requires coreos/zincati#1241 See coreos/fedora-coreos-tracker#1823

src/cincinnati/client.rs

src/cincinnati/mod.rs

src/rpm_ostree/cli_deploy.rs

src/rpm_ostree/cli_status.rs

src/cincinnati/mod.rs

src/rpm_ostree/actor.rs

tests/fixtures/rpm-ostree-oci-status.json

jlebon · 2025-01-28T21:56:47Z

src/rpm_ostree/cli_finalize.rs

+    // get the latest commit but that would be racy, so let's finalize the latest
+    // commit.
+    if release.is_oci {
+        cmd.arg("--allow-missing-checksum")


I think probably the cleaner thing is to have rpm-ostree accept a digested pullspec the way it accepts commit checksums and it can verify that it matches the digested pullspec that was staged. Then we can drop this.

Yes, definitely, but that requires work in rpm-ostree, let's not block on it ?

src/rpm_ostree/mod.rs

src/update_agent/actor.rs

jbtrystram · 2025-02-05T15:52:36Z

Allright, I reworked this PR a bunch following the review comments, that ended up being pretty substantial changes.

Overview :

A local deployment is now unserialized with an enum containing either a Checksum or a Pullspec. Both are simply strings, but I use that to match my way around, rather than a is_oci boolean.
custom-origin fields are now purely cosmetic : Zincati will cue from the container-image-reference.
Now properly parses the OCI manifest output from rpm-ostree --json and get the fedora-coreos.stream key from that. This is working around rpm-ostree status --json does not properly serialize ostree.manifest for OCI deployments rpm-ostree#5196
imported a couple of new crates : oci-spec for the above issue and ostree-rs to properly parse ostree image references and containers pull specs. I removed the hacky split(':') and split(":sha256")` in favor of that.

Testing

Here are my notes to test this out it case someone wants to experiment:

Start a VM with Zincati disabled : cosa run --qemu-image fedora-coreos-41.20241215.2.0-qemu.x86_64.qcow2 --add-ignition noautoupdate

Rebase to an OCI image in graph (recent enough to get rpm-ostree-2024.9) :

    sudo rpm-ostree rebase ostree-remote-image:fedora:registry:quay.io/fedora/fedora-coreos@sha256:74448159fae255797012a12eaf9089ea3c1018ffb0b3e76e03483cba59b50bb2 \
     --custom-origin-url=quay.io/fedora/fedora-coreos@sha256:74448159fae255797012a12eaf9089ea3c1018ffb0b3e76e03483cba59b50bb2 \
     --custom-origin-description="Fedora CoreOS testing stream"
     sudo reboot

Now import Zincati build with this PR then:

    sudo rm /etc/zincati/config.d/90-disable-auto-updates.toml
    sudo systemctl restart zincati

Zincati should now pick up an OCI update through the OCI graph

Getting zincati custom build in the COSA VM

While we can make a custom FCOS build with an override to get our Zincati build inside, it will be erased by the content of the OCI image with a rebase, so here is how I did, after building zincati and copy the binary into $COSA_DIR/tmp

rpm-ostree usroverlay
cp /mnt/workdir-tmp/zincati /usr/libexec/zincati
cp /mnt/workdir-tmp/zincati.rules /usr/share/polkit-1/rules.d/zincati.rules

# restart polkit
systemctl restart polkit.service

# re enable updates
rm /etc/zincati/config.d/90-disable-auto-updates.toml

# restart zincati
systemctl restart zincati.service

# Show the update
journalctl --no-pager -o cat -u zincati --follow

In the meantime, I'll squash this and try to make commits that make sense :)

jlebon

To be clear again though, this will make #498 much more visible once we switch over to this. Ideally, we fix that wart before switching production nodes over.

OK, whipped up #1259 for now.

src/rpm_ostree/cli_finalize.rs

jbtrystram · 2025-02-27T09:05:39Z

Ok I just re-did the testing, OCI and regular OSTree updates works as expected.
I think we are good to merge this if there are no further comments on the code

Add support for local OCI deployements, no longer erroring out. Use the `ostree-ext` crate to parse the ostree image reference properly. This will allow zincati to query the OCI graph and rebase to OCI images for updates. Also change the base-commit-meta object to support the OCI case where this is actually an escaped serialized OCI manifest. Use the `oci-spec` crate to deserialize it and extract the Fedora CoreOS stream info from the base OCI annotations. This works around coreos/rpm-ostree#5196

Local deployements are now unserialized with a enum being either `Checksum` or `Pullspec`. This will cue the cincinnati client to request the OCI graph in the pullspec case. [1] The custom-origin-description, if set, is passed along the rebase call, and the custom-url is updated to match the new pullspec. [2] Finally, update the polkit policy to allow Zincati to do a rebase operation. [1] Requires coreos/fedora-coreos-cincinnati#99 and coreos/rpm-ostree#5120

In the previous implementation, we finalize what's staged without knowing because rpm-ostree deploy does not returns the staged commit. Let's query the deployments and get the checksum that matches the pullspec we have in the update payload.

When going through the update graph, Zincati searches for a node that matches the current deployement, as the starting point for the update. If the graph does not contains a matching node, zincati errors silently. While this should almost never happen, it's useful to print a warning to explain the situation if we hit that case.

jlebon

Nice work!

in 32d76f6 we added ostree bindings to parse image references, which in turn requires glib-devel dependencies to build. Let's use the fcos-buildroot container to run the CI so we have to maintain the correct set of dependencies in one place.

jbtrystram · 2025-02-27T15:07:41Z

I just pushed because of an earlier unfortunate cargo update that ran into #1176
Now there should not be any changes in the lockfile other than what was added with ostree :)

jlebon · 2025-02-27T15:26:27Z

I just pushed because of an earlier unfortunate cargo update that ran into #1176

Opened #1260 for that one. Let's try to get the dependency bumps into this release.

Switch boot images to use OCI for updates. This is a step towards bootable containers support and bootc rebase. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates Requires coreos/zincati#1241 See coreos/fedora-coreos-tracker#1823

To simplify testing for coreos/fedora-coreos-tracker#1823 This script write a Zincati status override containing a fake rpm-ostree status output to appear like it's on an OCI deployement. Zincati will look under the well-known path of /run/zincati/booted-status-override.json. The content will trigger the OCI code path in Zincati. This will later run as ExecStartPre in the zincati.service environment Just ship the migration script for now, without the zincati service changes, to allow testing. See coreos/fedora-coreos-tracker#1890 See coreos/fedora-coreos-tracker#1823 (comment) Requires zincati 0.0.30 Ref coreos/zincati#1273 Ref coreos/zincati#1241

jbtrystram force-pushed the oci_scheme branch 2 times, most recently from f0713e1 to 1ca8a87 Compare December 4, 2024 14:18

jbtrystram changed the title ~~update_agent: support rebaseing to OCI pullspec~~ update_agent: support rebase to OCI pullspec Dec 4, 2024

travier reviewed Dec 4, 2024

View reviewed changes

cgwalters reviewed Dec 4, 2024

View reviewed changes

src/rpm_ostree/cli_deploy.rs Outdated Show resolved Hide resolved

dustymabe reviewed Dec 9, 2024

View reviewed changes

tests/fixtures/00-config-sample.toml Outdated Show resolved Hide resolved

jlebon reviewed Dec 10, 2024

View reviewed changes

tests/fixtures/00-config-sample.toml Outdated Show resolved Hide resolved

src/cincinnati/mod.rs Outdated Show resolved Hide resolved

src/cincinnati/mod.rs Show resolved Hide resolved

src/rpm_ostree/cli_deploy.rs Outdated Show resolved Hide resolved

jbtrystram force-pushed the oci_scheme branch 4 times, most recently from 11aa880 to 7000352 Compare December 16, 2024 13:51

jbtrystram commented Jan 13, 2025

View reviewed changes

src/rpm_ostree/cli_status.rs Outdated Show resolved Hide resolved

src/rpm_ostree/cli_deploy.rs Outdated Show resolved Hide resolved

src/rpm_ostree/cli_status.rs Show resolved Hide resolved

jbtrystram marked this pull request as draft January 13, 2025 14:36

jbtrystram commented Jan 15, 2025

View reviewed changes

src/rpm_ostree/cli_status.rs Outdated Show resolved Hide resolved

jbtrystram commented Jan 15, 2025

View reviewed changes

src/rpm_ostree/mod.rs Outdated Show resolved Hide resolved

jbtrystram requested review from cgwalters and jlebon January 15, 2025 17:07

jbtrystram marked this pull request as ready for review January 15, 2025 17:20

jbtrystram mentioned this pull request Jan 16, 2025

Move from OSTree to OCI for updates of new systems coreos/fedora-coreos-tracker#1823

Closed

jbtrystram force-pushed the oci_scheme branch from 90c5b25 to 450550b Compare January 23, 2025 09:59

jbtrystram mentioned this pull request Jan 23, 2025

[testing] deploy container images by default coreos/fedora-coreos-config#3333

Closed

cgwalters mentioned this pull request Jan 23, 2025

parse OCI deployment status into ImageManifest coreos/rpm-ostree#5208

Closed

jbtrystram force-pushed the oci_scheme branch 2 times, most recently from add9bf7 to c5e0459 Compare January 24, 2025 10:59

jlebon reviewed Jan 28, 2025

View reviewed changes

jbtrystram mentioned this pull request Jan 30, 2025

rust/rpmostree-client: parse OCI deployment manifest coreos/rpm-ostree#5266

Closed

jlebon reviewed Feb 26, 2025

View reviewed changes

src/rpm_ostree/cli_finalize.rs Outdated Show resolved Hide resolved

jbtrystram force-pushed the oci_scheme branch 3 times, most recently from 20f465a to 1eddd79 Compare February 27, 2025 11:28

apply clippy suggestions

3a4d475

jbtrystram force-pushed the oci_scheme branch 2 times, most recently from 3519a37 to 950aa08 Compare February 27, 2025 15:05

jbtrystram added 4 commits February 27, 2025 16:06

jlebon approved these changes Feb 27, 2025

View reviewed changes

jbtrystram force-pushed the oci_scheme branch from 950aa08 to 5b79f15 Compare February 27, 2025 15:06

jbtrystram enabled auto-merge February 27, 2025 16:17

jbtrystram merged commit 293df0f into coreos:main Feb 27, 2025
8 checks passed

jbtrystram mentioned this pull request Feb 27, 2025

Initial support for container image updates #878

Closed

jlebon mentioned this pull request Mar 6, 2025

leverage container-image-reference-digest from rpm-ostree for OCI updates #1272

Closed

prestist mentioned this pull request Feb 19, 2026

Dropped fedora-coreos.stream annotation stalls upgrades for zincati < 0.0.31 coreos/fedora-coreos-tracker#2112

Closed

Conversation

jbtrystram commented Dec 4, 2024

Uh oh!

travier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbtrystram left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbtrystram commented Jan 13, 2025

Uh oh!

Uh oh!

Uh oh!

jbtrystram commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jlebon Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

jbtrystram Feb 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jbtrystram commented Feb 5, 2025

Overview :

Testing

Getting zincati custom build in the COSA VM

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jbtrystram commented Feb 27, 2025

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

jbtrystram commented Feb 27, 2025

Uh oh!

jlebon commented Feb 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jbtrystram commented Jan 15, 2025 •

edited

Loading