write validation to let us know when packages in `next-devel` are older than `testing-devel`

[dustymabe]

The idea here is that we never want someone upgrading to have package downgrades (unless we explicitly decided to do that). In the period of time where next is on the N+1 Fedora if packages in that stream sort older (NVR) than what is in the testing streams things are setting up such that packages will "downgrade" at some point as part of an update. This "downgrades" are common in the lead up to the release of the next Fedora major version because there are periods of "freeze" for N+1, but N continues rolling.

One way to get in front of this is to make sure packages in next-devel always NVR sort newer than packages in testing-devel.

Let's write a bot that will let us know when this happens.

Today I use the following steps to determine which packages are downgraded:

podman run -i --rm registry.fedoraproject.org/fedora:36 <<EOF
dnf install -y fedora-repos-ostree ostree rpm-ostree
mkdir /tmp/repo
cd /tmp/repo
ostree init --mode=archive --repo=.
cat /etc/ostree/remotes.d/fedora-compose.conf >> ./config
ostree pull --commit-metadata-only --depth=0 fedora-compose:fedora/x86_64/coreos/testing-devel
ostree pull --commit-metadata-only --depth=0 fedora-compose:fedora/x86_64/coreos/next-devel
rpm-ostree --repo=. db diff fedora-compose:fedora/x86_64/coreos/testing-devel fedora-compose:fedora/x86_64/coreos/next-devel
EOF

[dustymabe]

For example, as of today the output includes:

Downgraded:
  ethtool 2:5.17-1.fc35 -> 2:5.16-2.fc36
  runc 2:1.1.1-1.fc35 -> 2:1.1.0-2.fc36
  squashfs-tools 4.5.1-1.fc35 -> 4.5-18.20220221gitbc0c097.fc36
  vim-data 2:8.2.4701-1.fc35 -> 2:8.2.4621-1.fc36
  vim-minimal 2:8.2.4701-1.fc35 -> 2:8.2.4621-1.fc36

So I'll go through and find updates for each of these and then add them as overrides to next-devel. For example for vim we can find the latest packages NVR in bodhi via the bodhi CLI (dnf install bodhi):

$ bodhi updates query --releases f36 --packages vim
*vim-8.2.4701-1.fc36                      rpm        testing   2022-04-07 (6)
*vim-8.2.4621-1.fc36                      rpm        stable    2022-03-29 (15)
 vim-8.2.4579-1.fc36                      rpm        unpushed  2022-03-17 (26)
 vim-8.2.4529-1.fc36                      rpm        unpushed  2022-03-09 (34)
 vim-8.2.4485-1.fc36                      rpm        unpushed  2022-03-01 (42)
*vim-8.2.4460-1.fc36                      rpm        unpushed  2022-02-24 (47)
 vim-8.2.4428-1.fc36                      rpm        stable    2022-02-21 (51)
 vim-8.2.4386-1.fc36                      rpm        stable    2022-02-15 (57)
 vim-8.2.4314-1.fc36                      rpm        stable    2022-02-10 (62)
 vim-8.2.4232-1.fc36                      rpm        stable    2022-01-27 (75)
 vim-8.2.4198-1.fc36                      rpm        stable    2022-01-24 (79)
 vim-8.2.4068-1.fc36                      rpm        stable    2022-01-12 (90)
 vim-8.2.4051-1.fc36                      rpm        stable    2022-01-10 (93)
 vim-8.2.4006-1.fc36                      rpm        stable    2022-01-05 (98)
 vim-8.2.3811-1.fc36                      rpm        stable    2021-12-15 (119)
 vim-8.2.3755-1.fc36                      rpm        stable    2021-12-07 (127)
 vim-8.2.3717-1.fc36                      rpm        stable    2021-12-02 (132)
 vim-8.2.3642-2.fc36                      rpm        stable    2021-11-26 (138)
 vim-8.2.3642-1.fc36                      rpm        stable    2021-11-22 (142)
 vim-8.2.3582-1.fc36                      rpm        stable    2021-11-08 (156)

And we can find the update URL from that:

$ bodhi updates query --releases f36 --builds vim-8.2.4701-1.fc36
================================================================================
     vim-8.2.4701-1.fc36
================================================================================
   Update ID: FEDORA-2022-44f5b2df35
Content Type: rpm
     Release: Fedora 36
      Status: testing
        Type: security
    Severity: urgent
       Karma: 5
   Autokarma: True  [-3, 3]
    Autotime: True
   CI Status: All required tests passed
All required tests passed
     Request: stable
        Bugs: 2070437 - CVE-2022-1160 vim: heap buffer overflow in
            : get_one_sourceline
            : 2070438 - CVE-2022-1160 vim: heap buffer overflow in
            : get_one_sourceline [fedora-all]
            : 2073013 - CVE-2022-1154 vim: use after free in utf_ptr2char
            : 2073014 - CVE-2022-1154 vim: use after free in utf_ptr2char
            : [fedora-all]
       Notes: The newest upstream commit
            : Security fix for CVE-2022-1160
            : Security fix for CVE-2022-1154
   Submitter: zdohnal
   Submitted: 2022-04-07 09:30:32
    Comments: bodhi - 2022-04-07 09:30:33 (karma 0)
              This update has been submitted for testing by zdohnal.
              bodhi - 2022-04-07 09:30:34 (karma 0)
              This update's test gating status has been changed to 'waiting'.
              bodhi - 2022-04-07 10:04:44 (karma 0)
              zdohnal edited this update.
              bodhi - 2022-04-07 12:37:28 (karma 0)
              This update's test gating status has been changed to 'failed'.
              bodhi - 2022-04-07 13:33:15 (karma 0)
              zdohnal edited this update.
              bodhi - 2022-04-07 13:33:17 (karma 0)
              This update's test gating status has been changed to 'waiting'.
              bodhi - 2022-04-07 15:34:03 (karma 0)
              This update's test gating status has been changed to 'failed'.
              bodhi - 2022-04-07 18:02:14 (karma 0)
              This update has been pushed to testing.
              bodhi - 2022-04-07 18:31:23 (karma 0)
              This update's test gating status has been changed to 'passed'.
              nixuser - 2022-04-07 23:20:39 (karma 1)
              This is working fine.
              geraldosimiao - 2022-04-08 13:54:33 (karma 1)
              bodhi - 2022-04-08 13:56:12 (karma 0)
              This update can be pushed to stable now if the maintainer wishes
              atim - 2022-04-08 14:28:40 (karma 1)
              bodhi - 2022-04-08 14:28:41 (karma 0)
              This update has been submitted for stable by bodhi.
              andilinux - 2022-04-09 21:14:41 (karma 1)
              works good
              lruzicka - 2022-04-12 12:36:37 (karma 1)
              Works brilliantly.

  https://bodhi.fedoraproject.org/updates/FEDORA-2022-44f5b2df35

1 updates found (1 shown)

Now we just need to run it through overrides.py to update the overrides manifest:

./ci/overrides.py fast-track -r https://github.com/coreos/fedora-coreos-tracker/issues/1101#issuecomment-1085987923 https://bodhi.fedoraproject.org/updates/FEDORA-2022-44f5b2df35

NOTE: you can pass more than one bodhi URL to overrides.py so you can collect all the bodhi URLs first before running overrides.py.

[bgilbert]

If you can reduce that process entirely to code, it would make sense to add a periodic GitHub Actions workflow that submits PRs, based on add-override.yml.

[dustymabe]

Exactly. This was the start of that. I wanted to get my steps down and then someone (or maybe me) can pick it up and push it forward. This was step 1.

[prestist]

This might not even be a bot, it could be a check added to the pipeline which warns if an older package is detected.

This could even be a test that is in the config repo.
(backlog grooming discussion)

[marmijo]

I'm attempting to write some automation for this but I've ran into a couple of scenarios that need to be addressed:

• overrides.py fails with some bodhi updates due to multi-arch components so it will probably need to be updated to handle that case.
• sometimes bodhi updates are not available for the F(n+1) release because the package maintainer hasn't submitted one yet. This could be handled as a warning output in automation, but eventually we'll need a human-in-the-loop (HITL) to resolve that issue.

What other scenarios could be a challenge?

What I've been doing right now is a mixture of automation and HITL. I'm running a script that performs the steps outlined by Dusty to produce a list of packages that are showing as downgraded along with their bodhi updates for F(n+1). I then have to manually update the manifest lock files or call overrides.py manually.

[dustymabe]

Nice. Thanks for working on this. Yes if we could update overrides.py to handle packages that are specific to particular architectures that would cause us much less pain not only for this but normal overrides when we do them.

Warnings sound good to me when there are cases that need extra attention.

One other case we migth want to flag is when the update we are fast-tracking isn't currently in the testing->stable state. We'd prefer to only fast-track packages that are in that state to know we're not shipping something that won't be in the rest of fedora eventually.