Tracking: Maintenance of EPEL 10 minor versions

[praiskup]

The problem was discussed in #1424, the [local] Koji repository contains 10.0 string, and we'll need to move it to 10.1 at some point (maintenance hell). We should avoid this inconvenience.

Important reading first:

• https://discussion.fedoraproject.org/t/epel-10-proposal/44304

Related discussions:

• fedora-distro-aliases for "epel-next" - epel-next-all target alias fedora-distro-aliases#14
• Copr issue: The "stable" vs. "next" EPEL 10 repository variants in Copr? fedora-copr/copr#3469
• Packit issue: Epel 10.y mapping packit/packit#2438
• DNF4 plugin: TODO: open issue once we know what to do
• DNF5 plugin: TODO: open issue once we know what to do

[carlwgeorge]

I think we have two possible options we can take for EPEL 10 mock configs. Which option we choose affects how the koji local baseurl problem would be handled.

option 1

We fully embrace minor versions for EPEL and RHEL in mock-core-configs. This could look something like this:

templates/centos-stream-10.tpl
centos-stream-10-aarch64.cfg
centos-stream-10-ppc64le.cfg
centos-stream-10-s390x.cfg
centos-stream-10-x86_64.cfg

templates/rhel-10.0.tpl
rhel-10.0-aarch64.cfg
rhel-10.0-ppc64le.cfg
rhel-10.0-s390x.cfg
rhel-10.0-x86_64.cfg

templates/epel-10.tpl
centos-stream+epel-10-aarch64.cfg
centos-stream+epel-10-ppc64le.cfg
centos-stream+epel-10-s390x.cfg
centos-stream+epel-10-x86_64.cfg
epel-10-aarch64.cfg -> centos-stream+epel-10-aarch64.cfg  (symlink)
epel-10-ppc64le.cfg -> centos-stream+epel-10-ppc64le.cfg  (symlink)
epel-10-s390x.cfg   -> centos-stream+epel-10-s390x.cfg    (symlink)
epel-10-x86_64.cfg  -> centos-stream+epel-10-x86_64.cfg   (symlink)

templates/epel-10-branched.tpl
rhel+epel-10.0-aarch64.cfg
rhel+epel-10.0-ppc64le.cfg
rhel+epel-10.0-s390x.cfg
rhel+epel-10.0-x86_64.cfg

With this setup, every six months we would need to create new rhel and rhel+epel minor version configs. The koji local baseurl in epel-10-branched.tpl can use $releasever_minor to keep working without changes, but the one in epel-10.tpl will need to be modified at the same time to point to the latest minor version. That would avoid the need for a koji symlink, which @nirik discouraged here. We could include a step in the EPEL branching SOP to submit a PR to make these changes every six months to help mitigate the maintenance burden.

Users who need to build for EPEL minor version branches (e.g. epel10.0 instead of epel10) would also have to create a new epel-10.<minor>-<arch>.cfg symlink every six months to their preferred EPEL base distro (RHEL, Alma, etc), because that is what fedpkg will look for. The expected config for the plain epel10 branch can be part of mock-core-configs because that symlink will always point to centos-stream+epel-10-<arch>.cfg.

option 2

We partially ignore RHEL and EPEL minor versions in mock-core-configs. We intend to primarily target two minor versions at a time, somewhat like we do now with rhel+epel and centos-stream+epel-next. This could look something like this:

templates/centos-stream-10.tpl
centos-stream-10-aarch64.cfg
centos-stream-10-ppc64le.cfg
centos-stream-10-s390x.cfg
centos-stream-10-x86_64.cfg

templates/rhel-10.tpl
rhel-10-aarch64.cfg
rhel-10-ppc64le.cfg
rhel-10-s390x.cfg
rhel-10-x86_64.cfg

templates/epel-10.tpl
centos-stream+epel-10-aarch64.cfg
centos-stream+epel-10-ppc64le.cfg
centos-stream+epel-10-s390x.cfg
centos-stream+epel-10-x86_64.cfg
rhel+epel-10-aarch64.cfg
rhel+epel-10-ppc64le.cfg
rhel+epel-10-s390x.cfg
rhel+epel-10-x86_64.cfg
epel-10-aarch64.cfg -> centos-stream+epel-10-aarch64.cfg  (symlink)
epel-10-ppc64le.cfg -> centos-stream+epel-10-ppc64le.cfg  (symlink)
epel-10-s390x.cfg   -> centos-stream+epel-10-s390x.cfg    (symlink)
epel-10-x86_64.cfg  -> centos-stream+epel-10-x86_64.cfg   (symlink)

This setup avoids the sprawl of minor version config files and symlink. Sharing an epel-10.tpl file for both CentOS and RHEL would mean that we would be forced to use the koji symlink approach. We would definitely need to include this symlink management into the EPEL branching SOP.

Users who need to build for EPEL minor version branches will still need to create a new epel-10.<minor>-<arch>.cfg symlink every six months to their preferred EPEL base distro (RHEL, Alma, etc).

[praiskup]

I'm sorry for not getting back to this sooner. This is more complicated than I thought before :-) because we are adding two more floating targets that need regular bumps, not just one.

Now I think I understood the EPEL 10 proposal: Minor versions are still built against RHEL + EPEL (nah, not easy to replicate in non-Koji environment, i.e., local mock), and the "future/development minor" (intentionally not using the word "next") will be built against (public && nice) CentOS Stream + EPEL. Until now I was frozen in the old-school EPEL policy with only one EPEL minor...

Ad option 1:

The koji local baseurl in epel-10-branched.tpl can use $releasever_minor to keep working without changes,

I'm afraid this is not possible in Mock; or? The DNF (e.g.) on Fedora 40 host doesn't know the variable value before finishing dnf --installroot <root> (btw., older DNFs on older distros do not even implement that feature). It means we have to hardcode the minor number somewhere in the mock's config. This implies periodic version bumps in core configs 🤷

Also, I don't think we need to introduce RHEL minor configs, do you think it brings any real value? The thing is that mock-core-configs users have no access to pre-release RHEL packages, and very rarely (if at all) need current_minor -1. What they care about is the "current stable minor".

Ad option 2:
Took me a while to understand the difference in proposals, and I think I like this better. I don't think we have a need for multiple minor RHEL variants. But lemme introduce another variant with separated "epel" configurations...

option 3
Let's have two epel variants that we care about as "epel" (we don't want to use "next" to not mixup with EPEL 9 policy) and "epel-branched" (stable). Then the configuration would look like:

templates/centos-stream-10.tpl
centos-stream-10-x86_64.cfg
... we already have those ...

templates/rhel-10.tpl
rhel-10-x86_64.cfg
... this is RHEL CDN, will be available once RHEL 10 is GA ...

templates/epel-10.tpl  (references the "development/future" EPEL minor, it means "10.0" right now)
centos-stream+epel-10-x86_64.cfg (based on epel-10.tpl + centos-stream-10.tpl)
epel-10-x86_64.cfg (symlink to centos-stream+epel-10-x86_64.cfg, as that's what EPEL 10 is in Koji)

templates/epel-branched-10.tpl (references currently released EPEL minor)

rhel+epel-branched-10-x86_64.cfg (based on epel-branched-10.tpl + rhel-10.tpl)
alma+epel-branched-10-x86_64.cfg (if anyone cares adding those alternatives)
epel-10-branched-x86_64.cfg (symlink created by user upon the first execution, similar to the current epel-9 logic)

Simply put we care about these EPEL 10 variants:

- epel-branched mirrored ("stable" minor)
- epel-branched koji ("stable local" minor)
- epel mirrored ("development/future" minor)
- epel koji ("development/future local" minor)

Then, the MINOR "bump event" is not going to happen at the same time in "koji" and "mirrors". The stable "koji" repo is going to be branched first, and probably later composes will appear, and start getting mirrored. This is similar to Rawhide branching, and then - one may argue that in mock-core-configs we'll probably want to make the switch once the bumped composes are available and mirrored.

The major problem is that we have been doing this dance ~2 a year already for Rawhide, and now we'll have to do this ~2x more because EPEL. We have the symlinks for rawhide (good) but we have to keep more versions in parallel (EPEL is just two latest).

Now, I hear that at the time of branching new 10.2 from 10.1, 10.1 becomes "stable branched", but "previous stable" 10.0 becomes obsoleted and archived on mirrors. This means that the "current mock's" epel-10-branched config becomes staled and non-working till we provide the updated mock-core-configs release - and doing it promptly is always very stressful. Similar with providing up2date "devel" version after branching. A LOT has been done to make the rawhide branching fluent, yet we made a mistake during the last branching event (and we hope for the best for the next events).

So to me it is hard to imagine all of this happening without Koji symlinks, but also without the MirrorManager symlinks - and I mean for both the branched and development variants! With

• mirror manager "10" that redirects to the "current stable minor", e.g. "10.0" (currently impossible to provide, since there's no GA EPEL)
• mirror manager "10-devel" to redirect to "current devel minor", e.g. "10.0" right now
• koji symlink "10 -> stable_minor"
• koji symlink "10 -> development minor" (currently 10.0)

... with those, we can have Mock configs for epel and epel-branched provided once, and keep using them without periodic bumps.

[carlwgeorge]

I'm afraid this is not possible in Mock; or? The DNF (e.g.) on Fedora 40 host doesn't know the variable value before finishing dnf --installroot <root> (btw., older DNFs on older distros do not even implement that feature). It means we have to hardcode the minor number somewhere in the mock's config.

This is what we have the container bootstrap feature for, right?

Also, I don't think we need to introduce RHEL minor configs, do you think it brings any real value?

I do think it brings value. I've had to create my own custom RHEL minor configs in the past when troubleshooting build failures. It was usually trying to answer the question "does this fail to build on the previous minor version too, or is there something in the new minor version at fault". The RHEL CDN has minor versions directories (for all releases, not just EUS), and the current RHEL templates work with this just by setting $releasever to the full major.minor version. These configs would also be useful for people that want to build specifically for RHEL EUS releases (granted not every minor is an EUS). These configs would also make the integration with fedpkg quite simple. fedpkg needs to map dist-git branches (epel10, epel10.0, epel10.1, etc.) to mock configs somehow.

Let's have two epel variants that we care about as "epel" (we don't want to use "next" to not mixup with EPEL 9 policy) and "epel-branched" (stable).

I think we're on the right path with this part. epel-10.tpl would use just the major version, and epel-branched-10.tpl would use the major and minor version. I think you're right that we'll still need the koji repo symlink for the leading minor version, because CentOS doesn't know its minor version ($releasever_minor is undefined). But for rhel+epel we can use $releasever_minor to point to the the exact minor version repo. So something like:

epel-10.tpl:
...
[local]
baseurl=https://kojipkgs.fedoraproject.org/repos/epel10/latest/$basearch/
...

epel-branched-10.tpl:
...
[local]
baseurl=https://kojipkgs.fedoraproject.org/repos/epel10.$releasever_minor-build/latest/$basearch/
...

As for the rest of option 3, how would you feel about removing the word "branched" from the configs (but not the template)? So rhel+epel-10-x86_64.cfg would be based on rhel-10.tpl and epel-branched-10.tpl. My thinking is that EPEL on RHEL means one thing, and EPEL on CentOS means another. Having separate template files is fine, but in my opinion the configs that include them don't need to repeat the word "branched".

However my bigger concern with option 3 is I don't know how we're going to have fedpkg map the minor version dist-git branches to the mock configs in this setup. This isn't something a koji repo symlink would solve. I guess we could just rely on users to create their own symlinks, not only for the preferred base distro (RHEL, Alma, etc.) but for the minor version as well. Perhaps most EPEL maintainers will just focus on the leading minor version to avoid this hassle, which would work out reasonably well in the new model. It's expected that some EPEL packages will be available for CentOS users first, and deferred until the next minor version for RHEL users (a bit like most RHEL updates).

Now, I hear that at the time of branching new 10.2 from 10.1, 10.1 becomes "stable branched", but "previous stable" 10.0 becomes obsoleted and archived on mirrors. This means that the "current mock's" epel-10-branched config becomes staled and non-working till we provide the updated mock-core-configs release - and doing it promptly is always very stressful.

I don't think it would stop working. Archived EPEL content works like Fedora. The content gets archived, mirrormanager detects it and starts returning the archive mirror links for the same requests it previously returned regular mirror links for.

• mirror manager "10" that redirects to the "current stable minor", e.g. "10.0" (currently impossible to provide, since there's no GA EPEL)
• mirror manager "10-devel" to redirect to "current devel minor", e.g. "10.0" right now

Early on in the proposal we talked about having symlinks in the published repo tree (epel/10 -> epel/10.0, epel/centos10 -> epel/10.1), but then we found out that mirrormanager intentionally skips symlinks when scanning. Because of that, and the fact that CentOS doesn't know its minor version, we had to make the leading repo just plain epel/10. Other repos will have the minor version, and RHEL users will be pointed at a corresponding repo based on their $releasever_minor value. So effectively, "mirrormanager 10 latest (devel)" already exists (well, it returns mirror links, but they are currently broken), but "mirrormanager 10 non-latest (stable)" does not and likely will not exist. I think the best approach is going to be relying on $releasever_minor and the container bootstrap feature for rhel+epel configs.

• koji symlink "10 -> stable_minor"
• koji symlink "10 -> development minor" (currently 10.0)

I can create the symlink for the latest minor and put a step in our SOP for updating it, but I'd like to hold off on the second symlink to see if we can get things working with $releasever_minor.

So, based on all of this, I think our initial setup can look like this:

templates/centos-stream-10.tpl
centos-stream-10-x86_64.cfg

templates/epel-10.tpl  (mirror and koji links using `10`)
centos-stream+epel-10-x86_64.cfg (based on epel-10.tpl + centos-stream-10.tpl)
epel-10-x86_64.cfg (symlink to centos-stream+epel-10-x86_64.cfg)

And then when RHEL 10 is released, we add these:

templates/rhel-10.tpl
rhel-10-x86_64.cfg

templates/epel-branched-10.tpl (mirror and koji links using `10.$releasever_minor`)
rhel+epel-10-x86_64.cfg (based on epel-branched-10.tpl + rhel-10.tpl)

And just make sure we have an appropriate message about both choosing your base distro and your minor version via symlink if you try to build for a non-existent config such as epel-10.0-x86_64.cfg.

How does that sound as the plan of record? I'm happy to implement these stages as they become feasible.

[praiskup]

The DNF (e.g.) on Fedora 40 host doesn't know the variable value before finishing dnf --installroot (btw., older DNFs on older distros do not even implement that feature). It means we have to hardcode the minor number somewhere in the mock's config.

This is what we have the container bootstrap feature for, right?

Yes but not always. Not always possible to use podman container for initializing the bootstrap chroot, and the bootstrap chroot - if already initialized - provides new/good enough DNF version so we had no need so far to re-initialize it. Practically it means that if user caches bootstrap chroot with releasever_minor 0, it will always install target chroot 10.0 even though we should be building against 10.1.

This would be precedent (no chroot so far requires bootstrap image feature). And we haven't got a solution for #1289.

[praiskup]

I submitted/closed the issue by mistake. I'm sorry. Continuing:

I've had to create my own custom RHEL minor configs in the past when troubleshooting build failures. It was usually trying to answer the question "does this fail to build on the previous minor version too, or is there something in the new minor version at fault".

Agreed, seems useful sometimes - but this deserves a separate PR anyway. I think that we could provide some macro magic for these use-cases, if there's an actual demand. Something like this, but named %rhel_minor or so.

Having separate template files is fine, but in my opinion the configs that include them don't need to repeat the word "branched".

Sounds good! Eventually, we can keep just one epel-10.tpl if there's a reasonable if-else logic, it is hard to predict now.

But for rhel+epel we can use $releasever_minor to point to the the exact minor version repo. So something like:
.. epel10.$releasever_minor-build..

See the "mock bootstrap cache problem" above; this is going to work but not always. I think it is much safer to hard-code the releasever_minor number in mock configuration, and make the maintenance harder :-(

I don't know how we're going to have fedpkg map the minor version dist-git branches to the mock configs in this setup.

Is this something where fedora-distro-aliases would help? We can solve this separately.

As for the rest of option 3, how would you feel about removing the word "branched" from the configs (but not the template)? So rhel+epel-10-x86_64.cfg would be based on rhel-10.tpl and epel-branched-10.tpl. My thinking is that EPEL on RHEL means one thing, and EPEL on CentOS means another.

Sounds good. Is it worth documenting these nuances somewhere? e.g. here?

I don't think it would stop working. Archived EPEL content works like Fedora. The content gets archived, mirrormanager detects it and starts returning the archive mirror

👍

I can create the symlink for the latest minor and put a step in our SOP for updating it, but I'd like to hold off on the second symlink to see if we can get things working with $releasever_minor.

👍 The RHEL+EPEL config is indeed a matter of the (near, post GA) future.

How does that sound as the plan of record?

I'm not confident about the releasever_minor thing, at least if not hard-coded. But otherwise sounds good!

I'm happy to implement these stages as they become feasible.

Yes! Thank you.