EL8 osg-wn containers

[smuzaffar]

I noticed that now EL8 osg-wn containers are built based on CentOS Stream8 (CS8) ( #9 ). We use osg-wn containers for CMS grid jobs and planning to move to EL8 as production architecture for CERN's LHC Run3 (which will end in 2026). I am wondering how EL8 osg-wn containers (based on CS8) will get security updates and other fixes after the EOL of CS8 (31st May 2024)?

FYI @dpiparo

[brianhlin]

Hi @smuzaffar, I imagine that we'll treat the CS8 EOL in the same way that we handled the CentOS 6 EOL: align the OSG Software support dates such that we do not support unsupported operating systems

[smuzaffar]

Thanks @brianhlin , so basically no updates or support after CS8 EOL :-)
As CentOS stream is a rolling release which gets updates every week, how often you are planning to rebuild osg-wn images?

[smuzaffar]

By the way, have you consider moving to one of the new RHEL clones (Alma Linux, Rocky Linux) which are 1:1 binary compatible with RHEL?

[bbockelm]

Yup - we keep abreast of the various clones.

In fact, we are looking at the idea of "build on Rocky, support on CentOS Stream" -- this way we build against the older ABI but support folks on a wider range of platforms. We've already seen a case (XRootD) where compiling against CS8 linked a library against a new shared lib ABI version which hadn't made it to RHEL yet - meaning it wasn't installable on Linux.

Honestly, I wouldn't worry too much about this right now. As @brianhlin says we'll align the OSG 3.x EOL with CS8; however, if LHC is still using EL8 because it's in the middle of Run 3, we'll have OSG 3.{x+1} support one of the clones. The impact CMS would see is having to switch to a different tag of the same container.

[brianhlin]

As CentOS stream is a rolling release which gets updates every week, how often you are planning to rebuild osg-wn images?

Right now we build on each OSG Software release but I think it'd be reasonable to rebuild weekly similar to many of our other containers.

[smuzaffar]

In fact, we are looking at the idea of "build on Rocky, support on CentOS Stream" -- this way we build against the older ABI but support folks on a wider range of platforms.

thanks @bbockelm , we are also looking in to building CMSSW for one of the RHEL clone which should allow us to run it on RHEL and CentOS Stream . We have tested CMSSW build on AlmaLinux and RH UBI and tests are very positive. Any special reason you mentioned Rocky ? We selected AlamaLinux for our tests as they support all the archs we use ( x86_64, aarch64 and ppc64le).

Honestly, I wouldn't worry too much about this right now. As @brianhlin says we'll align the OSG 3.x EOL with CS8; however, if LHC is still using EL8 because it's in the middle of Run 3, we'll have OSG 3.{x+1} support one of the clones. The impact CMS would see is having to switch to a different tag of the same container.

Why not support one of the clones right now? Why not provide two images one based on CS8 and one based on Alam or Rocky? We know that CMS is not going to change the OS in the middle of Run3 specially for CMS online/HLT and P5. It would be safe to do it now instead of changing things in the middle of run.

[smuzaffar]

We've already seen a case (XRootD) where compiling against CS8 linked a library against a new shared lib ABI version which hadn't made it to RHEL yet

@bbockelm , do you know which system library has a new ABI in CS8? We have not yet seen this issue with xrootd build with CMSSW ( may be because we most of the externals from cmssw software stack : https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_12_3_X/master/xrootd.spec#L29 )

[matyasselmeci]

Hi folks,
The library in question is libjson-c (in the json-c RPM). It's not an ABI difference, it is a difference in the symbols that are provided by the RPM. On Rocky, json-c provides (rpm -q --provides json-c):

json-c = 0.13.1-2.el8
json-c(x86-64) = 0.13.1-2.el8
libjson-c.so.4()(64bit)

on CS8, json-c provides

json-c = 0.13.1-3.el8
json-c(x86-64) = 0.13.1-3.el8
libjson-c.so.4()(64bit)
libjson-c.so.4(JSONC_0.14)(64bit)
libjson-c.so.4(JSONC_0.15)(64bit)

when xrootd is compiled against libjson-c on CS8, the RPM automatic dependency generator adds libjson-c.so.4(JSONC_0.14)(64bit) to the dependency list of the xrootd-server-libs RPM. When xrootd is compiled on Rocky, it only has libjson-c.so.4()(64bit).

[smuzaffar]

In fact, we are looking at the idea of "build on Rocky, support on CentOS Stream" -- this way we build against the older ABI but support folks on a wider range of platforms.

@bbockelm , any decision about this yet or osg-wn going to stay with CS8?

Honestly, I wouldn't worry too much about this right now. As @brianhlin says we'll align the OSG 3.x EOL with CS8; however, if LHC is still using EL8 because it's in the middle of Run 3, we'll have OSG 3.{x+1} support one of the clones. The impact CMS would see is having to switch to a different tag of the same container.

The impact on CMS will be that we have to validate the clone based container in the middle of the run too. We want to do this validation before the Run so that we can use one OS for the whole run. Can you already provide a container based on almalinux?

[smuzaffar]

ping #10 (comment)

[brianhlin]

Hi @smuzaffar the plan is to go with Alma 8 as the base and we're tracking that work here: https://opensciencegrid.atlassian.net/browse/SOFTWARE-5307. When do you need complete your validation work by?

[smuzaffar]

@brianhlin , sooner the better.
Note that we are already done with alma8 + osg-client validation as we build our own containers based on Alma8 and install OSG client SW in it. osg-wn containers based on Alma8 will allow us to directly use these without maintaining our own intermediate layer

[brianhlin]

@smuzaffar we've been working on various OS's for containers and haven't done a great job of communicating. Would you happen to be interested in AlmaLinux 9 containers?

[smuzaffar]

thanks @brianhlin for getting back here. As we needed non-x86-64 (aarch64/ppc64le) containers too, so for el8/el9 we already have started to build cms containers based on almalinux8/9 instead of opensciencegrid/osg-wn containers.