Nvidia container RPMs >= 1.18.1 breaking system installation

[cvolz]

Since 1.18.1, if we include the RPMs in an ISO for system installation (Anaconda kickstart installer, in our case with Rocky Linux 8), then the ISO installation fails.

The root cause is a new %pretrans scriptlet that requires /bin/sh. I have noticed that the RPMs now contain several empty scriptlets that were not there in previous releases:

rpm -qp --scripts nvidia-container-toolkit-1.18.1-1.x86_64.rpm 

pretrans program: /bin/sh 
preinstall program: /bin/sh 
[...] 
preuninstall program: /bin/sh 
[...] 
verify program: /bin/sh 

In particular, the pretrans scriptlet with a requirement for /bin/sh is an issue. The Fedora Packaging Guidelines give a good explanation why:

Note that the %pretrans scriptlet will, in the particular case of system installation, run before anything at all has been installed. This implies that it cannot have any dependencies at all. For this reason, %pretrans is best avoided, but if used it MUST (by necessity) be written in Lua.

After digging further, I am sure that this comes from re-packaging the RPMs with FPM, which has been introduced with the recent changes for 256 bit digests (e.g. 85cfb7f). When processing previous RPMs through FPM manually, I can see similar effects.

Other fields in the rpms are apparently altered by FPM, too, such as the changelog (is replaced with “Package created with FPM”) and the Obsoletes (is deleted).

I am wondering, I assume there is a reason why you don’t build the rpms is a newer build environment that would add the 256 bit payload digests directly?

[elezar]

I am wondering, I assume there is a reason why you don’t build the rpms is a newer build environment that would add the 256 bit payload digests directly?

The short answer is that we build the binaries on older systems to allow the broadest GLIBC compatibility. We are working on alternatives, but saw the "repackaging" with FPM as a low-effort solution to the missing digests.

[cvolz]

Hi elezar, if you would like to stay with the re-packaging approach, the rpmrebuild tool (package rpmrebuild from EPEL repo) may be an alternative. But of course I don't know if that would fit into your build environment.

[elezar]

Thanks @cvolz. I was not aware of that tool. Let me have a look.

Update: Installing the tooling in our build environment looks feasible, but it also seems as if some of our packages are not directly compatible with the tooling. I will continue the investigation though.

[elezar]

@cvolz I have created #1612 to switch to rpmrebuild. The PR is still marked as draft since I still need to run some tests. If you're able to test the temporary packages from the ghcr.io/nvidia/container-toolkit:eb844398-packaging image, that would be appreciated.

[cvolz]

Hi @elezar, I am not sure what to do with this Docker image. The easiest for me would be if you could point me to the resulting RPMs, then I could take a look at them.

[elezar]

Due to our build process, the packages will only be available once I produce a new RC. Would it be possible for you to extract the packages from the image as described here: #1599 (comment) to give us an early signal. If not, I could provide early packages through an alternate mechanism or wait until we produce the next release candidate.

To pull the packages you can use:

./hack/pull-packages.sh \
    ghcr.io/nvidia/container-toolkit:eb844398-packaging \
    test-package-eb844398

This will create a folder test-package-eb844398 including the rpm and deb packages.

[cvolz]

This description was exactly what I was missing. I was able to extract the RPMs from the image.

Inspection of the RPMs looks good to me. The %pretrans and other unused scriplets are gone again, which should solve the issue. The Changelog and Obsoletes entries are also back to the original values as in <= 1.18.0. I have also seen that the packages contain the SHA256 checksums as intended.

The packages in the image were not signed but I guess that is expected in the temporary build.

I have noticed that the packages now contain /usr/lib/.build-id links. I think this is not related to the rpmrebuild and may be an intended change.

I will try to use the packages in a full installation in our test environment (orignal use case) but this may take a few days.