Update rpm package to have 256bit digests #1411

elezar · 2025-11-03T17:07:23Z

This change uses fpm to update the built rpm packages to have
256bit digests without changing their contents.

This is required in cases where SHA1 or MD5 digests are not supported such as FIPS or newer RPM-based distributions. (See #1307).

For the v1.18.0 packages:

$ docker run --rm -ti -v $(pwd)/release-v1.18.0-stable:/work -w /work rockylinux:9 bash -c "find . -name '*.rpm' -exec rpm --checksig --verbose {} \;"
./packages/centos7/aarch64/libnvidia-container-static-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/libnvidia-container-tools-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/libnvidia-container1-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/libnvidia-container-libseccomp2-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/nvidia-container-toolkit-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/libnvidia-container1-debuginfo-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/nvidia-container-toolkit-base-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/nvidia-container-toolkit-operator-extensions-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/aarch64/libnvidia-container-devel-1.18.0-1.aarch64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container1-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-libseccomp2-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/nvidia-container-toolkit-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container1-debuginfo-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/nvidia-container-toolkit-base-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-tools-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/nvidia-container-toolkit-operator-extensions-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-devel-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-static-1.18.0-1.x86_64.rpm:
    Header SHA1 digest: OK
    MD5 digest: OK

With these changes:

$  docker run --rm -ti -v $(pwd)/release-v1.19.0-dev-stable:/work -w /work rockylinux:9 bash -c "find . -name '*.rpm' -exec rpm --checksig --verbose {} \;"
./packages/centos7/x86_64/libnvidia-container-static-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-libseccomp2-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container1-debuginfo-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/nvidia-container-toolkit-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/nvidia-container-toolkit-operator-extensions-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-devel-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/nvidia-container-toolkit-base-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container-tools-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
./packages/centos7/x86_64/libnvidia-container1-1.19.0~dev-1.x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK

elezar · 2025-11-04T11:22:38Z

/cherry-pick release-1.18

ArangoGutierrez

TIL - FPM (Effing Package Management) supports 256-bit digests for RPM packages

LGTM

.dockerignore

elezar · 2025-11-05T08:02:34Z

TIL - FPM (Effing Package Management) supports 256-bit digests for RPM packages

It is my understanding that fpm invokes rpmbuild in the background and the digests generated depend on the version of rpmbuild used. In this case it works because we're running this in a relatively new ubi9-based base image.

This change uses fpm to update the built rpm packages to have 256bit digests without changing their contents. Signed-off-by: Evan Lezar <[email protected]>

ArangoGutierrez · 2025-11-05T08:08:30Z

TIL - FPM (Effing Package Management) supports 256-bit digests for RPM packages

It is my understanding that fpm invokes rpmbuild in the background and the digests generated depend on the version of rpmbuild used. In this case it works because we're running this in a relatively new ubi9-based base image.

Yeah, that's why I put TIL (Today I learned) , I was not aware of that FPM tool, cool

antheas · 2025-11-05T09:56:21Z

It seems that drivers built with centos8-x86_64 works fine, so this is only needed for drivers built with centos7. Can this step be made to only run for centos7?

elezar · 2025-11-05T10:45:07Z

It seems that drivers built with centos8-x86_64 works fine, so this is only needed for drivers built with centos7. Can this step be made to only run for centos7?

Could you clarify what you mean? For our publicly released packages, we currently build on centos7. This change only updates the packages that we generate and doesn't change how their contents are built.

antheas · 2025-11-05T11:42:55Z

It seems that drivers built with centos8-x86_64 works fine, so this is only needed for drivers built with centos7. Can this step be made to only run for centos7?

Could you clarify what you mean? For our publicly released packages, we currently build on centos7. This change only updates the packages that we generate and doesn't change how their contents are built.

Got it, we started building artifacts in parallel for centos8 so that they would work. This is a separate containerfile that is not used for building so its a NOOP.

elezar · 2025-11-05T12:58:06Z

@antheas as a matter of interest, do you have problems using our public artifacts on centos8? The intent is that these should work as expected, since the primary reason for us using centos7 to build the artifacts is to ensure an as broad as possible glibc support. (These same artifacts are used in other RPM-based distributions too and we don't see any problems there).

antheas · 2025-11-05T13:01:12Z

@antheas as a matter of interest, do you have problems using our public artifacts on centos8? The intent is that these should work as expected, since the primary reason for us using centos7 to build the artifacts is to ensure an as broad as possible glibc support. (These same artifacts are used in other RPM-based distributions too and we don't see any problems there).

I am using them on F43. I found using centos 8 to produce them creates the correct signature and I did not want to modify the code further.

We started pre-building the nvidia drivers to match them with the kernel, so I also added the nvidia container toolkit to that.

elezar · 2025-11-05T13:04:22Z

Thanks for the clarification. With these changes here, you should be able to consume the packages from upstream. As a note, the SHA-tagged packaging image (e.g. ghcr.io/nvidia/container-toolkit:85cfb7f4-packaging for the latest commit here) can be used to extract the packages for a given SHA. We use this artifact to propagate the packages (that are built only once) through our release pipeline.

antheas · 2025-11-05T13:08:35Z

Thanks for the clarification. With these changes here, you should be able to consume the packages from upstream. As a note, the SHA-tagged packaging image (e.g. ghcr.io/nvidia/container-toolkit:85cfb7f4-packaging for the latest commit here) can be used to extract the packages for a given SHA. We use this artifact to propagate the packages (that are built only once) through our release pipeline.

I can look into doing that in the future. Would you be interested in adding a github attestation to your action? One of the reasons we did this was to get an extra artifact having SLSA L3

jgehrcke · 2025-11-05T14:19:53Z

deployments/container/Dockerfile

+        mkdir -p /artifacts/packages/centos7/${arch}; \
+        cd /artifacts/packages/centos7/${arch}; \
+        # For each package in the source folder we recreate a package to update
+        # the digests to 256-bit since we're running in a ubi9 container.


Hello! I got confused by the logic communicated by this sentence: we update the digests because we run in an ubi9 container?

Just glancing over this whole topic... is the following true?

We update the centos 7 RPM packages to contain 256 bit digests because of <some-reason-probably-compliance> and we do so by processing these RPM files in a UBI9 (rhel generation 9) container because only there we have the required tooling available.

This is for my own understanding, I am not asking to change the patch :).

When we originally build the packages, we do so in a centos7 container (for glibc reasons). The version of rpmbuild available in this container is:

# rpmbuild --version RPM version 4.11.3

which only generates SHA1 and MD5 digests and does not support 256-bit digests.

According to https://stackoverflow.com/questions/71942026/add-sha256-digests-to-rpm-packages, support for SHA256 digests was added in 4.14.3 (although it may not have been the default).

What we do in this case is us an intermediate container (rpmdigests) based on nvcr.io/nvidia/cuda:13.0.1-base-ubi9 (currently) with:

# rpmbuild --version RPM version 4.16.1.3

Use fpm to "recreate" the packages. What I assume happens is that fpm extracts the packages and creates new packages using rpmbuild. Since the version in the container supports SHA256 digests by default these new packages will include these. Note that the exising SHA1 and MD5 digests are maintained.

And yes, the reason that we do this is "compliance". There are programs like FIPS that require this and certain newer distributions (see #1307) also require this by default.

I have updated the description with the motivation.

elezar force-pushed the rpm-digests branch 2 times, most recently from a60b5f4 to 3de3949 Compare November 4, 2025 11:08

elezar changed the title ~~TOFIX~~ Update rpm package to have 256bit digests Nov 4, 2025

elezar marked this pull request as ready for review November 4, 2025 11:22

elezar requested review from ArangoGutierrez, cdesiniotis and tariq1890 November 4, 2025 11:22

elezar added this to the v1.18.1 milestone Nov 4, 2025

github-actions bot added the cherry-pick/release-1.18 label Nov 4, 2025

ArangoGutierrez approved these changes Nov 4, 2025

View reviewed changes

.dockerignore Outdated Show resolved Hide resolved

ArangoGutierrez mentioned this pull request Nov 4, 2025

Produce RPM packages with SHA256 header digests #1351

Draft

elezar force-pushed the rpm-digests branch from 3de3949 to 184a2df Compare November 5, 2025 08:03

Update rpm package to have 256bit digests

85cfb7f

This change uses fpm to update the built rpm packages to have 256bit digests without changing their contents. Signed-off-by: Evan Lezar <[email protected]>

elezar force-pushed the rpm-digests branch from 184a2df to 85cfb7f Compare November 5, 2025 08:06

jgehrcke reviewed Nov 5, 2025

View reviewed changes

Update rpm package to have 256bit digests #1411

Are you sure you want to change the base?

Update rpm package to have 256bit digests #1411

Conversation

elezar commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elezar commented Nov 4, 2025

Uh oh!

ArangoGutierrez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elezar commented Nov 5, 2025

Uh oh!

ArangoGutierrez commented Nov 5, 2025

Uh oh!

antheas commented Nov 5, 2025

Uh oh!

elezar commented Nov 5, 2025

Uh oh!

antheas commented Nov 5, 2025

Uh oh!

elezar commented Nov 5, 2025

Uh oh!

antheas commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elezar commented Nov 5, 2025

Uh oh!

antheas commented Nov 5, 2025

Uh oh!

jgehrcke Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elezar Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elezar commented Nov 3, 2025 •

edited

Loading

antheas commented Nov 5, 2025 •

edited

Loading

elezar Nov 5, 2025 •

edited

Loading