Description
Starting around 3:30am PST we noticed our automated pipelines are failing to build the docker image.
Errors below:
#6 1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
#6 1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
#6 1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]
#6 2.267 Reading package lists...
#6 2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
#6 2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.
#6 ERROR: process "/bin/sh -c sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint" did not complete successfully: exit code: 100
[2/2] RUN sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint:
1.376 Get:10 https://deb.nodesource.com/node_16.x buster/main amd64 Packages [776 B]
1.410 Get:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease [1192 B]
1.533 Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [307 kB]
1.643 Get:12 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]
2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.
debian:bullseye is the base
Activity
petedyer11 commentedon Nov 4, 2024
I see the same issue with v1.27
Creteil commentedon Nov 4, 2024
« apt dist-upgrade » under ubuntu 22.04 output same error related...
xmudrii commentedon Nov 4, 2024
The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key from the latest Kubernetes repository (v1.31):
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
We'll try to figure out if we can handle this in a better way in the future.
Update 2024-11-12:
On the RPM-based systems, you need to modify the Kubernetes repository definition to point to the latest GPG key. Open
/etc/yum.repos.d/kubernetes.repo
in your preferred text editor, then make sure that thegpgkey
URL is set tohttps://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
(which is the latest Kubernetes repository at the moment).As described in #3818 (comment), we at the moment recommend using the GPG key from the latest Kubernetes repository, as older Kubernetes repositories might not have the latest key. This GPG key from the latest Kubernetes repository should work with all other repositories under
pkgs.k8s.io
. We're debugging that and we'll follow up as we understand what's the best way forward.petedyer11 commentedon Nov 4, 2024
In my use case I have a loop which gets the Release.key for 1.27, 1.28, 1.29 and 1.30, followed by an
apt update
which fails for 1.27. I chose later the version I actually want to install.So have all the keys expired and been replaced?
Update kubernetes repository GPG key
Update kubernetes repository GPG key (#2451)
drzewiec commentedon Nov 5, 2024
There seems to be some problem beyond that, because one of the steps in my docker build is to download that key fresh. So basically I'm doing this:
That fails (with the errors mentioned in this issue) even though I am not caching the key.
petedyer11 commentedon Nov 6, 2024
It appears that you can use the v1.31 Release.key to pull the v1.24 version, but I am doubtful this is correct.
xmudrii commentedon Nov 6, 2024
@drzewiec I was able to reproduce the issue, I'll check what's going on and report back ASAP.
28 remaining items
BenTheElder commentedon Dec 17, 2024
yes, I don't think we want to, my understanding is that the current layout comes from OBS (also to be clear I don't work on this, but I do lead SIG K8s Infra which provides the CDN infra and manages sustainable cloud credit usage)
I think we understand what other deb/rpm repos look like, before OBS we used an instance of Google's package host which looked like what you're describing -- single key.
The problem is either to manage this within the constraints of OBS (hence the ask for prior art from other OBS users) or to outline replacing OBS (a potentially much bigger but doable task, we use our own build & release system for the other bits).
personally I didn't think we should be hosting/maintauning distro style packages at all, we have binaries and container images and distros could package kubectl / kubelet on their own and we already spend substantial resources on the other more popular binary/container image hosts.
BenTheElder commentedon Dec 17, 2024
unless I missed something, we still don't know this (I don't see evidence that node or Microsoft are using openBuildService)
BenTheElder commentedon Dec 17, 2024
... ideally the project would fully control the key and we could just publish it directly ...
ulidtko commentedon Dec 17, 2024
I think @BenTheElder OBS as the CI / build plaform is not material to the overall practice of package-signing-key management... It's more of an operational (Ops) matter
ulidtko commentedon Dec 17, 2024
Again: I'm not an active user of OBS; but it "seems easy"™ and perfectly doable to apply the generic principles to OBS specifics. E.g. finding this guide — https://en.opensuse.org/openSUSE:Build_Service_Signer — just set the
$keyfile
of all repos to 1 shared key, and bob's your uncle, right?..BenTheElder commentedon Dec 17, 2024
I don't think that's true, the package signing happens as part of the build service.
Those docs appear to me to be related to operating your own instance of the OBS software (specifically the signing service), releng / SIG Release is using the hosted public instance at https://build.opensuse.org/
@xmudrii would know more.
xmudrii commentedon Jan 6, 2025
@BenTheElder is correct about this. ^^
That's correct as well, and this option is already set for our projects, hence you can use any key as described here: #3818 (comment)
Now that the holidays break is over, I'll try to pick this up with the OBS folks again.
xmudrii commentedon Jan 6, 2025
Reducing the priority on this issue because this is affecting only EOL releases at the moment.
/remove-priority critical-urgent
/priority important-longterm
cunningr commentedon Jan 6, 2025
Although those releases are no longer under active development we (as I suspect many others in the community are) still need to operate and support clusters with those versions. Currently our image building is broken because we can't pull the packages we need to support our fleet.
xmudrii commentedon Jan 6, 2025
We have provided a workaround that you can use for the EOL releases: #3818 (comment)
These package repositories are maintained by the community, and often by people who are working on this in their free time. Those folks already have a lot on their plate, adding EOL releases on top of it is not fair to them. The Kubernetes support/EOL policy is very clear about this -- we only provide support for the actively maintained releases. We do understand that not anyone can update frequently, but we are not able to always accommodate such cases, and we kindly ask for understanding.