APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02

[kungfooflea]

Starting around 3:30am PST we noticed our automated pipelines are failing to build the docker image.

Errors below:

#6 1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
#6 1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
#6 1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]
#6 2.267 Reading package lists...
#6 2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
#6 2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.
#6 ERROR: process "/bin/sh -c sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint" did not complete successfully: exit code: 100

[2/2] RUN sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint:
1.376 Get:10 https://deb.nodesource.com/node_16.x buster/main amd64 Packages [776 B]
1.410 Get:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease [1192 B]
1.533 Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [307 kB]
1.643 Get:12 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]

2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.

debian:bullseye is the base

[petedyer11]

I see the same issue with v1.27

[Creteil]

« apt dist-upgrade » under ubuntu 22.04 output same error related...

oem@oem-ThinkPad-P15-Gen-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
oem@oem-ThinkPad-P15-Gen-1:~$

Err:16 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease                                                                                                          
  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>
Fetched 3917 B in 8s (491 B/s)                                                                                                                                                                                    
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>
W: Failed to fetch https://pkgs.k8s.io/core:/stable:/v1.31/deb/InRelease  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>
W: Some index files failed to download. They have been ignored, or old ones used instead.

[xmudrii]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key from the latest Kubernetes repository (v1.31):

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

Update 2024-11-12:

On the RPM-based systems, you need to modify the Kubernetes repository definition to point to the latest GPG key. Open /etc/yum.repos.d/kubernetes.repo in your preferred text editor, then make sure that the gpgkey URL is set to https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key (which is the latest Kubernetes repository at the moment).

As described in #3818 (comment), we at the moment recommend using the GPG key from the latest Kubernetes repository, as older Kubernetes repositories might not have the latest key. This GPG key from the latest Kubernetes repository should work with all other repositories under pkgs.k8s.io. We're debugging that and we'll follow up as we understand what's the best way forward.

[petedyer11]

In my use case I have a loop which gets the Release.key for 1.27, 1.28, 1.29 and 1.30, followed by an apt update which fails for 1.27. I chose later the version I actually want to install.
So have all the keys expired and been replaced?

[drzewiec]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

There seems to be some problem beyond that, because one of the steps in my docker build is to download that key fresh. So basically I'm doing this:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

That fails (with the errors mentioned in this issue) even though I am not caching the key.

[petedyer11]

It appears that you can use the v1.31 Release.key to pull the v1.24 version, but I am doubtful this is correct.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

[xmudrii]

@drzewiec I was able to reproduce the issue, I'll check what's going on and report back ASAP.