Skip to content

APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02 #3818

Open
@kungfooflea

Description

@kungfooflea

Starting around 3:30am PST we noticed our automated pipelines are failing to build the docker image.

Errors below:

#6 1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
#6 1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
#6 1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]
#6 2.267 Reading package lists...
#6 2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
#6 2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.
#6 ERROR: process "/bin/sh -c sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint" did not complete successfully: exit code: 100

[2/2] RUN sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint:
1.376 Get:10 https://deb.nodesource.com/node_16.x buster/main amd64 Packages [776 B]
1.410 Get:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease [1192 B]
1.533 Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [307 kB]
1.643 Get:12 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]

2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org
2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.

debian:bullseye is the base

Activity

added
area/release-engIssues or PRs related to the Release Engineering subproject
kind/bugCategorizes issue or PR as related to a bug.
sig/releaseCategorizes an issue or PR as relevant to SIG Release.
on Nov 4, 2024
petedyer11

petedyer11 commented on Nov 4, 2024

@petedyer11

I see the same issue with v1.27

Creteil

Creteil commented on Nov 4, 2024

@Creteil

« apt dist-upgrade » under ubuntu 22.04 output same error related...

oem@oem-ThinkPad-P15-Gen-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
oem@oem-ThinkPad-P15-Gen-1:~$
Err:16 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease                                                                                                          
  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>
Fetched 3917 B in 8s (491 B/s)                                                                                                                                                                                    
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>
W: Failed to fetch https://pkgs.k8s.io/core:/stable:/v1.31/deb/InRelease  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>
W: Some index files failed to download. They have been ignored, or old ones used instead.
xmudrii

xmudrii commented on Nov 4, 2024

@xmudrii
Member

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key from the latest Kubernetes repository (v1.31):

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

Update 2024-11-12:

On the RPM-based systems, you need to modify the Kubernetes repository definition to point to the latest GPG key. Open /etc/yum.repos.d/kubernetes.repo in your preferred text editor, then make sure that the gpgkey URL is set to https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key (which is the latest Kubernetes repository at the moment).

As described in #3818 (comment), we at the moment recommend using the GPG key from the latest Kubernetes repository, as older Kubernetes repositories might not have the latest key. This GPG key from the latest Kubernetes repository should work with all other repositories under pkgs.k8s.io. We're debugging that and we'll follow up as we understand what's the best way forward.

petedyer11

petedyer11 commented on Nov 4, 2024

@petedyer11

In my use case I have a loop which gets the Release.key for 1.27, 1.28, 1.29 and 1.30, followed by an apt update which fails for 1.27. I chose later the version I actually want to install.
So have all the keys expired and been replaced?

added a commit that references this issue on Nov 5, 2024
added a commit that references this issue on Nov 5, 2024
drzewiec

drzewiec commented on Nov 5, 2024

@drzewiec

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

There seems to be some problem beyond that, because one of the steps in my docker build is to download that key fresh. So basically I'm doing this:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

That fails (with the errors mentioned in this issue) even though I am not caching the key.

petedyer11

petedyer11 commented on Nov 6, 2024

@petedyer11

It appears that you can use the v1.31 Release.key to pull the v1.24 version, but I am doubtful this is correct.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl
xmudrii

xmudrii commented on Nov 6, 2024

@xmudrii
Member

@drzewiec I was able to reproduce the issue, I'll check what's going on and report back ASAP.

28 remaining items

BenTheElder

BenTheElder commented on Dec 17, 2024

@BenTheElder
Member

First and foremost, don't version the package-signing key. It makes zero sense to distribute one key for v1.24 and [potentially] another for v1.31. That's because PSKs have nothing to do with packaged software neither changes therein — they have all to do with release-engineering, authenticating packages' publisher, establishing a secure link in the supply chain. Coupling the key to kubectl version is gratuitous and completely unnecessary. Perhaps I'm missing some additional concerns of the SIG — but IMO doing this just creates complications for no good reason.

yes, I don't think we want to, my understanding is that the current layout comes from OBS (also to be clear I don't work on this, but I do lead SIG K8s Infra which provides the CDN infra and manages sustainable cloud credit usage)

I think we understand what other deb/rpm repos look like, before OBS we used an instance of Google's package host which looked like what you're describing -- single key.

The problem is either to manage this within the constraints of OBS (hence the ask for prior art from other OBS users) or to outline replacing OBS (a potentially much bigger but doable task, we use our own build & release system for the other bits).

personally I didn't think we should be hosting/maintauning distro style packages at all, we have binaries and container images and distros could package kubectl / kubelet on their own and we already spend substantial resources on the other more popular binary/container image hosts.

BenTheElder

BenTheElder commented on Dec 17, 2024

@BenTheElder
Member

to figure out how other projects on OBS are handling this

unless I missed something, we still don't know this (I don't see evidence that node or Microsoft are using openBuildService)

BenTheElder

BenTheElder commented on Dec 17, 2024

@BenTheElder
Member

... ideally the project would fully control the key and we could just publish it directly ...

ulidtko

ulidtko commented on Dec 17, 2024

@ulidtko

I think @BenTheElder OBS as the CI / build plaform is not material to the overall practice of package-signing-key management... It's more of an operational (Ops) matter

ulidtko

ulidtko commented on Dec 17, 2024

@ulidtko

Again: I'm not an active user of OBS; but it "seems easy"™ and perfectly doable to apply the generic principles to OBS specifics. E.g. finding this guide — https://en.opensuse.org/openSUSE:Build_Service_Signer — just set the $keyfile of all repos to 1 shared key, and bob's your uncle, right?..

BenTheElder

BenTheElder commented on Dec 17, 2024

@BenTheElder
Member

OBS as the CI / build plaform is not material to the overall practice of package-signing-key management... It's more of an operational (Ops) matter

I don't think that's true, the package signing happens as part of the build service.

E.g. finding this guide — https://en.opensuse.org/openSUSE:Build_Service_Signer — just set the $keyfile of all repos to 1 shared key, and bob's your uncle, right?..

Those docs appear to me to be related to operating your own instance of the OBS software (specifically the signing service), releng / SIG Release is using the hosted public instance at https://build.opensuse.org/

@xmudrii would know more.

xmudrii

xmudrii commented on Jan 6, 2025

@xmudrii
Member

I don't think that's true, the package signing happens as part of the build service.

@BenTheElder is correct about this. ^^

Those docs appear to me to be related to operating your own instance of the OBS software (specifically the signing service), releng / SIG Release is using the hosted public instance at https://build.opensuse.org/

That's correct as well, and this option is already set for our projects, hence you can use any key as described here: #3818 (comment)

Now that the holidays break is over, I'll try to pick this up with the OBS folks again.

xmudrii

xmudrii commented on Jan 6, 2025

@xmudrii
Member

Reducing the priority on this issue because this is affecting only EOL releases at the moment.
/remove-priority critical-urgent
/priority important-longterm

added
priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.
and removed
priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.
on Jan 6, 2025
cunningr

cunningr commented on Jan 6, 2025

@cunningr

Although those releases are no longer under active development we (as I suspect many others in the community are) still need to operate and support clusters with those versions. Currently our image building is broken because we can't pull the packages we need to support our fleet.

xmudrii

xmudrii commented on Jan 6, 2025

@xmudrii
Member

Currently our image building is broken because we can't pull the packages we need to support our fleet.

We have provided a workaround that you can use for the EOL releases: #3818 (comment)

Although those releases are no longer under active development we (as I suspect many others in the community are) still need to operate and support clusters with those versions.

These package repositories are maintained by the community, and often by people who are working on this in their free time. Those folks already have a lot on their plate, adding EOL releases on top of it is not fair to them. The Kubernetes support/EOL policy is very clear about this -- we only provide support for the actively maintained releases. We do understand that not anyone can update frequently, but we are not able to always accommodate such cases, and we kindly ask for understanding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/release-engIssues or PRs related to the Release Engineering subprojectkind/bugCategorizes issue or PR as related to a bug.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/releaseCategorizes an issue or PR as relevant to SIG Release.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

    Participants

    @ulidtko@meyerkev@BenTheElder@kungfooflea@saiteneffekt

    Issue actions

      APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02 · Issue #3818 · kubernetes/release