Grant refresh-gitaly-bundle create on pods/exec - #1436
Merged
Conversation
jjnesbitt
previously approved these changes
Aug 13, 2026
The refresh-gitaly-bundle CronJob has failed every run since 2026-07-30 20:21 UTC, when the prod EKS control plane was upgraded from Kubernetes 1.34 to 1.35. Every pod dies on its first `kubectl exec`: error: unable to upgrade connection: pods "gitlab-toolbox-..." is forbidden: User "system:serviceaccount:gitlab:refresh-gitaly-bundle" cannot create resource "pods/exec" in API group "" in the namespace "gitlab" The Role only ever granted `get`/`list` on pods/exec, never `create`. That was always wrong, but it worked because `kubectl exec` prefers the WebSocket transport, whose handshake is an HTTP GET (RFC 6455) -- and the API server derived the RBAC verb from the HTTP method, so WebSocket exec was authorized as `get`. Only the SPDY transport (an HTTP POST) was checked as `create`. Kubernetes 1.35 closes that privilege-escalation gap (kubernetes/kubernetes#78741) with the AuthorizePodWebsocketUpgradeCreatePermission feature gate, Beta and enabled by default, which requires `create` for any connection upgrade regardless of transport. The loophole this Role depended on is gone, so the latent misconfiguration now surfaces. Split the rule rather than adding `create` to the existing one, so the service account does not also gain create on pods and deployments. It also no longer holds the get/list on pods/exec it never needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mvandenburgh
force-pushed
the
fix/gitaly-bundle-exec-rbac
branch
from
August 13, 2026 20:01
434bac4 to
8bf98b6
Compare
mvandenburgh
marked this pull request as ready for review
August 13, 2026 20:01
jjnesbitt
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's broken
The
refresh-gitaly-bundleCronJob in thegitlabnamespace has failed every run since 2026-07-30. Each pod dies immediately on its firstkubectl exec:It runs every 15 minutes, so it's been burning 7 pods per attempt (default
backoffLimit: 6) for two weeks. The last successful run wasrefresh-gitaly-bundle-29757375, which finished at 2026-07-30 20:19:10 UTC.Impact: the Gitaly bundle URI for
spack/spackhasn't been regenerated in two weeks. Clones still work — clients fetch the stale bundle and then catch up on the delta — but that delta grows every day, so CI clone performance degrades steadily until this is fixed.Root cause
The Role only ever granted
get/listonpods/exec, nevercreate. That was always wrong — but it worked for 478 days by accident.kubectl execprefers the WebSocket transport, and a WebSocket handshake is by definition an HTTPGET(RFC 6455 §4.1). The API server derived the RBAC verb from the HTTP method, so:GETgetPOSTcreateSo WebSocket exec was authorized as
get, andverbs: ["get", "list"]was enough.That was a long-standing privilege-escalation hole (kubernetes/kubernetes#78741) — a role that reads as strictly read-only actually granted shell access into any pod in the namespace. Kubernetes 1.35 closed it with the
AuthorizePodWebsocketUpgradeCreatePermissionfeature gate (Beta, enabled by default), which requirescreatefor any connection upgrade regardless of transport.Why it broke exactly when it did
UpdateClusterVersiononspack-blue(1.34 → 1.35, #1416)2 minutes 33 seconds apart. The job succeeded on its last pre-upgrade run and has failed every run since. The cluster is now on
1.35/eks.20.Note the terraform apply ran ~2h before the merge commit for #1416 landed, so the merge timestamp is misleading here — CloudTrail is the accurate signal.
Verification
gitlab-setting-updateris the control case: it runs the samekubectl execintogitlab-toolbox, on the same node, and succeeds — because its Role already includescreate. This change bringsrefresh-gitaly-bundlein line with it.