Skip to content

Grant refresh-gitaly-bundle create on pods/exec - #1436

Merged
mvandenburgh merged 1 commit into
mainfrom
fix/gitaly-bundle-exec-rbac
Aug 13, 2026
Merged

Grant refresh-gitaly-bundle create on pods/exec#1436
mvandenburgh merged 1 commit into
mainfrom
fix/gitaly-bundle-exec-rbac

Conversation

@mvandenburgh

Copy link
Copy Markdown
Member

What's broken

The refresh-gitaly-bundle CronJob in the gitlab namespace has failed every run since 2026-07-30. Each pod dies immediately on its first kubectl exec:

error: unable to upgrade connection: pods "gitlab-toolbox-57dd9d8f48-q7bqm" is forbidden:
User "system:serviceaccount:gitlab:refresh-gitaly-bundle" cannot create resource "pods/exec"
in API group "" in the namespace "gitlab"

It runs every 15 minutes, so it's been burning 7 pods per attempt (default backoffLimit: 6) for two weeks. The last successful run was refresh-gitaly-bundle-29757375, which finished at 2026-07-30 20:19:10 UTC.

Impact: the Gitaly bundle URI for spack/spack hasn't been regenerated in two weeks. Clones still work — clients fetch the stale bundle and then catch up on the delta — but that delta grows every day, so CI clone performance degrades steadily until this is fixed.

Root cause

The Role only ever granted get/list on pods/exec, never create. That was always wrong — but it worked for 478 days by accident.

kubectl exec prefers the WebSocket transport, and a WebSocket handshake is by definition an HTTP GET (RFC 6455 §4.1). The API server derived the RBAC verb from the HTTP method, so:

Transport HTTP method RBAC verb required (pre-1.35)
WebSocket (tried first) GET get
SPDY (fallback) POST create

So WebSocket exec was authorized as get, and verbs: ["get", "list"] was enough.

That was a long-standing privilege-escalation hole (kubernetes/kubernetes#78741) — a role that reads as strictly read-only actually granted shell access into any pod in the namespace. Kubernetes 1.35 closed it with the AuthorizePodWebsocketUpgradeCreatePermission feature gate (Beta, enabled by default), which requires create for any connection upgrade regardless of transport.

Why it broke exactly when it did

Event Time (UTC)
Last successful run finishes 2026-07-30 20:19:10
CloudTrail UpdateClusterVersion on spack-blue (1.34 → 1.35, #1416) 2026-07-30 20:21:43

2 minutes 33 seconds apart. The job succeeded on its last pre-upgrade run and has failed every run since. The cluster is now on 1.35 / eks.20.

Note the terraform apply ran ~2h before the merge commit for #1416 landed, so the merge timestamp is misleading here — CloudTrail is the accurate signal.

Verification

  • gitlab-setting-updater is the control case: it runs the same kubectl exec into gitlab-toolbox, on the same node, and succeeds — because its Role already includes create. This change brings refresh-gitaly-bundle in line with it.

jjnesbitt
jjnesbitt previously approved these changes Aug 13, 2026
The refresh-gitaly-bundle CronJob has failed every run since 2026-07-30
20:21 UTC, when the prod EKS control plane was upgraded from Kubernetes
1.34 to 1.35. Every pod dies on its first `kubectl exec`:

  error: unable to upgrade connection: pods "gitlab-toolbox-..." is
  forbidden: User "system:serviceaccount:gitlab:refresh-gitaly-bundle"
  cannot create resource "pods/exec" in API group "" in the namespace
  "gitlab"

The Role only ever granted `get`/`list` on pods/exec, never `create`.
That was always wrong, but it worked because `kubectl exec` prefers the
WebSocket transport, whose handshake is an HTTP GET (RFC 6455) -- and
the API server derived the RBAC verb from the HTTP method, so WebSocket
exec was authorized as `get`. Only the SPDY transport (an HTTP POST)
was checked as `create`.

Kubernetes 1.35 closes that privilege-escalation gap
(kubernetes/kubernetes#78741) with the AuthorizePodWebsocketUpgradeCreatePermission
feature gate, Beta and enabled by default, which requires `create` for
any connection upgrade regardless of transport. The loophole this Role
depended on is gone, so the latent misconfiguration now surfaces.

Split the rule rather than adding `create` to the existing one, so the
service account does not also gain create on pods and deployments. It
also no longer holds the get/list on pods/exec it never needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mvandenburgh
mvandenburgh force-pushed the fix/gitaly-bundle-exec-rbac branch from 434bac4 to 8bf98b6 Compare August 13, 2026 20:01
@mvandenburgh
mvandenburgh marked this pull request as ready for review August 13, 2026 20:01
@mvandenburgh
mvandenburgh requested a review from jjnesbitt August 13, 2026 20:01
@mvandenburgh
mvandenburgh merged commit 4a42224 into main Aug 13, 2026
1 check passed
@mvandenburgh
mvandenburgh deleted the fix/gitaly-bundle-exec-rbac branch August 13, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants