Skip to content

Install kubectl from the release binaries so it is version-stamped - #777

Closed
rparsonsbb wants to merge 2 commits into
pulumi:mainfrom
rparsonsbb:fix/kubectl-version-stamped
Closed

Install kubectl from the release binaries so it is version-stamped#777
rparsonsbb wants to merge 2 commits into
pulumi:mainfrom
rparsonsbb:fix/kubectl-version-stamped

Conversation

@rparsonsbb

Copy link
Copy Markdown

Proposed changes

/usr/bin/kubectl in the pulumi/pulumi kitchen sink image is not version-stamped:

$ kubectl version --client
Client Version: v0.0.0-master+$Format:%H$
Kustomize Version: v5.7.1

major/minor are empty and gitVersion/gitCommit still contain the literal $Format:%H$
git-archive placeholders, i.e. the binary was built without the version -ldflags.

That string is not parseable as a version, so it breaks tooling that checks the kubectl client
version. @pulumi/eks calls semver.clean() on it, gets null, and throws
Invalid version. Must be a string. Got type "object". — a message that never mentions kubectl
(typeof null === "object"), which makes it expensive to trace back here. For us this presented as
a fleet-wide Pulumi Deployments outage on the default runner image: every stack, every tier, no
config change, while the same program succeeded locally on identical CLI/SDK/Node versions.

Cause. kubectl came from apt-get install kubectl, but the same layer also adds the Google
Cloud SDK apt repo two lines earlier, which publishes its own kubectl package. Which package
provides /usr/bin/kubectl is therefore an apt resolution outcome between two repos rather than an
explicit choice.

Fix. Install the official release binary and verify it against the published checksum, matching
how aws-iam-authenticator is already installed in this same layer (and consistent with #681,
which moved that tool to release binaries for similar reasons). The Kubernetes apt repo and keyring
setup become unused and are dropped. Version selection is unchanged in spirit — still whatever
https://dl.k8s.io/release/stable.txt reports, just the full patch version rather than truncating
to major.minor for the apt repo path.

I ran the exact install pipeline from this diff on linux/amd64:

KUBE_LATEST=v1.36.3
kubectl: OK                     # checksum verified
gitVersion: v1.36.3             # properly stamped

Test. Adds a Kubectl subtest to TestCLIToolTests asserting the client version is present,
free of $Format: placeholders, and parseable. This failure mode is silent until something
downstream tries to parse the version, so it is worth pinning. go vet ./... passes in tests/.

I could not build the image locally to run the full suite, so CI should be the judge on the build
itself. Happy to pin an explicit kubectl version (with a # renovate: comment, like
AWS_IAM_AUTHENTICATOR_VERSION) instead of tracking stable.txt if you'd prefer that — say the
word and I'll push it.

Related issues (optional)

Fixes #776.

The downstream @pulumi/eks crash this triggers is pulumi/pulumi-eks#2360, with a fix in
pulumi/pulumi-eks#2361 so the error at least names the tool it is complaining about.

The kitchen sink image installed kubectl via `apt-get install kubectl`.
The same layer also adds the Google Cloud SDK apt repo, which publishes
its own `kubectl` package, so which package provides /usr/bin/kubectl is
an apt resolution outcome rather than an explicit choice. In the 3.256.0
image the resulting binary is not version-stamped:

    $ kubectl version --client
    Client Version: v0.0.0-master+$Format:%H$

`major` and `minor` are empty and gitVersion/gitCommit still contain the
literal git-archive placeholders. That is not parseable as a version, and
it breaks tooling that checks the kubectl client version - @pulumi/eks
calls semver.clean() on it, gets null, and dies with a TypeError that
never mentions kubectl.

Install the official release binary and verify it against the published
checksum instead, matching how aws-iam-authenticator is already installed
in this layer, and drop the now-unused Kubernetes apt repo.

Adds a test asserting kubectl reports a parseable, non-placeholder
version, since this fails silently until something tries to parse it.
@rparsonsbb
rparsonsbb requested a review from a team as a code owner August 12, 2026 16:58
@julienp
julienp force-pushed the fix/kubectl-version-stamped branch from 89edfd9 to e1151d8 Compare August 14, 2026 10:01
@julienp

julienp commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Thank you! I moved the kubectl installation up to be next to the AWS CLI installation.

@julienp

julienp commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

I re-created the PR here #781 so w can run the tests. Closing this one.

@julienp julienp closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kubectl in the pulumi/pulumi image is an unstamped build (v0.0.0-master+$Format:%H$), breaking @pulumi/eks

2 participants