kubectl in the pulumi/pulumi image is an unstamped build (v0.0.0-master+$Format:%H$), breaking @pulumi/eks

[rparsonsbb]

What happened

/usr/bin/kubectl in public.ecr.aws/pulumi/pulumi:3.256.0 reports no real version:

$ kubectl version --client
Client Version: v0.0.0-master+$Format:%H$
Kustomize Version: v5.7.1

$ kubectl version --client=true --output=json
{
  "clientVersion": {
    "major": "",
    "minor": "",
    "gitVersion": "v0.0.0-master+$Format:%H$",
    "gitCommit": "$Format:%H$",
    "gitTreeState": "",
    "buildDate": "2026-07-22T17:11:40Z",
    "goVersion": "go1.26.5",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.7.1"
}

major/minor are empty and gitVersion/gitCommit still contain the literal $Format:%H$
git-archive placeholders, i.e. this binary was built without the -ldflags that inject version
information.

Impact

This breaks @pulumi/eks for every user on this image. assertCompatibleKubectlVersionExists does
semver.clean(gitVersion) (which returns null for that string) and passes the result to
semver.lt(), which throws:

error: eks:index:Cluster resource 'my-cluster' has a problem: Invalid version. Must be a string. Got type "object".
error: Error: failed to register new resource my-cluster [...]: 2 UNKNOWN: resource monitor shut down

Because typeof null === "object", the message points nowhere near kubectl, which makes this
expensive to diagnose — it reads as a Pulumi provider/plugin version bug. (Filed separately against
pulumi/pulumi-eks, which should also guard the null; but the image should still ship a
version-stamped kubectl.)

This surfaced for us as a fleet-wide Pulumi Deployments outage: every stack, every tier, no
config change, while the identical program on the same CLI/SDK/Node versions succeeded locally. The
only difference was the kubectl binary in the runner.

Verified directly — a preview that fails on this image succeeds unchanged when a stamped kubectl is
installed first via a pre-run command:

KV=$(curl -sSL https://dl.k8s.io/release/stable.txt)
curl -sSLo /usr/bin/kubectl "https://dl.k8s.io/release/$KV/bin/linux/amd64/kubectl"
chmod +x /usr/bin/kubectl

Affected image

• public.ecr.aws/pulumi/pulumi@sha256:28db863e7128f8c09b6c8b4f97b356be58483367ab4d594c52f29bf844f88dc7
  (amd64 child manifest sha256:ef4e87e3ec0d8…), Pulumi 3.256.0
• created: 2026-08-04T19:55:27Z
• org.opencontainers.image.revision: 1cefc13481fca9dd3eeb654fa45b80cde6f6666f
• This is the current default runner image for Pulumi Deployments, so it affects Deployments
  users who never chose the image.

Likely cause

From the image history, docker/pulumi/Dockerfile installs kubectl via apt:

KUBE_LATEST=$(curl -L -s https://dl.k8s.io/release/stable.txt | awk 'BEGIN { FS="." } { printf "%s.%s", $1, $2 }') && \
curl -fsSL https://pkgs.k8s.io/core:/stable:/${KUBE_LATEST}/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg && \
echo "deb [signed-by=...] https://pkgs.k8s.io/core:/stable:/${KUBE_LATEST}/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list && \
apt-get update -y && \
apt-get install -y azure-cli docker-ce google-cloud-cli google-cloud-cli-gke-gcloud-auth-plugin kubectl

Two lines earlier the same layer adds the Google Cloud SDK repo:

echo "deb [signed-by=...] http://packages.cloud.google.com/apt cloud-sdk-$(lsb_release -cs) main" | tee /etc/apt/sources.list.d/google-cloud-sdk.list

That repo also publishes a kubectl package, so which one apt selects is a resolution outcome
between two repos rather than an explicit choice — a plausible route for an unstamped build to land
in the image. (I have not confirmed which package provides the installed binary; dpkg -S /usr/bin/kubectl inside the image would settle it.)

Suggested fix

Install the official release binary instead of relying on apt resolution, which also makes the
installed version explicit and pinnable:

KV=$(curl -fsSL https://dl.k8s.io/release/stable.txt) && \
curl -fsSLo /usr/bin/kubectl "https://dl.k8s.io/release/${KV}/bin/linux/${TARGETARCH}/kubectl" && \
curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/${TARGETARCH}/kubectl.sha256" | \
  awk -v f=/usr/bin/kubectl '{print $1"  "f}' | sha256sum --check - && \
chmod 755 /usr/bin/kubectl

This matches how the same Dockerfile already installs aws-iam-authenticator (release binary +
checksum verification), and drops kubectl from the apt-get install list.

Whatever the fix, a smoke test asserting kubectl version --client parses as valid semver would
stop this regressing — the failure is silent until a downstream tool tries to parse the version.