Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .ci/build-push-bundle.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,15 @@ source ~/.bashrc

export BUILD_ONLY=${BUILD_ONLY:-1}
export CONTAINERD_VERSION=${CONTAINERD_VERSION:-1.7.26}
export KUBERNETES_VERSION=${KUBERNETES_VERSION:-1.32.2-1.1}
export RUNC_VERSION=${RUNC_VERSION:-1.2.5}
export KUBERNETES_VERSION=${KUBERNETES_VERSION:-1.32.2}
export KUBERNETES_MAJOR_VERSION=${KUBERNETES_MAJOR_VERSION:-v1.32}
export BUNDLE_VERSION=${BUNDLE_VERSION:-v1.32.2}
export ARCH=${ARCH:-amd64}
export CRITOOL_VERSION=${CRITOOL_VERSION:-1.32.0-1.1}
export CRITOOL_VERSION=${CRITOOL_VERSION:-1.32.0}
export UBUNTU_VERSION=${UBUNTU_VERSION:-"22.04"} # Default to 22.04, can be overridden
export OS=${OS:-linux}
export CNI_VERSION=${CNI_VERSION:-1.3.0}

#alias shasum="sha512sum"
echo "installing imgpkg"
Expand All @@ -25,7 +28,7 @@ docker build -t byoh-bundle .
docker rm -f byoh-bundle-container

echo "executing docker image"
docker run -e CRITOOL_VERSION -e BUILD_ONLY -e CONTAINERD_VERSION -e KUBERNETES_VERSION -e KUBERNETES_MAJOR_VERSION -e ARCH -e UBUNTU_VERSION --name byoh-bundle-container -i byoh-bundle /bin/bash
docker run -e CRITOOL_VERSION -e BUILD_ONLY -e CONTAINERD_VERSION -e RUNC_VERSION -e KUBERNETES_VERSION -e KUBERNETES_MAJOR_VERSION -e ARCH -e UBUNTU_VERSION -e OS -e CNI_VERSION --name byoh-bundle-container -i byoh-bundle /bin/bash

echo "creating bundle dir to push k8s packages"
mkdir -p ./bundle
Expand Down
118 changes: 118 additions & 0 deletions docs/proposals/k8s-bundle-upstream-sourcing-adr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# ADR: Source k8s bundle ingredients from upstream releases instead of apt

**Status:** Blocked — do not merge; see §5 for why
**Date:** 2026-08-18
**Deciders:** TBD
**Related:** `installer/bundle_builder/` (bundle builder), `installer/internal/algo/ubuntu-templates/` (install/uninstall scripts)

---

## 1. Context

The BYOH k8s installer bundle packages kubeadm, kubelet, kubectl, containerd, crictl, and CNI
plugins into an OCI image (`installer/bundle_builder/`), pushed to
`quay.io/platform9/byoh-bundle-<os>_k8s:<k8s-version>` and pulled onto BYO hosts by
`installer/internal/algo/ubuntu-templates/install.sh.tmpl` via `imgpkg`.

Until now, `installer/bundle_builder/ingredients/deb/download.sh` sourced kubeadm/kubelet/kubectl/
cri-tools/kubernetes-cni as `.deb` packages from the Kubernetes apt repository
(`pkgs.k8s.io/core:/stable:/<major.minor>/deb/`), installed on the host via `dpkg --install`. This
has real drawbacks for BYOH's use case specifically: BYO hosts are already-provisioned Linux boxes
the operator doesn't fully control, so depending on `apt-get update` succeeding against an
external apt repo (GPG key handling, network egress, potential lock contention with other host
config management) is a heavier and more fragile runtime dependency than fetching a handful of
pinned binaries directly. It also means Ubuntu-version-specific `.deb` availability/naming quirks
leak into the bundle builder (already visible in `.ci/build-push-bundle.sh`'s per-Ubuntu-version
bundle-name switch).

kubeadm's own documented "without a package manager" install path is exactly this: download
`kubeadm`/`kubelet`/`kubectl` as raw binaries from `https://dl.k8s.io/release/<version>/bin/<os>/
<arch>/<binary>`, verify against the sibling `<binary>.sha256`, `chmod +x`, and place them on
`$PATH` — no apt/dpkg involved.

## 2. Decision

Fetch ingredients directly from their upstream release artifacts instead of apt, and install them
as plain files instead of `.deb` packages:

- **kubeadm, kubelet, kubectl** — raw binaries from `dl.k8s.io/release/<version>/bin/<os>/<arch>/`,
each verified against its `.sha256` before use (matching Kubernetes' own documented procedure —
the previous apt-based script did no integrity verification at all).
- **crictl** — the release tarball from `kubernetes-sigs/cri-tools`, still the actively maintained
upstream for it.
- **CNI plugins** — the release tarball from `containernetworking/plugins`.
- **containerd** — the plain `containerd-<version>-<os>-<arch>.tar.gz` release tarball, **not** the
`cri-containerd-cni-*` bundle the script used previously. That bundle ships its own
`cri-containerd.DEPRECATED.txt`: it has been deprecated since containerd 1.6, "does not work on
some Linux distributions," and will be removed in containerd 2.0. It also silently duplicates
crictl and CNI plugins already sourced independently above, risking version skew between the two
copies. containerd's own docs recommend fetching containerd, runc, and CNI plugins as three
separate artifacts instead — this decision follows that guidance.
- **runc** — added as a new, separate ingredient (`opencontainers/runc` releases), since the plain
containerd tarball — unlike the deprecated `cri-containerd-cni` bundle — does not include it.

`installer/internal/algo/ubuntu-templates/install.sh.tmpl` and `uninstall.sh.tmpl` are updated to
install/remove these files directly (`install -m 0755` for the raw binaries, `tar -x` into the
appropriate `/usr/local/...` / `/opt/cni/bin` paths for the tarballs) instead of `dpkg --install`/
`dpkg --purge`.

## 3. Consequences

**Positive.** No dependency on apt/dpkg or the Kubernetes apt repo being reachable from the host.
Ingredient versions are pinned exactly to upstream release tags rather than whatever apt happens
to resolve. kubeadm/kubelet/kubectl integrity is now checksum-verified before installation. Removes
a deprecated upstream artifact (`cri-containerd-cni-*`) before its containerd-2.0 removal breaks
the bundle outright, and removes the crictl/CNI-plugins duplication that artifact caused.

**Negative / cost.** The bundle builder now makes six separate upstream HTTP round-trips per build
instead of one `apt-get download` batch; acceptable since bundle builds are infrequent and manual
(see §5's related finding). `RUNC_VERSION` is a new version knob that must be bumped independently
of `CONTAINERD_VERSION` going forward — previously the deprecated bundle included a runc that
tracked containerd's own release cadence for free.

## 4. Alternatives considered

| Alternative | Why rejected |
|---|---|
| **Keep the `cri-containerd-cni-*` bundle, accept the deprecation** | Works today, smaller diff, but ships an artifact upstream has explicitly flagged for removal in containerd 2.0 — this bundle would need to be revisited again regardless, just later and under time pressure once it actually breaks. |
| **Switch only kubeadm/kubelet/kubectl to raw binaries, leave crictl/CNI/containerd on apt** | Rejected: apt's `cri-tools`/`kubernetes-cni` packages are the same fragile external-repo dependency this ADR is trying to remove; a half-migration keeps most of the original problem. |
| **Vendor/mirror upstream artifacts into an internal registry before the bundle builder consumes them** | Would remove the runtime dependency on GitHub/dl.k8s.io availability during bundle builds, but is a larger infrastructure change (mirroring, retention, update automation) out of scope for this fix; nothing here precludes adding it later. |

## 5. Why this is blocked

**The install/uninstall script changes in §2 are not safely deployable on their own, because the
bundle OCI tag and the manager binary that renders these scripts are not version-linked.**

`installer/bundle_downloader.go`'s `GetBundleAddr` constructs the pull tag as
`<repo>/<bundle-name>:<k8s-version>` — e.g. `quay.io/platform9/byoh-bundle-ubuntu_22.04_x86-64_k8s:
v1.32.2`. That tag encodes only the Kubernetes version, nothing about the *bundle's internal file
layout*. `install.sh.tmpl`/`uninstall.sh.tmpl` are embedded in the **manager** binary
(`//go:embed`, `installer/internal/algo/common_ubuntu.go`) and rendered fresh on every
`K8sInstallerConfig` reconcile — a manager rollout and a bundle push to quay.io happen on
completely independent schedules (one via controller deployment, the other via a human manually
running `.ci/build-push-bundle.sh`).

Concretely, both directions break:

- **A manager running this ADR's new templates, pulling an existing `v1.32.2` tag that still has
the old `.deb`-based content** (because nobody has re-pushed it yet) — `install -m 0755
"$BUNDLE_PATH/kubeadm"` fails outright, no `kubeadm` file exists in that bundle.
- **A manager still running the old `.deb`-based templates, after someone pushes a `v1.32.2` tag
rebuilt with this ADR's new content** — `dpkg --install "$BUNDLE_PATH/kubeadm.deb"` fails
outright, no `.deb` exists in the new bundle.

Since `v1.32.2` is a single mutable tag reused by every manager version that has ever requested
that Kubernetes version, there is no window in which an old-format and new-format bundle can both
be served correctly from that tag — re-pushing it to fix one manager's install path breaks every
other manager (past or future) still relying on the old format at that same tag. **Any already-
deployed manager in production is at risk the moment this bundle format changes and the
corresponding tag gets re-pushed**, regardless of merge order between the manager change and the
bundle push.

This is not a testing gap — it is a missing versioning axis in the bundle addressing scheme
itself. Resolving it requires the OCI tag to also encode a bundle-format/schema version (so old
and new formats live at non-colliding tags), which touches `installer/bundle_downloader.go`,
`installer/registry.go`, and `controllers/infrastructure/k8sinstallerconfig_controller.go` — a
separate, larger change than this ADR's ingredient-sourcing fix. Until that's designed and landed,
merging §2's install/uninstall script changes ahead of it (or behind it, in isolation) can break
already-deployed managers, and this PR must stay in draft / not be merged.
26 changes: 14 additions & 12 deletions installer/bundle_builder/build-bundle.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/bin/bash

# Copyright 2021 VMware, Inc. All Rights Reserved.
# Copyright 2026 Platform9, Inc. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0


Expand All @@ -16,23 +17,24 @@ ls -l $INGREDIENTS_PATH

cd /bundle
echo Strip version to well-known names
# Mandatory
cp $INGREDIENTS_PATH/*containerd* containerd.tar
cp $INGREDIENTS_PATH/*kubeadm*.deb ./kubeadm.deb
cp $INGREDIENTS_PATH/*kubelet*.deb ./kubelet.deb
cp $INGREDIENTS_PATH/*kubectl*.deb ./kubectl.deb
# Optional
cp $INGREDIENTS_PATH/*cri-tools*.deb cri-tools.deb > /dev/null | true
cp $INGREDIENTS_PATH/*kubernetes-cni*.deb kubernetes-cni.deb > /dev/null | true

cp "$INGREDIENTS_PATH"/kubeadm .
cp "$INGREDIENTS_PATH"/kubelet .
cp "$INGREDIENTS_PATH"/kubectl .
cp "$INGREDIENTS_PATH"/runc.* runc

cp "$INGREDIENTS_PATH"/containerd-*.tar.gz containerd.tar.gz
cp "$INGREDIENTS_PATH"/crictl-*.tar.gz crictl.tar.gz
cp "$INGREDIENTS_PATH"/cni-plugins-*.tgz cni-plugins.tgz

echo Configuration $CONFIG_PATH
ls -l $CONFIG_PATH

echo Add configuration under well-known name
(cd $CONFIG_PATH && tar -cvf conf.tar *)
cp $CONFIG_PATH/conf.tar .
(cd "$CONFIG_PATH" && tar -cvf conf.tar .)
cp "$CONFIG_PATH"/conf.tar .

echo Creating bundle tar
echo "Creating BYOH bundle tar..."
tar -cvf /bundle/bundle.tar *

echo Done
echo "Done"
12 changes: 9 additions & 3 deletions installer/bundle_builder/ingredients/deb/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
# Copyright 2021 VMware, Inc. All Rights Reserved.
# Copyright 2026 Platform9, Inc. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

# Downloads bundle ingredients : containerd as tar, kubelet, kubeadm, kubectl as Debian packages
# Downloads bundle ingredients: containerd, runc, crictl and CNI plugins as
# release tarballs/binaries, and kubeadm, kubelet, kubectl as raw binaries
# from dl.k8s.io.
#
# Usage:
# 1. Mount a host path as /ingredients
Expand All @@ -12,9 +15,12 @@ ARG BASE_IMAGE=ubuntu:20.04
FROM $BASE_IMAGE as build

# Override to download other version
ENV CONTAINERD_VERSION=1.6.26
ENV KUBERNETES_VERSION=1.26.6-00
ENV CONTAINERD_VERSION=1.7.26
ENV RUNC_VERSION=1.2.5
ENV KUBERNETES_VERSION=1.32.2
ENV ARCH=amd64
ENV OS=linux
ENV CNI_VERSION=1.3.0

RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends sudo
Expand Down
73 changes: 53 additions & 20 deletions installer/bundle_builder/ingredients/deb/download.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,62 @@

set -e

echo Update the apt package index and install packages needed to use the Kubernetes apt repository
apt-get update
apt-get install -y apt-transport-https ca-certificates curl
echo "Starting BYOH bundle ingredient download..."

echo Download containerd
curl -LOJR https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/cri-containerd-cni-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz
: "${ARCH:?ARCH must be set}"
: "${OS:?OS must be set}"
: "${CONTAINERD_VERSION:?CONTAINERD_VERSION must be set}"
: "${RUNC_VERSION:?RUNC_VERSION must be set}"
: "${KUBERNETES_VERSION:?KUBERNETES_VERSION must be set}"
: "${CRITOOL_VERSION:?CRITOOL_VERSION must be set}"
: "${CNI_VERSION:?CNI_VERSION must be set}"

echo Download the Google Cloud public signing key
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg
# Strip any trailing Debian-revision suffix (e.g. "1.32.2-1.1") left over from
# the old apt-based versioning scheme; upstream GitHub/dl.k8s.io release tags
# are plain semver.
K8S_VERSION="v${KUBERNETES_VERSION%%-*}"
CRICTL_VERSION="v${CRITOOL_VERSION%%-*}"
RUNC_VERSION="v${RUNC_VERSION%%-*}"
CNI_VERSION="v${CNI_VERSION%%-*}"

echo Add the Kubernetes apt repository
mkdir -p /ingredients

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MAJOR_VERSION}/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
echo "Downloading containerd..."
curl -LO "https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-${OS}-${ARCH}.tar.gz"
mv "containerd-${CONTAINERD_VERSION}-${OS}-${ARCH}.tar.gz" /ingredients/

mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MAJOR_VERSION}/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "Downloading runc..."
curl -LO "https://github.com/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.${ARCH}"
chmod +x "runc.${ARCH}"
mv "runc.${ARCH}" /ingredients/

echo Update apt package index, install kubelet, kubeadm and kubectl
apt-get update
chown -Rv _apt:root /bundle/
chown -R _apt:root /ingredients
mv cri-containerd-cni-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz /ingredients/
cd /ingredients
apt-get download {kubelet,kubeadm,kubectl}:$ARCH=$KUBERNETES_VERSION
apt-get download kubernetes-cni:$ARCH
apt-get download cri-tools:$ARCH=$CRITOOL_VERSION
echo "Downloading crictl..."
curl -LO "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-${OS}-${ARCH}.tar.gz"
mv "crictl-${CRICTL_VERSION}-${OS}-${ARCH}.tar.gz" /ingredients/

echo "Downloading kubeadm..."
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/${OS}/${ARCH}/kubeadm"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/${OS}/${ARCH}/kubeadm.sha256"
echo "$(cat kubeadm.sha256) kubeadm" | sha256sum --check
chmod +x kubeadm
mv kubeadm /ingredients/

echo "Downloading kubelet..."
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/${OS}/${ARCH}/kubelet"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/${OS}/${ARCH}/kubelet.sha256"
echo "$(cat kubelet.sha256) kubelet" | sha256sum --check
chmod +x kubelet
mv kubelet /ingredients/

echo "Downloading kubectl..."
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/${OS}/${ARCH}/kubectl"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/${OS}/${ARCH}/kubectl.sha256"
echo "$(cat kubectl.sha256) kubectl" | sha256sum --check
chmod +x kubectl
mv kubectl /ingredients/

echo "Downloading CNI plugins..."
curl -LO "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-${OS}-${ARCH}-${CNI_VERSION}.tgz"
mv "cni-plugins-${OS}-${ARCH}-${CNI_VERSION}.tgz" /ingredients/

echo "All ingredients downloaded and stored in /ingredients"
55 changes: 55 additions & 0 deletions installer/internal/algo/common_ubuntu_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,58 @@ func TestBaseUbuntuInstallerUninstallKernelModuleCleanup(t *testing.T) {
})
}
}

// TestBaseUbuntuInstallerInstallsBundleContent guards against install.sh.tmpl drifting from what
// the bundle builder (installer/bundle_builder/build-bundle.sh) actually packages: kubeadm,
// kubelet and kubectl as raw binaries, crictl and CNI plugins as tarballs, runc as a raw binary,
// and containerd as a plain (non-deprecated) release tarball -- not apt .deb packages.
func TestBaseUbuntuInstallerInstallsBundleContent(t *testing.T) {
installer, err := algo.NewBaseUbuntuInstaller(context.Background(), "amd64", "test-bundle", "", false)
require.NoError(t, err)

installScript := installer.Install()

wantSubstrings := []string{
`install -m 0755 "$BUNDLE_PATH/kubeadm" /usr/bin/kubeadm`,
`install -m 0755 "$BUNDLE_PATH/kubelet" /usr/bin/kubelet`,
`install -m 0755 "$BUNDLE_PATH/kubectl" /usr/bin/kubectl`,
`tar -C /usr/local/bin -xvf "$BUNDLE_PATH/crictl.tar.gz"`,
`tar -C /opt/cni/bin -xvf "$BUNDLE_PATH/cni-plugins.tgz"`,
`install -m 0755 "$BUNDLE_PATH/runc" /usr/local/sbin/runc`,
`tar -C /usr/local -xvf "$BUNDLE_PATH/containerd.tar.gz"`,
}
for _, want := range wantSubstrings {
assert.Contains(t, installScript, want)
}

unwantedSubstrings := []string{"dpkg --install", ".deb"}
for _, unwanted := range unwantedSubstrings {
assert.NotContains(t, installScript, unwanted)
}
}

// TestBaseUbuntuInstallerUninstallsBundleContent is the removal-side counterpart of
// TestBaseUbuntuInstallerInstallsBundleContent -- every path install.sh.tmpl creates must have a
// matching removal in uninstall.sh.tmpl.
func TestBaseUbuntuInstallerUninstallsBundleContent(t *testing.T) {
installer, err := algo.NewBaseUbuntuInstaller(context.Background(), "amd64", "test-bundle", "", false)
require.NoError(t, err)

uninstallScript := installer.Uninstall()

wantSubstrings := []string{
"rm -f /usr/bin/kubeadm /usr/bin/kubelet /usr/bin/kubectl",
"rm -f /usr/local/sbin/runc",
`tar tzf "$BUNDLE_PATH/crictl.tar.gz"`,
`tar tzf "$BUNDLE_PATH/containerd.tar.gz"`,
"rm -rf /opt/cni/",
}
for _, want := range wantSubstrings {
assert.Contains(t, uninstallScript, want)
}

unwantedSubstrings := []string{"dpkg --purge", "dpkg -l"}
for _, unwanted := range unwantedSubstrings {
assert.NotContains(t, uninstallScript, unwanted)
}
}
22 changes: 16 additions & 6 deletions installer/internal/algo/ubuntu-templates/install.sh.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,25 @@ fi
modprobe overlay && modprobe br_netfilter

## adding os configuration
tar -C / -xvf "$BUNDLE_PATH/conf.tar" && sysctl --system
tar -C / -xvf "$BUNDLE_PATH/conf.tar" && sysctl --system

## installing deb packages
for pkg in cri-tools kubernetes-cni kubectl kubelet kubeadm; do
dpkg --install "$BUNDLE_PATH/$pkg.deb" && apt-mark hold $pkg
done
## installing kubeadm, kubelet, kubectl binaries
install -m 0755 "$BUNDLE_PATH/kubeadm" /usr/bin/kubeadm
install -m 0755 "$BUNDLE_PATH/kubelet" /usr/bin/kubelet
install -m 0755 "$BUNDLE_PATH/kubectl" /usr/bin/kubectl

## installing crictl
tar -C /usr/local/bin -xvf "$BUNDLE_PATH/crictl.tar.gz"

## installing CNI plugins
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -xvf "$BUNDLE_PATH/cni-plugins.tgz"

## installing runc
install -m 0755 "$BUNDLE_PATH/runc" /usr/local/sbin/runc

## intalling containerd
tar -C / -xvf "$BUNDLE_PATH/containerd.tar"
tar -C /usr/local -xvf "$BUNDLE_PATH/containerd.tar.gz"
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
{{.ContainerdConfig}}
Expand Down
Loading
Loading