Skip to content

fix(lifecycle): support Kubernetes 1.33 upgrade flow#6929

Merged
cuisongliu merged 10 commits into
labring:mainfrom
lingdie:fix-1.33
Jun 10, 2026
Merged

fix(lifecycle): support Kubernetes 1.33 upgrade flow#6929
cuisongliu merged 10 commits into
labring:mainfrom
lingdie:fix-1.33

Conversation

@lingdie

@lingdie lingdie commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

What

  • Wait for the embedded registry endpoint before CRI/kubeadm pulls images, avoiding transient connection refused failures during cluster creation.
  • Install the target kubeadm before listing/pulling upgrade images so the image matrix matches the target Kubernetes minor version.
  • Avoid passing InitConfiguration/ClusterConfiguration via kubeadm upgrade apply --config for newer kubeadm, and retry node upgrade while the API server settles.

Validation

  • Built linux/amd64 sealos with Docker.
  • Replaced sealos on root@192.168.10.230 and ran sealos reset --force.
  • Verified sealos run docker.io/labring/kubernetes:v1.33.3 --masters=192.168.10.230 --nodes=192.168.0.185 succeeds.
  • Confirmed logs no longer contain registry connection refused or image pull failures.

@lingdie
lingdie requested a review from a team as a code owner April 28, 2026 03:50
@lingdie lingdie changed the title fix: support Kubernetes 1.33 upgrade flow fix(lifecycle): support Kubernetes 1.33 upgrade flow Apr 28, 2026
@lingdie

lingdie commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator Author

Added a follow-up fix for the 1.26 -> 1.27 kubelet failure path:

  • Fix UpdateKubeletConfig to write kubelet-config data under KubeletBaseConfigurationConfigMapKey; it was previously writing under the ConfigMap name key, so cleaned KubeletConfiguration was not actually used by kubeadm.
  • Sync converted kubelet config to each node before kubelet restarts during upgrade, so removed/GA feature gates do not break the new kubelet.
  • If the mounted Kubernetes image version already matches but node kubelet versions differ, or NodeReady is Unknown, continue upgrade recovery instead of skipping.

Validation on root@192.168.10.230:

  • Reproduced kubelet v1.27.1 failing with unrecognized feature gate: EphemeralContainers from stale /var/lib/kubelet/config.yaml.
  • Rebuilt linux/amd64 sealos and reran sealos run docker.io/labring/kubernetes:v1.27.1 without manually editing kubelet config.
  • Verified kubelet is active, node yy-dev-v2 is Ready on v1.27.1, and both local kubelet config plus kubelet-config ConfigMap no longer contain EphemeralContainers.
  • Added/ran go test ./pkg/client-go/kubernetes -run TestUpdateKubeletConfigUsesConfigMapDataKey.

@lingdie

lingdie commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator Author

Updated with recovery fixes from the 1.26 -> 1.27 remote upgrade test:

  • ignore kubeadm ControlPlaneNodesReady as a warning during upgrade apply so CNI-not-ready recovery states do not block version upgrades
  • sanitize kubelet config for targets below v1.30 to avoid strict decode warnings from newer fields (containerRuntimeEndpoint, imageMaximumGCAge, logging.options.text)
  • clean removed kubelet args from 10-kubeadm.conf, kubelet-flags.env, and kubeadm-flags.env so kubelet v1.27+ does not fail on --container-runtime
  • suppress old-rootfs sealctl render noise and make temporary registry serve command fall back across flag variants before SSH fallback

Validated on root@192.168.10.230 with sealos run docker.io/labring/kubernetes:v1.27: control plane and kubelet reached v1.27.1; kubelet is active. The node remains NotReady only because CNI is not installed in this Kubernetes-only test cluster.

@lingdie

lingdie commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up for the old rootfs sealctl issue:

  • MountRootfs now refreshes /var/lib/sealos/data/<cluster>/rootfs/opt/sealctl from the local sealctl next to/in PATH of the running sealos when the local binary is Linux and the remote arch matches. It skips the copy if the remote digest already matches.
  • temporary registry sync now probes whether the rootfs sealctl registry serve filesystem supports --port or -p before starting it. If unsupported, it directly uses SSH sync, so old rootfs images no longer print unknown flag: --port / unknown shorthand flag: p.

Validated on root@192.168.10.230 by deleting the old rootfs opt/sealctl, installing rebuilt sealos + sealctl, and running sealos run docker.io/labring/kubernetes:v1.27. The command auto-restored rootfs opt/sealctl; the log no longer contains unknown flag, unknown shorthand, or temporary registry connection fallback noise.

@lingdie

lingdie commented Apr 29, 2026

Copy link
Copy Markdown
Collaborator Author

Update for the temporary registry issue seen as error exit status 1 during MirrorRegistry:

  • Root cause on the test host was an orphaned sealctl registry serve filesystem still listening on :5050, so a new temporary registry failed to bind while stderr was redirected.
  • The CLI now reuses an already-alive remote temporary registry before starting a new one.
  • New starts use --pid-file/--disable-logging only when the remote rootfs sealctl supports those flags, and cleanup is scoped to the expected rootfs sealctl registry serve filesystem process.
  • Added registry command generation unit tests.

Local/remote verification:

  • go test ./pkg/filesystem/registry -count=1 passed in linux/amd64 Docker.
  • Built linux/amd64 sealos and sealctl.
  • Installed them on 192.168.10.230; with the stale :5050 process still present, sealos run docker.io/labring/kubernetes:v1.29.15 passed MirrorRegistry without error exit status 1, unknown flag, or bind-address noise. It then stopped later because cluster metadata rejects v1.29.15 as older than desired v1.30.6.

@lingdie

lingdie commented Apr 29, 2026

Copy link
Copy Markdown
Collaborator Author

Update for the 1.29 -> 1.30 upgrade failure:

Root cause: kubeadm v1.30 runs the upgrade health-check Job (CreateJob) before applying the control-plane upgrade. In the recovery/no-CNI/NotReady scenarios we are exercising, that Job may not complete in 15s and kubeadm treats it as fatal unless explicitly ignored. Sealos already ignored SystemVerification and ControlPlaneNodesReady; this adds CreateJob to the upgrade apply ignore list without using all.

Verification on 192.168.10.230:

  • Rebuilt linux/amd64 sealos and installed it on the host.
  • Re-ran sealos run docker.io/labring/kubernetes:v1.30.6 from the failed 1.29.15 -> 1.30.6 state.
  • kubeadm passed the previous health-check point and printed [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.30.6".
  • kubectl version now reports server/client v1.30.6; node yy-dev-v2 is Ready with kubelet v1.30.6.

Tests:

  • go test ./pkg/runtime/kubernetes ./pkg/filesystem/registry -count=1 passed in linux/amd64 Docker.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the lifecycle Kubernetes upgrade/create flows to better support Kubernetes v1.33.x by (a) reducing transient embedded-registry connectivity failures during image pulls, and (b) aligning kubeadm/kubelet configuration handling with newer kubeadm behaviors during upgrades.

Changes:

  • Wait for the embedded registry endpoint to become reachable before kubeadm/CRI image pull operations.
  • Install the target kubeadm binary before listing/pulling upgrade images so the image list matches the target Kubernetes minor version.
  • Adjust upgrade behavior for newer kubeadm (avoid kubeadm upgrade apply --config with non-UpgradeConfiguration) and add retry logic for node upgrade while the API server settles; add tests around kubelet config sanitization and registry serve command generation.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
lifecycle/pkg/runtime/kubernetes/upgrade.go Updates upgrade command flow, kubelet config sync/sanitization, and adds retry/seq helpers.
lifecycle/pkg/runtime/kubernetes/upgrade_test.go Adds unit tests for kubelet config sanitization and upgrade apply flags.
lifecycle/pkg/runtime/kubernetes/runtime.go Allows “same version” upgrades to proceed when nodes are not aligned (kubelet version/readiness).
lifecycle/pkg/runtime/kubernetes/master.go Ensures registry is reachable before listing/pulling images.
lifecycle/pkg/runtime/kubernetes/kubeadm.go Adds V1300 semver constant used for kubelet config sanitization gating.
lifecycle/pkg/registry/helpers/helpers.go Adds WaitRegistryReady helper (bash/timeout + /dev/tcp probe) and shell quoting.
lifecycle/pkg/filesystem/rootfs/rootfs_default.go Attempts to sync a local sealctl to remote rootfs before rendering templates.
lifecycle/pkg/filesystem/registry/sync.go Improves temporary registry lifecycle (probe flags, pid-file cleanup, readiness checks, fallback modes).
lifecycle/pkg/filesystem/registry/sync_test.go Adds tests for registry serve/cleanup command construction.
lifecycle/pkg/client-go/kubernetes/expansion.go Fixes kubelet ConfigMap update to use the correct data key constant.
lifecycle/pkg/client-go/kubernetes/expansion_test.go Adds a fake-client test validating the correct ConfigMap key is used.
lifecycle/pkg/bootstrap/registry.go Waits for registry readiness after starting the registry service.

Comment on lines +132 to +139
go func(target string) {
if err := waitRemoteTemporaryRegistry(ctx, target, defaultTemporaryPort); err != nil {
logger.Warn("cannot connect to remote temporary registry %s: %v, fallback using ssh mode instead", ep, err)
syncOptionChan <- &syncOption{target: target, typ: sshMode}
} else {
syncOptionChan <- &syncOption{target: ep, typ: httpMode}
}
}(hosts[i])
Comment on lines 129 to 134
upgradeConfigName := "kubeadm-upgrade.yaml"
upgradeConfigPath := path.Join(k.pathResolver.EtcPath(), upgradeConfigName)

err = k.sshCmdAsync(master0ip,
//install kubeadm:{version} at master0
fmt.Sprintf(installKubeadmCmd, kubeBinaryPath),
err = k.sshCmdAsyncSeq(master0ip,
// write kubeadm config to file
fmt.Sprintf(writeKubeadmConfig, upgradeConfigPath, string(config)),
* build(lifecycle): auto-detect C compiler and CC

Add CC and CC_<platform> Makefile options and implement compiler
resolution for CGO-enabled sealos/sealctl builds. Update DEVELOPGUIDE
with usage and a cross-compile example.

Signed-off-by: Yun Pan <dinoallo@netc.it>

* fix(kubernetes): align cert identity handling with kubeadm v1.29+

- migrate remote control-plane certs across the v1.29 identity boundary
- repair remote admin.conf on all masters during upgrade
- keep local sealos PKI and kubeconfigs aligned with the v1.29+ model
- keep super-admin.conf local-only during renew and upgrade flows
- preserve pre-v1.31 long-lived cert policy and use v1beta4 fields on
  v1.31+
- fix v1.31 remote admin.conf migration config distribution across
  masters

Signed-off-by: Yun Pan <dinoallo@netc.it>

---------

Signed-off-by: Yun Pan <dinoallo@netc.it>
@pull-request-size

Copy link
Copy Markdown

Whoa! Easy there, Partner!

This PR is too big. Please break it up into smaller PRs.

@cuisongliu cuisongliu added this to the v5.2 milestone May 15, 2026
@dinoallo
dinoallo requested a review from Copilot May 15, 2026 04:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

lifecycle/pkg/runtime/kubernetes/upgrade.go:392

  • This local identity refresh regenerates the full PKI set, which requires the local etcd CA files. External-etcd kubeadm clusters do not have /etc/kubernetes/pki/etcd/ca.{crt,key} managed locally, so this step can fail even though no local etcd certificates need renewal. Carry the hasLocalEtcd result into this path and renew only the certs/kubeconfigs that are present for external-etcd clusters.
	if err := cert.RenewLeafCertsForKubeVersion(
		k.pathResolver.PkiPath(),
		k.pathResolver.PkiEtcdPath(),

lifecycle/pkg/runtime/kubernetes/upgrade.go:622

  • This has the same ordering issue as master0: the target kubelet config is written before the existing kubelet may be restarted by CRI/flag cleanup. When the target config contains fields unsupported by the current kubelet, the node can be broken before the target kubelet is installed. Defer this sync until after installing the target kubelet, or sanitize using the current kubelet version for the pre-install restart window.
		if err := k.syncKubeletConfig(ip, conversion, version); err != nil {

Comment on lines +240 to +245
for _, caFile := range []string{
"ca.crt", "ca.key",
"front-proxy-ca.crt", "front-proxy-ca.key",
"etcd/ca.crt", "etcd/ca.key",
"sa.key", "sa.pub",
} {
func (k *KubeadmRuntime) upgradeMaster0(conversion *types.ConvertedKubeadmConfig, version string) error {
master0ip := k.getMaster0IP()
sver := semver.MustParse(version)
if err := k.syncKubeletConfig(master0ip, conversion, version); err != nil {

@caijian76 caijian76 May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I try use this branch to upgrade 1.29->1.30,but unsuccess.
图片
‘’‘
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.30.9"
[upgrade/versions] Cluster version: v1.29.9
[upgrade/versions] kubeadm version: v1.30.9
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W0520 12:23:08.448258 178213 checks.go:844] detected that the sandbox image "sealos.hub:5000/pause:3.9" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.9" as the CRI sandbox image.
2026-05-20T12:28:04 error upgrade cluster failed
2026-05-20T12:28:04 info using v1beta3 kubeadm config
Error: signal: killed
’‘’

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you provide more info? sealos run with debug flag, os info, etc...

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A clean Ubuntu 22.04 environment. Install Kubernetes using the official Sealos tool with the command:
sealos run registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9 registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4 registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4 --single
Wait for Kubernetes to become ready.
Run git clone -b fix-1.33 --single-branch https://github.com/lingdie/sealos.git, compile Sealos, then upgrade Kubernetes by running:
sealos-fix133 run registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9
The upgrade process hangs at the following output for 5 minutes before throwing an error:

[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W0521 17:14:58.479649   20302 checks.go:844] detected that the sandbox image "sealos.hub:5000/pause:3.9" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.9" as the CRI sandbox image.
2026-05-21T17:19:54 error upgrade cluster failed

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

full log:

root@cj:~# ./sealos run registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9 --debug
2026-05-21T17:14:46 debug using file /etc/containers/storage.conf as container storage config
2026-05-21T17:14:46 debug loadClusterFile loadRenderValues: map[]
2026-05-21T17:14:46 debug rendered Clusterfile: apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  creationTimestamp: "2026-05-21T16:20:13Z"
  name: default
spec:
  hosts:
  - ips:
    - 10.211.55.5:22
    roles:
    - master
    - amd64
  image:
  - registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9
  - registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4
  - registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4
  ssh: {}
status:
  conditions:
  - lastHeartbeatTime: "2026-05-21T16:20:13Z"
    message: Applied to cluster successfully
    reason: Ready
    status: "True"
    type: ApplyClusterSuccess
  mounts:
  - env:
      SEALOS_SYS_CRI_ENDPOINT: /var/run/containerd/containerd.sock
      SEALOS_SYS_IMAGE_ENDPOINT: /var/run/image-cri-shim.sock
      criData: /var/lib/containerd
      defaultVIP: 10.103.97.2
      disableApparmor: "false"
      registryConfig: /etc/registry
      registryData: /var/lib/registry
      registryDomain: sealos.hub
      registryPassword: passw0rd
      registryPort: "5000"
      registryUsername: admin
      sandboxImage: pause:3.9
    imageName: registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9
    labels:
      check: check.sh $registryData
      clean: clean.sh && bash clean-cri.sh $criData
      clean-registry: clean-registry.sh $registryData $registryConfig
      image: ghcr.io/labring/lvscare:v5.0.1
      init: init-cri.sh $registryDomain $registryPort && bash init.sh
      init-registry: init-registry.sh $registryData $registryConfig
      io.buildah.version: 1.30.0
      org.opencontainers.image.description: kubernetes-v1.29.9 container image
      org.opencontainers.image.licenses: MIT
      org.opencontainers.image.source: https://github.com/labring-actions/cache
      sealos.io.type: rootfs
      sealos.io.version: v1beta1
      version: v1.29.9
      vip: $defaultVIP
    mountPoint: /var/lib/containers/storage/overlay/8c2c24d5400c9edd2dce877c316e3d0476d9f038a61ea5aadcd747c4838d714c/merged
    name: default-k8cueg42
    type: rootfs
  - cmd:
    - cp -f opt/helm /usr/bin/helm
    imageName: registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4
    labels:
      io.buildah.version: 1.30.0
    mountPoint: /var/lib/containers/storage/overlay/ba0c58ab35166c5f61ea9917a4ad06aa9f2e1b3509973f7087ef340402dd45e0/merged
    name: default-wvljii5r
    type: application
  - cmd:
    - cp opt/cilium /usr/bin/
    - cp opt/hubble /usr/bin/
    - cilium install --chart-directory charts/cilium --helm-set kubeProxyReplacement=strict,k8sServiceHost=apiserver.cluster.local,k8sServicePort=6443
    imageName: registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4
    labels:
      io.buildah.version: 1.30.0
    mountPoint: /var/lib/containers/storage/overlay/563ba6045e7a8c1bb4adcab2be197dc5c8a335b6f540d0b2673ce30497906764/merged
    name: default-erwimaps
    type: application
  phase: ClusterSuccess

2026-05-21T17:14:46 debug run new images: [registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9]
2026-05-21T17:14:46 info start to install app in this cluster
2026-05-21T17:14:46 info Executing SyncStatusAndCheck Pipeline in InstallProcessor
2026-05-21T17:14:46 debug sync cluster status is: apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  creationTimestamp: "2026-05-21T16:20:13Z"
  name: default
spec:
  hosts:
  - ips:
    - 10.211.55.5:22
    roles:
    - master
    - amd64
  image:
  - registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9
  - registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4
  - registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4
  ssh: {}
status:
  conditions:
  - lastHeartbeatTime: "2026-05-21T16:20:13Z"
    message: Applied to cluster successfully
    reason: Ready
    status: "True"
    type: ApplyClusterSuccess
  mounts:
  - env:
      SEALOS_SYS_CRI_ENDPOINT: /var/run/containerd/containerd.sock
      SEALOS_SYS_IMAGE_ENDPOINT: /var/run/image-cri-shim.sock
      criData: /var/lib/containerd
      defaultVIP: 10.103.97.2
      disableApparmor: "false"
      registryConfig: /etc/registry
      registryData: /var/lib/registry
      registryDomain: sealos.hub
      registryPassword: passw0rd
      registryPort: "5000"
      registryUsername: admin
      sandboxImage: pause:3.9
    imageName: registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9
    labels:
      check: check.sh $registryData
      clean: clean.sh && bash clean-cri.sh $criData
      clean-registry: clean-registry.sh $registryData $registryConfig
      image: ghcr.io/labring/lvscare:v5.0.1
      init: init-cri.sh $registryDomain $registryPort && bash init.sh
      init-registry: init-registry.sh $registryData $registryConfig
      io.buildah.version: 1.30.0
      org.opencontainers.image.description: kubernetes-v1.29.9 container image
      org.opencontainers.image.licenses: MIT
      org.opencontainers.image.source: https://github.com/labring-actions/cache
      sealos.io.type: rootfs
      sealos.io.version: v1beta1
      version: v1.29.9
      vip: $defaultVIP
    mountPoint: /var/lib/containers/storage/overlay/8c2c24d5400c9edd2dce877c316e3d0476d9f038a61ea5aadcd747c4838d714c/merged
    name: default-k8cueg42
    type: rootfs
  - cmd:
    - cp -f opt/helm /usr/bin/helm
    imageName: registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4
    labels:
      io.buildah.version: 1.30.0
    mountPoint: /var/lib/containers/storage/overlay/ba0c58ab35166c5f61ea9917a4ad06aa9f2e1b3509973f7087ef340402dd45e0/merged
    name: default-wvljii5r
    type: application
  - cmd:
    - cp opt/cilium /usr/bin/
    - cp opt/hubble /usr/bin/
    - cilium install --chart-directory charts/cilium --helm-set kubeProxyReplacement=strict,k8sServiceHost=apiserver.cluster.local,k8sServicePort=6443
    imageName: registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4
    labels:
      io.buildah.version: 1.30.0
    mountPoint: /var/lib/containers/storage/overlay/563ba6045e7a8c1bb4adcab2be197dc5c8a335b6f540d0b2673ce30497906764/merged
    name: default-erwimaps
    type: application
  phase: ClusterSuccess

2026-05-21T17:14:46 info Executing ConfirmOverrideApps Pipeline in InstallProcessor
2026-05-21T17:14:46 info Executing PreProcess Pipeline in InstallProcessor
2026-05-21T17:14:46 debug images f52e993f9df1a1200751fac07fb0ecadddbddc1cd45468a05e83a859fd114a80 are pulled
2026-05-21T17:14:46 debug parse reference f52e993f9df1a1200751fac07fb0ecadddbddc1cd45468a05e83a859fd114a80 with transport containers-storage
2026-05-21T17:14:46 debug Pull Policy for pull [missing]
2026-05-21T17:14:46 debug parse reference f52e993f9df1a1200751fac07fb0ecadddbddc1cd45468a05e83a859fd114a80 with transport containers-storage
2026-05-21T17:14:46 debug clusterfile config is empty!
2026-05-21T17:14:46 info Executing pipeline MountRootfs in InstallProcessor.
2026-05-21T17:14:46 debug render env dir: /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/etc
2026-05-21T17:14:46 debug render env dir: /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/scripts
2026-05-21T17:14:46 debug render env dir: /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/manifests
2026-05-21T17:14:46 debug Directory /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/manifests does not exist, skipping
2026-05-21T17:14:49 debug send mount image, target: 10.211.55.5:22, image: registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9, type: rootfs
2026-05-21T17:14:49 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/Kubefile to dst /var/lib/sealos/data/default/rootfs/Kubefile on 10.211.55.5:22 locally
2026-05-21T17:14:49 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/README.md to dst /var/lib/sealos/data/default/rootfs/README.md on 10.211.55.5:22 locally
2026-05-21T17:14:49 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/bin to dst /var/lib/sealos/data/default/rootfs/bin on 10.211.55.5:22 locally
2026-05-21T17:14:50 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/cri to dst /var/lib/sealos/data/default/rootfs/cri on 10.211.55.5:22 locally
2026-05-21T17:14:50 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/etc to dst /var/lib/sealos/data/default/rootfs/etc on 10.211.55.5:22 locally
2026-05-21T17:14:50 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/images to dst /var/lib/sealos/data/default/rootfs/images on 10.211.55.5:22 locally
2026-05-21T17:14:50 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/opt to dst /var/lib/sealos/data/default/rootfs/opt on 10.211.55.5:22 locally
2026-05-21T17:14:50 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/scripts to dst /var/lib/sealos/data/default/rootfs/scripts on 10.211.55.5:22 locally
2026-05-21T17:14:50 debug copy files src /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/statics to dst /var/lib/sealos/data/default/rootfs/statics on 10.211.55.5:22 locally
2026-05-21T17:14:50 info Executing pipeline MirrorRegistry in InstallProcessor.
2026-05-21T17:14:50 debug registry nodes is: [10.211.55.5:22]
2026-05-21T17:14:50 info trying default http mode to sync images to hosts [10.211.55.5:22]
2026-05-21T17:14:50 debug checking if endpoint http://10.211.55.5:5050 is alive
2026-05-21T17:14:51 debug running temporary registry on host 10.211.55.5:22
2026-05-21T17:14:51 debug checking if endpoint http://10.211.55.5:5050 is alive
2026-05-21T17:14:51 debug http endpoint http://10.211.55.5:5050 is alive
2026-05-21T17:14:51 debug registry config: &{Version:0.1 Log:{AccessLog:{Disabled:true} Level:info Formatter: Fields:map[] Hooks:[]} Loglevel: Storage:map[filesystem:map[rootdirectory:/var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged/registry] maintenance:map[uploadpurging:map[enabled:false]]] Auth:map[] Middleware:map[] Reporting:{Bugsnag:{APIKey: ReleaseStage: Endpoint:} NewRelic:{LicenseKey: Name: Verbose:false}} HTTP:{Addr::23069 Net: Host: Prefix: Secret:asecretforlocaldevelopment RelativeURLs:false DrainTimeout:0s TLS:{Certificate: Key: ClientCAs:[] MinimumTLS: CipherSuites:[] LetsEncrypt:{CacheFile: Email: Hosts:[]}} Headers:map[] Debug:{Addr: Prometheus:{Enabled:false Path:}} HTTP2:{Disabled:false}} Notifications:{EventConfig:{IncludeReferences:false} Endpoints:[]} Redis:{Addr: Password: DB:0 TLS:{Enabled:false} DialTimeout:0s ReadTimeout:0s WriteTimeout:0s Pool:{MaxIdle:0 MaxActive:0 IdleTimeout:0s}} Health:{FileCheckers:[] HTTPCheckers:[] TCPCheckers:[] StorageDriver:{Enabled:false Interval:0s Threshold:0}} Proxy:{RemoteURL: Username: Password:} Compatibility:{Schema1:{TrustKey: Enabled:false}} Validation:{Enabled:false Disabled:false Manifests:{URLs:{Allow:[] Deny:[]}}} Policy:{Repository:{Classes:[]}}}
2026-05-21T17:14:51 debug checking if endpoint http://127.0.0.1:23069 is alive
2026-05-21T17:14:51 debug http endpoint http://127.0.0.1:23069 is alive
2026-05-21T17:14:51 debug syncing repos [{coredns/coredns  0 false false false} {etcd  0 false false false} {kube-apiserver  0 false false false} {kube-controller-manager  0 false false false} {kube-proxy  0 false false false} {kube-scheduler  0 false false false} {labring/lvscare  0 false false false} {pause  0 false false false}] from 127.0.0.1:23069 to 10.211.55.5:5050
2026-05-21T17:14:51 debug syncing 10.211.55.5:5050/coredns/coredns:v1.11.3 with selection 1
2026-05-21T17:14:51 debug syncing 10.211.55.5:5050/coredns/coredns:v1.11.3 with selection 0
2026-05-21T17:14:51 debug syncing 10.211.55.5:5050/etcd:3.5.15-0 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/etcd:3.5.15-0 with selection 0
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-apiserver:v1.30.9 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-apiserver:v1.30.9 with selection 0
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-controller-manager:v1.30.9 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-controller-manager:v1.30.9 with selection 0
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-proxy:v1.30.9 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-proxy:v1.30.9 with selection 0
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-scheduler:v1.30.9 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/kube-scheduler:v1.30.9 with selection 0
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/labring/lvscare:v5.0.1 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/labring/lvscare:v5.0.1 with selection 0
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/pause:3.9 with selection 1
2026-05-21T17:14:52 debug syncing 10.211.55.5:5050/pause:3.9 with selection 0
2026-05-21T17:14:52 info Executing UpgradeIfNeed Pipeline in InstallProcessor
2026-05-21T17:14:52 info Change ClusterConfiguration up to newVersion if need.
2026-05-21T17:14:52 debug get cluster configmap data:
apiServer:
  certSANs:
  - 127.0.0.1
  - apiserver.cluster.local
  - 10.103.97.2
  - 10.211.55.5
  extraArgs:
    audit-log-format: json
    audit-log-maxage: "7"
    audit-log-maxbackup: "10"
    audit-log-maxsize: "100"
    audit-log-path: /var/log/kubernetes/audit.log
    audit-policy-file: /etc/kubernetes/audit-policy.yml
    authorization-mode: Node,RBAC
    enable-aggregator-routing: "true"
    feature-gates: ""
  extraVolumes:
  - hostPath: /etc/kubernetes
    mountPath: /etc/kubernetes
    name: audit
    pathType: DirectoryOrCreate
  - hostPath: /var/log/kubernetes
    mountPath: /var/log/kubernetes
    name: audit-log
    pathType: DirectoryOrCreate
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    pathType: File
    readOnly: true
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: apiserver.cluster.local:6443
controllerManager:
  extraArgs:
    bind-address: 0.0.0.0
    cluster-signing-duration: 876000h
    feature-gates: ""
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    pathType: File
    readOnly: true
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.30.9
networking:
  dnsDomain: cluster.local
  podSubnet: 100.64.0.0/10
  serviceSubnet: 10.96.0.0/22
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
    feature-gates: ""
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    pathType: File
    readOnly: true

2026-05-21T17:14:52 debug get kubelet configmap data:
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebugFlagsHandler: true
enableDebuggingHandlers: true
enableProfilingHandler: true
enableServer: true
enableSystemLogHandler: true
enforceNodeAllocatable:
- pods
eventBurst: 100
eventRecordQPS: 50
evictionHard:
  imagefs.available: 10%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 0.0.0.0
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMaximumGCAge: 0s
imageMinimumGCAge: 2m0s
imageServiceEndpoint: unix:///var/run/image-cri-shim.sock
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 100
kubeAPIQPS: 50
localStorageCapacityIsolation: true
logging:
  flushFrequency: 5000000000
  format: text
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
memoryManagerPolicy: None
memorySwap: {}
memoryThrottlingFactor: 0.8
nodeLeaseDurationSeconds: 40
nodeStatusMaxImages: 50
nodeStatusReportFrequency: 10s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registerNode: true
registryBurst: 100
registryPullQPS: 50
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
seccompDefault: false
serializeImagePulls: false
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
topologyManagerPolicy: none
topologyManagerScope: container
volumePluginDir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
volumeStatsAggPeriod: 1m0s

2026-05-21T17:14:52 info using v1beta3 kubeadm config
2026-05-21T17:14:52 debug update cluster config:
apiServer:
  certSANs:
  - 127.0.0.1
  - apiserver.cluster.local
  - 10.103.97.2
  - 10.211.55.5
  extraArgs:
    audit-log-format: json
    audit-log-maxage: "7"
    audit-log-maxbackup: "10"
    audit-log-maxsize: "100"
    audit-log-path: /var/log/kubernetes/audit.log
    audit-policy-file: /etc/kubernetes/audit-policy.yml
    authorization-mode: Node,RBAC
    enable-aggregator-routing: "true"
    feature-gates: ""
  extraVolumes:
  - hostPath: /etc/kubernetes
    mountPath: /etc/kubernetes
    name: audit
    pathType: DirectoryOrCreate
  - hostPath: /var/log/kubernetes
    mountPath: /var/log/kubernetes
    name: audit-log
    pathType: DirectoryOrCreate
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    pathType: File
    readOnly: true
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: apiserver.cluster.local:6443
controllerManager:
  extraArgs:
    bind-address: 0.0.0.0
    cluster-signing-duration: 876000h
    feature-gates: ""
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    pathType: File
    readOnly: true
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.30.9
networking:
  dnsDomain: cluster.local
  podSubnet: 100.64.0.0/10
  serviceSubnet: 10.96.0.0/22
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
    feature-gates: ""
  extraVolumes:
  - hostPath: /etc/localtime
    mountPath: /etc/localtime
    name: localtime
    pathType: File
    readOnly: true

2026-05-21T17:14:52 debug update kubelet config:
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebugFlagsHandler: true
enableDebuggingHandlers: true
enableProfilingHandler: true
enableServer: true
enableSystemLogHandler: true
enforceNodeAllocatable:
- pods
eventBurst: 100
eventRecordQPS: 50
evictionHard:
  imagefs.available: 10%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 0.0.0.0
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMaximumGCAge: 0s
imageMinimumGCAge: 2m0s
imageServiceEndpoint: unix:///var/run/image-cri-shim.sock
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 100
kubeAPIQPS: 50
localStorageCapacityIsolation: true
logging:
  flushFrequency: 5000000000
  format: text
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
memoryManagerPolicy: None
memorySwap: {}
memoryThrottlingFactor: 0.8
nodeLeaseDurationSeconds: 40
nodeStatusMaxImages: 50
nodeStatusReportFrequency: 10s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registerNode: true
registryBurst: 100
registryPullQPS: 50
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
seccompDefault: false
serializeImagePulls: false
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
topologyManagerPolicy: none
topologyManagerScope: container
volumePluginDir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
volumeStatsAggPeriod: 1m0s

2026-05-21T17:14:52 info start to upgrade master0
2026-05-21T17:14:52 info sync kubelet config to node 10.211.55.5
2026-05-21T17:14:53 debug registry config data info: # Copyright © 2022 sealos.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

domain: sealos.hub
port: "5000"
username: "admin"
password: "passw0rd"
data: "/var/lib/registry"

2026-05-21T17:14:53 debug show registry info, IP: 10.211.55.5:22, Domain: sealos.hub, Data: /var/lib/registry
2026-05-21T17:14:53 info waiting for registry sealos.hub:5000 to be ready on 10.211.55.5
2026-05-21T17:14:53 info start to pull images: sealos.hub:5000/kube-apiserver:v1.30.9, sealos.hub:5000/kube-controller-manager:v1.30.9, sealos.hub:5000/kube-scheduler:v1.30.9, sealos.hub:5000/kube-proxy:v1.30.9, sealos.hub:5000/coredns/coredns:v1.11.3, sealos.hub:5000/pause:3.9, sealos.hub:5000/etcd:3.5.15-0
Image is up to date for sha256:4f53be91109c4dd4658bb0141e8af556b94293ec9fad72b2b62a617edb48e5c4
2026-05-21T17:14:53 info succeeded in pulling image sealos.hub:5000/kube-apiserver:v1.30.9
Image is up to date for sha256:d4203c1bb2593a7429c3df3c040da333190e5d7e01f377d0255b7b813ca09568
2026-05-21T17:14:53 info succeeded in pulling image sealos.hub:5000/kube-controller-manager:v1.30.9
Image is up to date for sha256:41cce68b0c8c3c4862ff55ac17be57616cce36a04e719aee733e5c7c1a24b725
2026-05-21T17:14:53 info succeeded in pulling image sealos.hub:5000/kube-scheduler:v1.30.9
Image is up to date for sha256:4c369683c359609256b8907f424fc2355f1e7e3eeb7295b1fd8ffc5304f4cede
2026-05-21T17:14:53 info succeeded in pulling image sealos.hub:5000/kube-proxy:v1.30.9
Image is up to date for sha256:c69fa2e9cbf5f42dc48af631e956d3f95724c13f91596bc567591790e5e36db6
2026-05-21T17:14:54 info succeeded in pulling image sealos.hub:5000/coredns/coredns:v1.11.3
Image is up to date for sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c
2026-05-21T17:14:54 info succeeded in pulling image sealos.hub:5000/pause:3.9
Image is up to date for sha256:2e96e5913fc06e3d26915af3d0f2ca5048cc4b6327e661e80da792cbf8d8d9d4
2026-05-21T17:14:54 info succeeded in pulling image sealos.hub:5000/etcd:3.5.15-0
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0521 17:14:54.431760   20302 utils.go:69] The recommended value for "healthzBindAddress" in "KubeletConfiguration" is: 127.0.0.1; the provided value is: 0.0.0.0
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.30.9"
[upgrade/versions] Cluster version: v1.29.9
[upgrade/versions] kubeadm version: v1.30.9
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
W0521 17:14:58.479649   20302 checks.go:844] detected that the sandbox image "sealos.hub:5000/pause:3.9" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.9" as the CRI sandbox image.
2026-05-21T17:19:54 error upgrade cluster failed
2026-05-21T17:19:54 info using v1beta3 kubeadm config
2026-05-21T17:19:54 debug save objects into local: /root/.sealos/default/Clusterfile, objects: apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  creationTimestamp: "2026-05-21T16:20:13Z"
  name: default
spec:
  hosts:
  - ips:
    - 10.211.55.5:22
    roles:
    - master
    - amd64
  image:
  - registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9
  - registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4
  - registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4
  - registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9
  ssh: {}
status:
  commandCondition:
  - images:
    - registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9
    lastHeartbeatTime: "2026-05-21T17:19:54Z"
    message: 'signal: killed'
    reason: Apply Command
    status: "False"
    type: ApplyCommandError
  conditions:
  - lastHeartbeatTime: "2026-05-21T16:20:13Z"
    message: Applied to cluster successfully
    reason: Ready
    status: "True"
    type: ApplyClusterSuccess
  mounts:
  - env:
      SEALOS_SYS_CRI_ENDPOINT: /var/run/containerd/containerd.sock
      SEALOS_SYS_IMAGE_ENDPOINT: /var/run/image-cri-shim.sock
      criData: /var/lib/containerd
      defaultVIP: 10.103.97.2
      disableApparmor: "false"
      registryConfig: /etc/registry
      registryData: /var/lib/registry
      registryDomain: sealos.hub
      registryPassword: passw0rd
      registryPort: "5000"
      registryUsername: admin
      sandboxImage: pause:3.9
    imageName: registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.29.9
    labels:
      check: check.sh $registryData
      clean: clean.sh && bash clean-cri.sh $criData
      clean-registry: clean-registry.sh $registryData $registryConfig
      image: ghcr.io/labring/lvscare:v5.0.1
      init: init-cri.sh $registryDomain $registryPort && bash init.sh
      init-registry: init-registry.sh $registryData $registryConfig
      io.buildah.version: 1.30.0
      org.opencontainers.image.description: kubernetes-v1.29.9 container image
      org.opencontainers.image.licenses: MIT
      org.opencontainers.image.source: https://github.com/labring-actions/cache
      sealos.io.type: rootfs
      sealos.io.version: v1beta1
      version: v1.29.9
      vip: $defaultVIP
    mountPoint: /var/lib/containers/storage/overlay/8c2c24d5400c9edd2dce877c316e3d0476d9f038a61ea5aadcd747c4838d714c/merged
    name: default-k8cueg42
    type: rootfs
  - cmd:
    - cp -f opt/helm /usr/bin/helm
    imageName: registry.cn-shanghai.aliyuncs.com/labring/helm:v3.9.4
    labels:
      io.buildah.version: 1.30.0
    mountPoint: /var/lib/containers/storage/overlay/ba0c58ab35166c5f61ea9917a4ad06aa9f2e1b3509973f7087ef340402dd45e0/merged
    name: default-wvljii5r
    type: application
  - cmd:
    - cp opt/cilium /usr/bin/
    - cp opt/hubble /usr/bin/
    - cilium install --chart-directory charts/cilium --helm-set kubeProxyReplacement=strict,k8sServiceHost=apiserver.cluster.local,k8sServicePort=6443
    imageName: registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.13.4
    labels:
      io.buildah.version: 1.30.0
    mountPoint: /var/lib/containers/storage/overlay/563ba6045e7a8c1bb4adcab2be197dc5c8a335b6f540d0b2673ce30497906764/merged
    name: default-erwimaps
    type: application
  - env:
      SEALOS_SYS_CRI_ENDPOINT: /var/run/containerd/containerd.sock
      SEALOS_SYS_IMAGE_ENDPOINT: /var/run/image-cri-shim.sock
      criData: /var/lib/containerd
      defaultVIP: 10.103.97.2
      disableApparmor: "false"
      registryConfig: /etc/registry
      registryData: /var/lib/registry
      registryDomain: sealos.hub
      registryPassword: passw0rd
      registryPort: "5000"
      registryUsername: admin
      sandboxImage: pause:3.9
    imageName: registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.30.9
    labels:
      check: check.sh $registryData
      clean: clean.sh && bash clean-cri.sh $criData
      clean-registry: clean-registry.sh $registryData $registryConfig
      image: ghcr.io/labring/lvscare:v5.0.1
      init: init-cri.sh $registryDomain $registryPort && bash init.sh
      init-registry: init-registry.sh $registryData $registryConfig
      io.buildah.version: 1.30.0
      org.opencontainers.image.description: kubernetes-v1.30.9 container image
      org.opencontainers.image.licenses: MIT
      org.opencontainers.image.source: https://github.com/labring-actions/cache
      sealos.io.type: rootfs
      sealos.io.version: v1beta1
      version: v1.30.9
      vip: $defaultVIP
    mountPoint: /var/lib/containers/storage/overlay/0054838d40c8ba79aa671c19f9a164befc183f7a72c0a693ca9fe5416692f797/merged
    name: default-amyqzbb6
    type: rootfs
  phase: ClusterSuccess

2026-05-21T17:19:54 debug sync workdir: /root/.sealos/default
2026-05-21T17:19:54 debug copy files src /root/.sealos/default to dst /root/.sealos/default on 10.211.55.5:22 locally
Error: signal: killed

@lingdie lingdie May 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try this commit: lingdie@8ed6e89

git clone -b fix-1.33 --single-branch https://github.com/lingdie/sealos.git
git checkout 8ed6e89a1d78b763176108461c70ddd83cb0265a

I successfully upgraded from 1.29 to 1.30 using this commit.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commit 8ed6 It still cannot operate normally, with the same error occurring.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lingdie I identified the cause of the failure: the process got hold while pulling the image registry.k8s.io/xxx:yyy. but local image names sealos.hub:5000/xxx.yyy. I manually tagged sealos.hub:5000/kube-apiserver:v1.30.5 as registry.k8s.io/kube-apiserver:v1.30.5, along with other related images, and the upgrade completed successfully.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird~

@github-actions github-actions Bot added the area/imageCRIShim about image-cri-shim label May 27, 2026
@github-actions github-actions Bot removed the area/imageCRIShim about image-cri-shim label Jun 9, 2026
@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.86%. Comparing base (9632e12) to head (30ba675).
⚠️ Report is 53 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #6929   +/-   ##
=======================================
  Coverage   61.86%   61.86%           
=======================================
  Files           8        8           
  Lines         653      653           
=======================================
  Hits          404      404           
  Misses        202      202           
  Partials       47       47           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lingdie

lingdie commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

use this image list:

docker.io/labring/kubernetes:v1.25.16
docker.io/labring/kubernetes:v1.26.15
docker.io/labring/kubernetes:v1.27.16
docker.io/labring/kubernetes:v1.28.15
docker.io/labring/kubernetes:v1.29.15
docker.io/labring/kubernetes:v1.30.14
docker.io/labring/kubernetes:v1.31.14
docker.io/labring/kubernetes:v1.32.10
docker.io/labring/kubernetes:v1.33.6

@cuisongliu
cuisongliu merged commit 14a1998 into labring:main Jun 10, 2026
93 of 100 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🤖 says: cherry pick action finished successfully 🎉!
See: https://github.com/labring/sealos/actions/runs/27264444803

cuisongliu pushed a commit that referenced this pull request Jun 10, 2026
fix(lifecycle): support Kubernetes 1.33 upgrade flow (#6929)

* fix: wait for registry before pulling kubeadm images

* fix: support kubeadm upgrade to newer minors

* fix: recover kubelet config during upgrades

* fix: handle kubelet upgrade recovery edge cases

* fix: refresh rootfs sealctl before use

* fix: tolerate existing temporary registry

* fix: ignore kubeadm upgrade health check job

* fix(kubernetes): align cert identity handling with kubeadm v1.29+ (#83)

* build(lifecycle): auto-detect C compiler and CC

Add CC and CC_<platform> Makefile options and implement compiler
resolution for CGO-enabled sealos/sealctl builds. Update DEVELOPGUIDE
with usage and a cross-compile example.



* fix(kubernetes): align cert identity handling with kubeadm v1.29+

- migrate remote control-plane certs across the v1.29 identity boundary
- repair remote admin.conf on all masters during upgrade
- keep local sealos PKI and kubeconfigs aligned with the v1.29+ model
- keep super-admin.conf local-only during renew and upgrade flows
- preserve pre-v1.31 long-lived cert policy and use v1beta4 fields on
  v1.31+
- fix v1.31 remote admin.conf migration config distribution across
  masters



---------



* fix(lifecycle): sync and upgrade codex

* fix: cert and add timeout for cluster role binding create

---------

Signed-off-by: Yun Pan <dinoallo@netc.it>
Co-authored-by: yy <56745951+lingdie@users.noreply.github.com>
Co-authored-by: Yun Pan <dinoallo@netc.it>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants