feat: bump rancher to v2.14.2-alpha6 and rke2 to v1.35.5+rke2r1 by wheatdog · Pull Request #1278 · harvester/harvester-installer

wheatdog · 2026-05-15T02:33:04Z

Problem:

Solution:

Related Issue(s):

harvester/harvester#10566

Test plan:

Additional documentation or context

Signed-off-by: Tim Liou <tim.liou@suse.com>

Copilot

Pull request overview

Version bump of Rancher to v2.14.2-alpha1 and RKE2 to v1.35.5-rc2+rke2r1, with corresponding updates to bundled images.

Changes:

Bump Rancher version (v2.14.0 → v2.14.2-alpha1) in version scripts and rancherd defaults
Bump RKE2 version (v1.35.2+rke2r1 → v1.35.5-rc2+rke2r1)
Update related bundled images (fleet, rancher-webhook, system-agent, system-upgrade-controller, turtles)

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
scripts/version-rke2	Bumps RKE2_VERSION to v1.35.5-rc2+rke2r1
scripts/version-rancher	Bumps RANCHER_VERSION to v2.14.2-alpha1
scripts/images/rancherd-bootstrap-images.txt	Updates system-agent-installer-rancher image tag
scripts/images/rancher-images.txt	Updates fleet, rancher, rancher-agent, rancher-webhook, system-agent, system-upgrade-controller, turtles image tags
package/harvester-os/files/usr/share/rancher/rancherd/config.yaml.d/50-defaults.yaml	Updates Rancher image tag and installer image tag

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

w13915984028

LGTM, thanks.

bk201 · 2026-05-15T11:24:28Z

missing images:

bk201 · 2026-05-15T11:25:13Z

@wheatdog Does the UI work for you? Mine is showing

wheatdog · 2026-05-16T02:24:40Z

@bk201 thanks for the comments! I confirmed both issues on my side. https://github.com/rancher/rancher/releases/tag/v2.14.2-alpha2 is out, I'll give it a try.

Edit: somehow, some of the image tags in https://github.com/rancher/rancher/releases/download/v2.14.2-alpha2/rancher-images.txt are incorrect. I looked up https://github.com/rancher/rancher/blob/v2.14.2-alpha2/build.yaml and https://github.com/rancher/rancher/blob/v2.14.2-alpha2/package/Dockerfile#L201 and update the tags in daeb665

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog · 2026-05-16T07:08:55Z

I'll continue checking #1278 (comment) next week.

wheatdog · 2026-05-18T10:55:51Z

Regarding of #1278 (comment), this change in Rancher v2.14.1 seems suspicious: rancher/rancher@0c91da3. I'm still testing out.

Edit: Confimed it's indeed related to rancher/rancher@0c91da3.

With the help from Claude Opus 4.7, we have the following analysis:

Root cause: rancher/rancher#54215 wrapped the cattle-local-user-passwords namespace creation in pkg/rancher/rancher.go with if features.MCM.Enabled(). In embedded mode the namespace is never created, so on first boot pkg/data/management/role_data.go:BootstrapAdmin does the following:

Creates the admin User CR (succeeds).
Calls pbkdf2.CreatePassword → fails because the namespace doesn't exist → function returns at line 485.
The GlobalRoleBinding / ClusterRoleBinding and admincreated ConfigMap creations are skipped.

On the next reconcile, BootstrapAdmin checks len(users.Items) == 0 against the unfiltered user list. The deterministic system user u-mo773yttt4 (principalIds: [system://provisioning/fleet-local/local]) is always present in embedded mode, so the bootstrap block is skipped and the admincreated ConfigMap is created, permanently closing the bootstrap path.

Result: admin exists with mustChangePassword=true, no password Secret, no admin ClusterRoleBinding. POST /v3-public/localProviders/local?action=login with admin/admin returns 401, so the Dashboard falls back to /auth/login.

Mitigation (additive, no deletes, no restart):

kubectl create namespace cattle-local-user-passwords
Create a Secret named after the admin User CR's metadata.name, namespace cattle-local-user-passwords, annotation cattle.io/password-hash: pbkdf2sha3512, data keys password and salt (both 32 bytes, pbkdf2_hmac('sha3_512', b'admin', salt, 210000, 32)), and an ownerReferences entry to the admin User.
Create a ClusterRoleBinding (generateName: default-admin-, label authz.management.cattle.io/bootstrapping=admin-user, subject = admin user, roleRef = cluster-admin).

Bootstrap login then returns 200 and the Dashboard reaches /auth/setup normally.

Report to rancher: rancher/rancher#55134

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog · 2026-05-21T06:02:28Z

Re-test result looks good after bumping rancher to 2.14.2-alpha5.

harvester-node-0:/home/rancher # helm list -f '^rancher$' -n cattle-system
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
rancher cattle-system   1               2026-05-21 05:34:20.140540691 +0000 UTC deployed        rancher-2.14.2-alpha5   840c048-dirty

harvester-node-0:/home/rancher # kubectl get ns cattle-local-user-passwords
NAME                          STATUS   AGE
cattle-local-user-passwords   Active   24m

harvester-node-0:/home/rancher # kubectl get secret -n cattle-local-user-passwords
NAME         TYPE     DATA   AGE
user-m96h5   Opaque   2      24m

harvester-node-0:/home/rancher # kubectl get ClusterRoleBinding default-admin-62h9v -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2026-05-21T05:35:00Z"
  generateName: default-admin-
  labels:
    authz.management.cattle.io/bootstrapping: admin-user
  name: default-admin-62h9v
  resourceVersion: "1507"
  uid: 8a740f56-7184-40f0-a6a6-c9f92c6a1be1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: user-m96h5

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog · 2026-05-26T10:14:12Z

PASS - Upgrade from single node v1.7.1

 # kubectl get upgrades hvst-upgrade-8fnwl -n harvester-system -o yaml
apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
  annotations:
    harvesterhci.io/image-cleanup-plan-completed: "true"
  creationTimestamp: "2026-05-26T09:40:01Z"
  finalizers:
  - wrangler.cattle.io/harvester-upgrade-controller
  generateName: hvst-upgrade-
  generation: 16
  labels:
    harvesterhci.io/latestUpgrade: "true"
    harvesterhci.io/upgradeCleanup: Succeeded
    harvesterhci.io/upgradeState: Succeeded
  name: hvst-upgrade-8fnwl
  namespace: harvester-system
  resourceVersion: "32751"
  uid: fca907d3-e0d9-4ad9-99cc-cd081d1329aa
spec:
  image: ""
  logEnabled: true
  version: v1.8.1-dev
status:
  conditions:
  - lastUpdateTime: "2026-05-26T10:09:34Z"
    status: "True"
    type: Completed
  - lastUpdateTime: "2026-05-26T09:40:36Z"
    status: "True"
    type: LogReady
  - lastUpdateTime: "2026-05-26T09:42:31Z"
    status: "True"
    type: ImageReady
  - lastUpdateTime: "2026-05-26T09:42:33Z"
    status: "True"
    type: RepoReady
  - lastUpdateTime: "2026-05-26T09:46:08Z"
    status: "True"
    type: NodesPrepared
  - lastUpdateTime: "2026-05-26T09:55:33Z"
    status: "True"
    type: SystemServicesUpgraded
  - lastUpdateTime: "2026-05-26T10:09:34Z"
    status: "True"
    type: NodesUpgraded
  imageID: harvester-system/hvst-upgrade-8fnwl
  nodeStatuses:
    harvester-node-0:
      state: Succeeded
  previousVersion: v1.7.1
  repoInfo: |
    release:
      harvester: aff907f8
      harvesterChart: 1.8.1-dev-20260524
      os: Harvester master
      kubernetes: v1.35.5+rke2r1
      rancher: v2.14.2-alpha6
      monitoringChart: 108.0.2+up77.9.1-rancher.11
  singleNode: harvester-node-0
  upgradeLog: hvst-upgrade-8fnwl-upgradelog

wheatdog · 2026-05-26T11:12:08Z

Stuck: Upgrade from v1.8.0

In logs/harvester-system/hvst-upgrade-r5nk6-apply-manifests-5chxg/apply.log

2026-05-26T08:25:32.950453759Z Happy Containering!
2026-05-26T08:25:33.073556337Z Wait for Rancher to be upgraded to v2.14.2-alpha6...
2026-05-26T08:25:38.133008005Z Wait for Rancher to be upgraded to v2.14.2-alpha6...
2026-05-26T08:25:43.373794969Z Wait for Rancher to be upgraded to v2.14.2-alpha6...
2026-05-26T08:25:48.433338208Z Wait for Rancher dependencies helm releases...
2026-05-26T08:25:48.433372816Z wait helm release cattle-fleet-system fleet fleet-109.0.2+up0.15.2-rc.3 0.15.2-rc.3 deployed
2026-05-26T08:27:04.446756475Z completed helm release cattle-fleet-system fleet fleet-109.0.2+up0.15.2-rc.3 0.15.2-rc.3 deployed
2026-05-26T08:27:04.447227192Z wait helm release cattle-fleet-system fleet-crd fleet-crd-109.0.2+up0.15.2-rc.3 0.15.2-rc.3 deployed
2026-05-26T08:27:14.693304241Z completed helm release cattle-fleet-system fleet-crd fleet-crd-109.0.2+up0.15.2-rc.3 0.15.2-rc.3 deployed
2026-05-26T08:27:14.693493121Z wait helm release cattle-system rancher-webhook rancher-webhook-109.0.2+up0.10.5-rc.4 0.10.5-rc.4 deployed
2026-05-26T08:27:24.866774472Z completed helm release cattle-system rancher-webhook rancher-webhook-109.0.2+up0.10.5-rc.4 0.10.5-rc.4 deployed
2026-05-26T08:27:24.866799012Z Wait for Rancher deployment rollout...
2026-05-26T08:27:24.866801968Z wait rollout -n cattle-system deployment rancher
2026-05-26T08:27:24.929958124Z deployment "rancher" successfully rolled out
2026-05-26T08:27:24.931215325Z Rancher deployment and pods
2026-05-26T08:27:24.966555573Z NAME      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                           SELECTOR
2026-05-26T08:27:24.966689340Z rancher   1/1     1            1           36m   rancher      rancher/rancher:v2.14.2-alpha6   app=rancher
2026-05-26T08:27:25.001492218Z NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE               NOMINATED NODE   READINESS GATES
2026-05-26T08:27:25.001565321Z rancher-558b6bc786-4vzqq   1/1     Running   0          115s   10.52.0.108   harvester-node-0   <none>           <none>
2026-05-26T08:27:25.002728496Z Wait for Rancher dependencies rollout...
2026-05-26T08:27:25.002737794Z wait rollout -n cattle-fleet-system deployment fleet-controller
2026-05-26T08:27:25.042808526Z deployment "fleet-controller" successfully rolled out
2026-05-26T08:27:25.044370512Z wait rollout -n cattle-system deployment rancher-webhook
2026-05-26T08:27:25.083911475Z deployment "rancher-webhook" successfully rolled out
2026-05-26T08:27:25.120383712Z wait rollout -n cattle-fleet-local-system deployment fleet-agent
2026-05-26T08:27:25.225660520Z deployment "fleet-agent" successfully rolled out
2026-05-26T08:27:25.227647107Z Wait for cluster settling down...
2026-05-26T08:27:25.297339029Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:27:30.377175513Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:27:35.445316041Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:27:40.513307617Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:27:45.582994248Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:27:50.640288252Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:27:55.695143662Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:28:00.771525183Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...
2026-05-26T08:28:05.834459899Z Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 2)...

https://github.com/harvester/harvester/blob/master/package/upgrade/upgrade_manifests.sh#L832-L833

From v1.8.0, embedded CAPI is replaced by Rancher Turtles #1205. Empirically, we noticed that a v1.8.x → v1.8.x (v1.9.x could also have the same issue) Rancher upgrade legitimately does not mutate the CAPI cluster kubectl get clusters.cluster.x-k8s.io -n fleet-local local. The generation never advances and apply-manifests hangs forever at "Wait for CAPI cluster settling down...".

supportbundle_323b7be4-a583-4b06-a77c-0470ab030c7f_2026-05-26T08-33-03Z.zip

I'm testing my patch of upgrade_manifests.sh to see if it mitigates this issue. I'll report later.

Edit: harvester/harvester#10719 to follow up.

wheatdog · 2026-05-27T01:31:50Z

I managed to overcome the CAPI cluster stuck (#1278 (comment)) but got another issue: mcc-harvester bundle stuck WaitApplied(1) [Cluster fleet-local/local]

In logs/harvester-system/hvst-upgrade-wg969-apply-manifests-mhpqm/apply.log

2026-05-26T14:45:19.672768764Z kubevirt comparePatches entry with jsonPointer /spec/workloadUpdateStrategy/workloadUpdateMethods already exists in harvester.yaml, skip adding
2026-05-26T14:45:19.737111517Z the fleet-local/harvester increased timeoutSeconds is 602
2026-05-26T14:45:19.740931256Z The final content of harvester patch file
2026-05-26T14:45:19.741832720Z apiVersion: management.cattle.io/v3
2026-05-26T14:45:19.741845947Z kind: ManagedChart
2026-05-26T14:45:19.741849437Z metadata:
2026-05-26T14:45:19.741851923Z   name: harvester
2026-05-26T14:45:19.741854017Z   namespace: fleet-local
2026-05-26T14:45:19.741856014Z   annotations:
2026-05-26T14:45:19.741857973Z     upgrade.harvesterhci.io/last-upgrade-name: hvst-upgrade-wg969
2026-05-26T14:45:19.741864167Z     upgrade.harvesterhci.io/last-upgrade-target-version: 0.0.0-fix-v18-upgrade-a50cf7bd
2026-05-26T14:45:19.741866047Z     upgrade.harvesterhci.io/last-upgrade-time: "2026-05-26T14:45:19Z"
2026-05-26T14:45:19.741867679Z spec:
2026-05-26T14:45:19.741869185Z   chart: harvester
2026-05-26T14:45:19.741870736Z   defaultNamespace: harvester-system
2026-05-26T14:45:19.741989388Z   diff:
2026-05-26T14:45:19.741993892Z     comparePatches:
2026-05-26T14:45:19.741996208Z       - apiVersion: storage.k8s.io/v1
2026-05-26T14:45:19.741998241Z         jsonPointers:
2026-05-26T14:45:19.742000250Z           - /metadata/annotations
2026-05-26T14:45:19.742002313Z         kind: StorageClass
2026-05-26T14:45:19.742004112Z         name: harvester-longhorn
2026-05-26T14:45:19.742005965Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742007965Z         jsonPointers:
2026-05-26T14:45:19.742009789Z           - /status/acceptedNames
2026-05-26T14:45:19.742011559Z           - /status/conditions
2026-05-26T14:45:19.742013610Z           - /status/storedVersions
2026-05-26T14:45:19.742015353Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742017069Z         name: engineimages.longhorn.io
2026-05-26T14:45:19.742018788Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742020517Z         jsonPointers:
2026-05-26T14:45:19.742022292Z           - /status/acceptedNames
2026-05-26T14:45:19.742025061Z           - /status/conditions
2026-05-26T14:45:19.742026930Z           - /status/storedVersions
2026-05-26T14:45:19.742028697Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742030439Z         name: nodes.longhorn.io
2026-05-26T14:45:19.742032388Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742329284Z         jsonPointers:
2026-05-26T14:45:19.742332691Z           - /status/acceptedNames
2026-05-26T14:45:19.742334743Z           - /status/conditions
2026-05-26T14:45:19.742336699Z           - /status/storedVersions
2026-05-26T14:45:19.742338540Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742340442Z         name: volumes.longhorn.io
2026-05-26T14:45:19.742352172Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742354056Z         jsonPointers:
2026-05-26T14:45:19.742355973Z           - /status/acceptedNames
2026-05-26T14:45:19.742357835Z           - /status/conditions
2026-05-26T14:45:19.742367312Z           - /status/storedVersions
2026-05-26T14:45:19.742369356Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742371156Z         name: settings.longhorn.io
2026-05-26T14:45:19.742372924Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742374760Z         jsonPointers:
2026-05-26T14:45:19.742376447Z           - /status/acceptedNames
2026-05-26T14:45:19.742378190Z           - /status/conditions
2026-05-26T14:45:19.742380050Z           - /status/storedVersions
2026-05-26T14:45:19.742381979Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742383854Z         name: replicas.longhorn.io
2026-05-26T14:45:19.742385757Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742387512Z         jsonPointers:
2026-05-26T14:45:19.742389252Z           - /status/acceptedNames
2026-05-26T14:45:19.742390986Z           - /status/conditions
2026-05-26T14:45:19.742392729Z           - /status/storedVersions
2026-05-26T14:45:19.742394442Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742396210Z         name: instancemanagers.longhorn.io
2026-05-26T14:45:19.742397939Z       - apiVersion: apiextensions.k8s.io/v1
2026-05-26T14:45:19.742402074Z         jsonPointers:
2026-05-26T14:45:19.742403563Z           - /status/acceptedNames
2026-05-26T14:45:19.742405074Z           - /status/conditions
2026-05-26T14:45:19.742406621Z           - /status/storedVersions
2026-05-26T14:45:19.742408067Z         kind: CustomResourceDefinition
2026-05-26T14:45:19.742409537Z         name: engines.longhorn.io
2026-05-26T14:45:19.742411016Z       - apiVersion: kubevirt.io/v1
2026-05-26T14:45:19.742412511Z         jsonPointers:
2026-05-26T14:45:19.742414310Z           - /spec/workloadUpdateStrategy/workloadUpdateMethods
2026-05-26T14:45:19.742415789Z         kind: KubeVirt
2026-05-26T14:45:19.742417439Z         name: kubevirt
2026-05-26T14:45:19.742419001Z   paused: false
2026-05-26T14:45:19.742420638Z   releaseName: harvester
2026-05-26T14:45:19.742422091Z   repoName: harvester-charts
2026-05-26T14:45:19.742423719Z   targets:
2026-05-26T14:45:19.742425234Z     - clusterName: local
2026-05-26T14:45:19.742426773Z       clusterSelector:
2026-05-26T14:45:19.742428232Z         matchExpressions:
2026-05-26T14:45:19.742429698Z           - key: provisioning.cattle.io/unmanaged-system-agent
2026-05-26T14:45:19.742431225Z             operator: DoesNotExist
2026-05-26T14:45:19.742432669Z   timeoutSeconds: 602
2026-05-26T14:45:19.742440686Z   values:
2026-05-26T14:45:19.742442217Z     containers:
2026-05-26T14:45:19.742443752Z       apiserver:
2026-05-26T14:45:19.742445448Z         hciMode: true
2026-05-26T14:45:19.742446974Z         image:
2026-05-26T14:45:19.742448474Z           imagePullPolicy: IfNotPresent
2026-05-26T14:45:19.742449977Z     harvester-load-balancer:
2026-05-26T14:45:19.742451999Z       enabled: true
2026-05-26T14:45:19.742453456Z     harvester-network-controller:
2026-05-26T14:45:19.742454992Z       enabled: true
2026-05-26T14:45:19.742482341Z       image:
2026-05-26T14:45:19.742484358Z         pullPolicy: IfNotPresent
2026-05-26T14:45:19.742486071Z       vipEnabled: true
2026-05-26T14:45:19.742487851Z     harvester-networkfs-manager:
2026-05-26T14:45:19.742489617Z       enabled: true
2026-05-26T14:45:19.742491328Z       image:
2026-05-26T14:45:19.742493056Z         pullPolicy: IfNotPresent
2026-05-26T14:45:19.742494877Z     harvester-node-disk-manager:
2026-05-26T14:45:19.742496701Z       enabled: true
2026-05-26T14:45:19.742498425Z       image:
2026-05-26T14:45:19.742500148Z         pullPolicy: IfNotPresent
2026-05-26T14:45:19.742501948Z       labelFilter:
2026-05-26T14:45:19.742503688Z         - COS_*
2026-05-26T14:45:19.742505564Z         - HARV_*
2026-05-26T14:45:19.742522588Z     kube-vip:
2026-05-26T14:45:19.742525102Z       enabled: true
2026-05-26T14:45:19.742526978Z     kube-vip-cloud-provider:
2026-05-26T14:45:19.742528696Z       enabled: true
2026-05-26T14:45:19.742530416Z     kubevirt:
2026-05-26T14:45:19.742532167Z       spec:
2026-05-26T14:45:19.742533899Z         monitorAccount: rancher-monitoring-operator
2026-05-26T14:45:19.742540220Z         monitorNamespace: cattle-monitoring-system
2026-05-26T14:45:19.742542281Z     longhorn:
2026-05-26T14:45:19.742544048Z       defaultSettings:
2026-05-26T14:45:19.742546027Z         concurrentAutomaticEngineUpgradePerNodeLimit: "null"
2026-05-26T14:45:19.742547827Z         defaultDataPath: /var/lib/harvester/defaultdisk
2026-05-26T14:45:19.742549762Z         detachManuallyAttachedVolumesWhenCordoned: true
2026-05-26T14:45:19.742551684Z         nodeDrainPolicy: allow-if-replica-is-stopped
2026-05-26T14:45:19.742553433Z         storageReservedPercentageForDefaultDisk: "0"
2026-05-26T14:45:19.742556656Z         taintToleration: kubevirt.io/drain:NoSchedule
2026-05-26T14:45:19.742558150Z       enabled: true
2026-05-26T14:45:19.742559787Z     multus:
2026-05-26T14:45:19.742561243Z       enabled: false
2026-05-26T14:45:19.742562740Z     promote:
2026-05-26T14:45:19.742564248Z       clusterDNS: 10.53.0.10
2026-05-26T14:45:19.742585595Z       clusterPodCIDR: 10.52.0.0/16
2026-05-26T14:45:19.742588388Z       clusterServiceCIDR: 10.53.0.0/16
2026-05-26T14:45:19.742590117Z     rancherEmbedded: true
2026-05-26T14:45:19.742591624Z     service:
2026-05-26T14:45:19.742593124Z       vip:
2026-05-26T14:45:19.742594693Z         enabled: true
2026-05-26T14:45:19.742604451Z         hwAddress: 02:00:00:03:3D:61
2026-05-26T14:45:19.742606212Z         ip: 192.168.10.131
2026-05-26T14:45:19.742607698Z         mode: dhcp
2026-05-26T14:45:19.742609287Z     webhook:
2026-05-26T14:45:19.742610945Z       image:
2026-05-26T14:45:19.742612468Z         imagePullPolicy: IfNotPresent
2026-05-26T14:45:19.742614027Z   version: 0.0.0-fix-v18-upgrade-a50cf7bd
2026-05-26T14:45:19.742617046Z Upgrading...
2026-05-26T14:45:20.144259840Z managedchart.management.cattle.io/harvester configured
2026-05-26T14:45:20.151392055Z Waiting for ManagedChart fleet-local/harvester from generation 4
2026-05-26T14:45:20.151410974Z Target version: 0.0.0-fix-v18-upgrade-a50cf7bd, Target state: ready
2026-05-26T14:45:20.247453972Z Current version: 0.0.0-fix-v18-upgrade-a50cf7bd, Current ready clusters: 0/1, Current state: WaitApplied, Current generation: 4
2026-05-26T14:45:25.249829783Z Sleep for 5 seconds to retry
2026-05-26T14:45:25.349049383Z Current version: 0.0.0-fix-v18-upgrade-a50cf7bd, Current ready clusters: 0/1, Current state: WaitApplied, Current generation: 5
2026-05-26T14:45:30.354185655Z Sleep for 5 seconds to retry
2026-05-26T14:45:30.486466006Z Current version: 0.0.0-fix-v18-upgrade-a50cf7bd, Current ready clusters: 0/1, Current state: WaitApplied, Current generation: 5

In logs/cattle-fleet-local-system/fleet-agent-87cbf764c-74ml2/fleet-agent.log

2026-05-26T15:28:25.737578630Z {"level":"error","ts":"2026-05-26T15:28:25Z","msg":"Reconciler error","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"mcc-harvester","namespace":"cluster-fleet-local-local-1a3d67d0a899"},"namespace":"cluster-fleet-local-local-1a3d67d0a899","name":"mcc-harvester","reconcileID":"c1b5ed1e-0be1-4423-aa02-8b7ade27260e","error":"failed deploying bundle: conflict occurred while applying object harvester-system/kubevirt kubevirt.io/v1, Kind=KubeVirt: Apply failed with 1 conflict: conflict with \"kubectl-patch\" using kubevirt.io/v1: .spec.workloadUpdateStrategy.workloadUpdateMethods","errorCauses":[{"error":"failed deploying bundle: conflict occurred while applying object harvester-system/kubevirt kubevirt.io/v1, Kind=KubeVirt: Apply failed with 1 conflict: conflict with \"kubectl-patch\" using kubevirt.io/v1: .spec.workloadUpdateStrategy.workloadUpdateMethods"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:495\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313"}

In yamls/namespaced/harvester-system/kubevirt.io/v1/kubevirts.yaml

    - apiVersion: kubevirt.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:workloadUpdateStrategy:
            f:workloadUpdateMethods: {}
      manager: kubectl-patch
      operation: Update
      time: "2026-05-26T14:44:19Z"

It seems related to harvester/harvester#10349 harvester/harvester#10386

https://github.com/harvester/harvester/blob/bfc85b48cbd6283fc50060244e7713a1c05fdcbf/package/upgrade/upgrade_manifests.sh#L1022-L1034

supportbundle_292e5abd-38d1-4282-88cb-7aab593c65d5_2026-05-26T15-32-15Z.zip

Update:

Thanks @bk201's help. We notice this line in logs/harvester-system/hvst-upgrade-wg969-apply-manifests-mhpqm/apply.log

2026-05-26T14:45:19.672768764Z kubevirt comparePatches entry with jsonPointer /spec/workloadUpdateStrategy/workloadUpdateMethods already exists in harvester.yaml, skip addingand

there are 2 apply-manifest pods (the 1st one failed in the middle with transient error)

hvst-upgrade-wg969-apply-manifests-mhpqm/              <- 2nd
hvst-upgrade-wg969-apply-manifests-nbbrg/              <- 1st

And this issue is not reproducible when the 1st apply-manifest pod succeed.

bk201

Thanks!

wheatdog · 2026-05-27T07:42:15Z

I'm going to merge this PR first. We'll bump rancher to v2.14.2 when it's out. Thanks!

wheatdog · 2026-05-27T07:45:45Z

@Mergifyio backport master

mergify · 2026-05-27T07:45:50Z

backport master

✅ Backports have been created

Details

#1283 feat: bump rancher to v2.14.2-alpha6 and rke2 to v1.35.5+rke2r1 (backport #1278) has been created for branch master

Signed-off-by: Tim Liou <tim.liou@suse.com> (cherry picked from commit faabeb3)

feat: bump rke2 to v1.35.5-rc2+rke2r1

304cd0c

Signed-off-by: Tim Liou <tim.liou@suse.com>

Copilot AI review requested due to automatic review settings May 15, 2026 02:33

Copilot started reviewing on behalf of wheatdog May 15, 2026 02:33 View session

wheatdog requested review from Vicente-Cheng, bk201, starbops and w13915984028 May 15, 2026 02:34

Copilot AI reviewed May 15, 2026

View reviewed changes

w13915984028 previously approved these changes May 15, 2026

View reviewed changes

feat: bump rancher to v2.14.2-alpha2

368823b

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog dismissed w13915984028’s stale review via 368823b May 16, 2026 02:48

wheatdog force-pushed the 10566-bump branch from aa3bc9d to 368823b Compare May 16, 2026 02:48

wheatdog changed the title ~~feat: bump rancher to v2.14.2-alpha1 and rke2 to v1.35.5-rc2+rke2r1~~ feat: bump rancher to v2.14.2-alpha2 and rke2 to v1.35.5-rc2+rke2r1 May 16, 2026

Update rancher-webhook and turtles image tags

daeb665

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog mentioned this pull request May 19, 2026

[BUG] Embedded Rancher v2.14.1 in Harvester: first login redirects to /auth/login instead of /auth/setup rancher/rancher#55134

Open

wheatdog added 2 commits May 21, 2026 12:02

feat: bump rancher to v2.14.2-alpha5

4627cb7

Signed-off-by: Tim Liou <tim.liou@suse.com>

feat: bump rke2 to v1.35.5+rke2r1

0a2e925

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog changed the title ~~feat: bump rancher to v2.14.2-alpha2 and rke2 to v1.35.5-rc2+rke2r1~~ feat: bump rancher to v2.14.2-alpha5 and rke2 to v1.35.5+rke2r1 May 21, 2026

feat: bump rancher to v2.14.2-alpha6

9c014a9

Signed-off-by: Tim Liou <tim.liou@suse.com>

wheatdog changed the title ~~feat: bump rancher to v2.14.2-alpha5 and rke2 to v1.35.5+rke2r1~~ feat: bump rancher to v2.14.2-alpha6 and rke2 to v1.35.5+rke2r1 May 25, 2026

wheatdog mentioned this pull request May 27, 2026

fix(upgrade): wait_capi_cluster no longer requires a generation bump harvester/harvester#10672

Merged

bk201 approved these changes May 27, 2026

View reviewed changes

wheatdog merged commit faabeb3 into harvester:v1.8 May 27, 2026
10 checks passed

mergify Bot mentioned this pull request May 27, 2026

feat: bump rancher to v2.14.2-alpha6 and rke2 to v1.35.5+rke2r1 (backport #1278) #1283

Merged

mergify Bot pushed a commit that referenced this pull request May 27, 2026

feat: bump rancher to v2.14.2-alpha6 and rke2 to v1.35.5+rke2r1 (#1278)

294f08c

Signed-off-by: Tim Liou <tim.liou@suse.com> (cherry picked from commit faabeb3)

bk201 mentioned this pull request May 28, 2026

fix: override kubevirt workloadUpdateMethods with chart values harvester/harvester#10675

Merged

Conversation

wheatdog commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem:

Solution:

Related Issue(s):

Test plan:

Additional documentation or context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

w13915984028 left a comment

Choose a reason for hiding this comment

Uh oh!

bk201 commented May 15, 2026

Uh oh!

bk201 commented May 15, 2026

Uh oh!

wheatdog commented May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wheatdog commented May 16, 2026

Uh oh!

wheatdog commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wheatdog commented May 21, 2026

Uh oh!

wheatdog commented May 26, 2026

Uh oh!

wheatdog commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wheatdog commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bk201 left a comment

Choose a reason for hiding this comment

Uh oh!

wheatdog commented May 27, 2026

Uh oh!

Uh oh!

wheatdog commented May 27, 2026

Uh oh!

mergify Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wheatdog commented May 15, 2026 •

edited

Loading

wheatdog commented May 16, 2026 •

edited

Loading

wheatdog commented May 18, 2026 •

edited

Loading

wheatdog commented May 26, 2026 •

edited

Loading

wheatdog commented May 27, 2026 •

edited

Loading

mergify Bot commented May 27, 2026 •

edited

Loading