Default cluster rkeConfig chartValues wipe valid/non-empty YAML files

[shane-davidson]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

On upgrading to rancher 2.14.x the cluster rkeConfig adds default chartValues for:

rke2-calico: {}
rke2-ingress-nginx: {}
rke2-traefik: {}

These empty values wipe existing HelmChartConfigs resources in the cluster. This causes a lot of issues when upgrading or trying to use existing HelmChartConfigs. (We sync ours using fleet)

Any update to the cluster resource will wipe the Helm Chart config file every single time -> restart the ingress controller -> the file will be restored with fleet -> then restart the ingress controller again.

Can you please document this as an issue when upgrading to 2.14.x or ensure the empty config doesn't wipe existing files

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- Architecture:
- Fleet Version:
- Cluster:
  - Provider:
  - Options:
  - Kubernetes Version:

Logs

Anything else?

No response

[rancher-max]

Is this the same as and resolved by rancher/dashboard#17498?

[cmhdave]

I don't believe this is the same as the issue listed above. We are seeing this same issue but we edit our HelmChartConfig via the UI but only as "Edit as YAML" which the rancher/dashboard#17498 states should not be affected. Edited to add... this happened to us when upgrading from RKE2 1.34.3+rke2r1 to 1.35.4+rke2r1, not when upgrading from Rancher 2.13.x to 2.14.x

[rancher-max]

Thank you for the details! Let me just be sure of the flow, since you say "Edit as YAML". There are two ways that I know of to get to the YAML edit:

1. 3 dots --> Edit Config --> Edit as YAML
2. 3 dots --> Edit YAML

If Flow 1 is used, then it hits the bug mentioned, but if this is just Flow 2, then it seems we indeed do have another issue.

Attaching a couple screenshots to illustrate my meaning:

[cmhdave]

Actually, neither of those. We edit ours by going to the Rancher UI Left navigation -> More Resources -> Helm -> HelmChartConfig which has a list of the rke2-traefik, rke2-ingress-nginx, etc. On the right of those select three dots, Edit YAML.

[rancher-max]

ahh okay that's great info! I'll do some digging on my end. Likely we'll need to put in another fix for that flow, and then confirm that any similar flow also is fixed. Or, if I'm lucky, the last fix will have fixed all possible options! Will update here soon with more details -- likely tomorrow.

[cmhdave]

So to be clear here is the workflow

1. Edit rke2-traefik HelmChartConfig from More Resources -> Helm -> HelmChartConfig
2. Upgrade from RKE2 1.34.3 to 1.35.4
3. Check HelmChartConfig after upgrade and see that it is reset to default

[eva-vashkevich]

Hi @cmhdave how did you upgrade the k8s version? Was it through the Edit Config UI? Because if so, that is where you are hitting the issue Max has linked above.

[cmhdave]

Ah, yes. Upgrade through the Cluster Management -> Three dots on cluster to upgrade -> select new version.

[cmhdave]

An interesting side note... I was working on something unrelated in our upgraded cluster and restarted "rke2-server" on one of the control plane nodes. "systemctl restart rke2-server" When that is done on a single control plane node the helm install restarts for rke2-traefik and rke2-ingress-nginx which again wipes out the HelmChartConfig. So no interaction at all with the Rancher UI.

helm-install-rke2-ingress-nginx-8nq2w 0/1 Completed 0 35s
helm-install-rke2-traefik-88nk2 0/1 Completed 0 35s

[rancher-max]

I believe it's all related. Eva spotted that since it was upgraded via Edit Config, that is when the bug was triggered, and thus causing any subsequent operations with the cluster (either inside or outside of rancher manager) would be stuck with the default helm chart values for all charts.