[ISSUE] Helm upgrade without secrets extraction steps does not trigger validation error between v9 and v10

[jessesimpson36]

Describe the issue:

https://camunda.slack.com/archives/C03UR0V2R2M/p1715673565744409

There are situations where a user may upgrade from 9.3.3 to 10.0.5 without specifying their existingSecrets, and for the error message must not be empty, please add '--set ... to not happen.

Actual behavior:

Helm upgrade will fail initially due to

Error: UPGRADE FAILED: cannot patch "cpt-identity" with kind Deployment: Deployment.apps "cpt-identity" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"camunda-platform", "app.kubernetes.io/component":"identity", "app.kubernetes.io/instance":"cpt", "app.kubernetes.io/managed-by":"Helm", "app.kubernetes.io/name":"camunda-platform", "app.kubernetes.io/part-of":"camunda-platform"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

And if you delete the identity deployment to bypass that, then the helm upgrade will succeed unexpectedly.

Expected behavior:

Error message is expected due to not supplying existingSecrets during the upgrade process:

Error: UPGRADE FAILED: execution error at (camunda-platform/charts/identity/templates/tasklist-secret.yaml:10:22):
PASSWORDS ERROR: You must provide your current passwords when upgrading the release.
                 Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.
                 Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases

    'global.identity.auth.tasklist.existingSecret' must not be empty, please add '--set global.identity.auth.tasklist.existingSecret=$TASKLIST_SECRET' to the command. To get the current value:

        export TASKLIST_SECRET=$(kubectl get secret --namespace "camunda" "camunda-platform-test-tasklist-identity-secret" -o jsonpath="{.data.tasklist-secret}" | base64 --decode)

How to reproduce:

1. Install helm install cpt camunda/camunda-platform --version 9.3.3
2. Upgrade helm upgrade cpt camunda/camunda-platform --version 10.0.5
3. If you didn't get any validation errors before it tries to apply a manifest, then you've reproduced it.

Logs:

No error message means no logs 😢 .

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform: KIND
• Helm CLI version: v3.15.0
• Chart version: 9.3.3 -> 10.0.5
• Values file: default values.yaml

[jessesimpson36]

If you run into

Error: UPGRADE FAILED: cannot patch "cpt-identity" with kind Deployment: Deployment.apps "cpt-identity" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"camunda-platform", "app.kubernetes.io/component":"identity", "app.kubernetes.io/instance":"cpt", "app.kubernetes.io/managed-by":"Helm", "app.kubernetes.io/name":"camunda-platform", "app.kubernetes.io/part-of":"camunda-platform"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

when testing, you can run

kubectl delete deployment cpt-identity

And then continue with the upgrade. I would still expect the secrets extraction validation error message to appear, but it does not

[jessesimpson36]

After doing a git bisect, I found that the following commit caused this behavior:

7fb7b0a feat(pre-alpha): initial support multi-namespace deployment (#1084)

I used the following script to find it

#!/bin/bash
releaseName=cpt make -e helm.uninstall
helm install --set global.identity.keycloak.url.host=jesse.com --set global.identity.keycloak.url.port=80 --set global.identity.keycloak.url.protocol=http cpt charts/camunda-platform 
export go_to="$(git log --oneline main | grep -B 1 $(git rev-parse --short HEAD) | tail -n 1 | awk '{print $1}')"
export return_to="$(git rev-parse --short HEAD)"
git checkout "$go_to"
helm upgrade --set global.identity.keycloak.url.host=jesse.com --set global.identity.keycloak.url.port=80 --set global.identity.keycloak.url.protocol=http cpt charts/camunda-platform 
git checkout $return_to

[jessesimpson36]

lookup is used to look up resource in a running cluster. When used with the helm template command it always returns an empty response.

https://helm.sh/docs/chart_template_guide/function_list/#lookup

I think this might be relevant.