Description
What happened?
With immutable configmaps and secrets are not watched by the kubelet. This also means it does not refresh the loaded content when they are deleted and recreated.
The documentation of immutable configmaps though is misleading on how this needs to be handled by pods:
https://kubernetes.io/docs/concepts/configuration/configmap/#configmap-immutable
Once a ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data or the binaryData field. You can only delete and recreate the ConfigMap. Because existing Pods maintain a mount point to the deleted ConfigMap, it is recommended to recreate these pods.
This to me implies deleting and recreating the configmap and restarting the pods using it is sufficient, but it is not as the kublet will still have the old version loaded and won't update it until it is restarted.
This behavior is more or less expected but the documentation should be more explicit about this pitfall.
What did you expect to happen?
The documentation should state that immutable objects mounted into pods need to be deleted and recreated with a different name in order for the (recreated) pods to see the new version reliably.
How can we reproduce it (as minimally and precisely as possible)?
Start a deployment with an immutable configmap mounted. Change it, delete and recreate it with the same name and restart the deployment.
Pods on the kubelet will still have the old version until the kubelet is restarted and then the pods restarted.
see #42359 (comment) for an example deployment
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider
bare metall
OS version
No response