Progressive sync out of sync when changing image sha #26585
Description
Describe the bug
In case when using progressive sync with application sets and with image sha versions defined in the generator list (which we are patching within CI when we want to rollout new image version), with v3.3 version now our ArgoCD applications are out of sync always.
When we merge some changes to the origin sources it takes some time until we run CI job that is patching image sha version in the generator list - and assumption is that sync argcd already syncs to that last merged commit, any changes later to the argo application set will result in with out of sync.
The image sha value is part of the generator list and is used inside the rendered Application spec (e.g. via kustomize/helm).
Therefore, patching the generator modifies the rendered Application spec.
To Reproduce
Example of ArgoCD Application Set - mimicking the one we are using:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: dummy
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- matrix:
generators:
- list:
elements:
- images:
dummy: "registry.gitlab.com/dummy/dummy:6d16ee7a916d27706b606f802e1930457368c9a7"
- list:
elements:
- cluster: cluster-europe-west4
region: europe-west4
- cluster: cluster-us-west1
region: us-west1
- cluster: cluster-us-east4
region: us-east4
- cluster: cluster-asia-southeast1
region: asia-southeast1
strategy:
rollingSync:
steps:
- matchExpressions:
- key: region
operator: In
values:
- us-east4
maxUpdate: 1
- matchExpressions:
- key: region
operator: In
values:
- europe-west4
maxUpdate: 1
- matchExpressions:
- key: region
operator: In
values:
- us-west1
maxUpdate: 1
- matchExpressions:
- key: region
operator: In
values:
- asia-southeast1
maxUpdate: 1
type: RollingSync
template:
metadata:
labels:
region: '{{.region}}'
name: "dummy-{{.region}}"
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: dummy
name: "{{.cluster}}"
project: prototypes
sources:
- repoURL: "https://gitlab.com/dummy/dummy.git"
path: chart
targetRevision: master
helm:
values: |
replicaCount: 1
image:
repository: registry.gitlab.com/dummy/dummy
tag: "{{ .images.dummy }}"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- Validate=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- RespectIgnoreDifferences=true
- ApplyOutOfSyncOnly=true
- ServerSideApply=true
Once we merge some change to the repo in this example to https://gitlab.com/dummy/dummy.git to master branch and after 5 minutes or so ArgoCD syncs all changes to underlaying ArgoCD Applications.
Now, after some time (crucial is that we are patching after argoCD already syncs Applications to the latest revision - change in the master branch), our CI is patching ArgoCD Application Set to change image sha in the generator list to the new one, that change won't be rollout but all argoCD Applications will stay out of sync until we manually press Sync button.
This behavior started to pop up after we migrated to v3.3, while on v2.14 we didn't have such cases.
Expected behavior
I would expected that whenever there is a change in Application set that it will streamline changes to underlaying Applications and rollout them.
Version
v3.3.0Is this behavior expected with the current RollingSync implementation in v3.3.0,
or is this a regression compared to v2.14?
If expected, what is the supported way to trigger a new progressive rollout when generator values change without modifying Git targetRevision?
The workaround we found working for us is to add a new source that is targeting to the commit sha and that way its "forcing" argoCD Applications to be in sync - the additional source does not deploy any manifests.
It is used only to force a change in targetRevisions so that RollingSync detects a new revision and starts a new rollout cycle.
- repoURL: "https://gitlab.com/dummy/dummy.git"
targetRevision: "{{ .images.dummy }}"
ref: rollout-marker