Running `argocd sync app --dry-run` commands affect metrics

[jsolana]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Related to #21661

Currently, executing the command argocd app sync --dry-run affects both the application’s state and the internal metrics exposed by ArgoCD (eg: argocd_app_info).

The main issue is that if there are alerts based on these metrics, and the dry-run execution identifies an error (e.g., a change that violates a Kyverno policy or an invalid CRD schema), the application state changes to SyncErr. This also updates the metrics, which can potentially trigger alerts based on these metrics.

For example:

# alert manager alert definition
- alert: ArgoCDApplicationUnknown
  expr: sum by (cluster) (argocd_app_info{sync_status="Unknown"}) > 0
  for: 15m
...

Since the definition of a dry-run is to execute requests without persistence, I wonder if it makes sense to handle it in a way that ensures no changes are made.

Alternatively, adding a dry_run label to differentiate operational requests from dry-run requests could also be an option (a similar change has been proposed for kyverno link).

To Reproduce

Run argocd app sync --dry-run command.

Expected behavior

There are different proposals:

1. Add a dryrun label to distinguish dryrun activity from real ones.
2. Ignore dryrun executions (not affecting metrics).

Initially I fond of the 1 because dryrun has a cost associated in terms of resources consumption / performance. Ignoring activity related to dryrun make extremely hard to identify the reason of performance issues.

Version

It is affecting whatever version because currently dryrun executions are not distinguished.

[harshitSkyscanner]

I am trying to get more exposure to argocd codebase and develop some understanding. It looks like an easy fix if it is just to add a label.
I can help contribute to it.

[jsolana]

Thanks mate, I ve just create this MR to fix it. Let me know wdyt

[agaudreault]

Have you investigated why the sync status changes for dry runs? This seems more like it should be fixed by not modifying the persisted sync status on dry runs rather than adding it to the metrics.

[harshitSkyscanner]

Thanks mate, I ve just create this MR to fix it. Let me know wdyt

Ahh, I just saw this. I created the PR too. I did more or less the same thing but also updated some other metrics as well. I wasn't sure about my approach so I didn't added tests just yet.

#22014

[harshitSkyscanner]

Have you investigated why the sync status changes for dry runs? This seems more like it should be fixed by not modifying the persisted sync status on dry runs rather than adding it to the metrics.

@agaudreault - That is something that can be investigated in parallel but should not affect this PR. Metrics should have an indicator if they were generated via dry run or a normal operation.

[jsolana]

Thanks mate, I ve just create this MR to fix it. Let me know wdyt

Ahh, I just saw this. I created the PR too. I did more or less the same thing but also updated some other metrics as well. I wasn't sure about my approach so I didn't added tests just yet.

#22014

Taking in count this comment not sure if this is the way to go.

Have you investigated why the sync status changes for dry runs? This seems more like it should be fixed by not modifying the persisted sync status on dry runs rather than adding it to the metrics.

The idea is to split this issue in two different ones:

• This one related to metrics
• Another one related to change and persist the app status running a dryrun op

Wdyt?

Want to know more about how repo server persists app status, any idea where to look?
Thanks!

[harshitSkyscanner]

Want to know more about how repo server persists app status, any idea where to look?
Thanks!

I have been looking into this today. Shouldn't this be somewhere in controller/sync.go?
So maybe a fix like this - #22064

I am happy to add more stuff to it if I am even remotely on the right track here.

Also, just to clarify what happens here is during dry run sync, it is that updates the application sync state incorrectly right?
(Any logs here would be helpful as well to see the flow of code)

[crenshaw-dev]

I agree that it's weird to update the sync state for a dry run... but if we don't do that, does the controller have a way to communicate the success/failure of the dry run up to the UI/CLI? If not, maybe instead of not persisting the state, we just need to persist it to a different field so that we preserve both the real sync state and the dry run sync state?

[harshitSkyscanner]

I agree that it's weird to update the sync state for a dry run... but if we don't do that, does the controller have a way to communicate the success/failure of the dry run up to the UI/CLI? If not, maybe instead of not persisting the state, we just need to persist it to a different field so that we preserve both the real sync state and the dry run sync state?

I was just trying to reproduce this in my local with argo-cd version 2.14.2.

The full behaviour that I can see is -
SYNC STATUS - remains unchanged
LAST SYNC - Changes to whatever the result of dry run sync was

I kind of agree with @crenshaw-dev here. Looking at the code, if we don't update anything, we won't be able to return the result back correctly. So we need to handle the results of dry run sync in a different manner

[crenshaw-dev]

I think @jsolana's PR probably solves the metrics problem. But yeah, a more general solution would probably involve CRD, controller, API, CLI, and UI work.