Unexpected Kubernetes Rollout Restart Deployment Behavior

[RickArroues]

I've floated the information below through a few other communities a few months back with no feedback. More troubleshooting went down this week and one of the steps taken was to uninstall ArgoCD from a cluster and then attempt a rollout restart of all deployments. That resulted in a 100% successful restart.

So I'm now pasting my earlier troubleshooting here, but with the added scenario where my problem seems to go away with Argo. I'm after evidence of what we may have misconfigured or if there's an app bug in play.

Also I believe it's worth pointing out that we have two new deployments that seem to be added to the misbehaving list of 4 below. This makes me think we're somehow influencing this, but I haven't been able to figure out where/how when comparing working apps to misbehaving apps.

Now on to my historical notes...

I have 4 deployments that are not restarting properly out of ~25 that are developed internally. They all live in the same namespace in EKS and are deployed by Helm via ArgoCD. These apps are also deployed to 15+ AWS accounts and the behavior is consistent in each cluster.

When attempting a kubectl rollout restart all but the four deployments restart as you’d expect. We have default settings in play so 25% of each deployments’ pods go down at a time and new ones spin up. The other four however behave differently. Their behavior is like so:

1. DeploymentApp1 is running with ReplicaSetApp1.rev5 with 3 pods running
2. You perform a rollout restart on DeploymentApp1.
3. DeploymentApp1 scales up ReplicaSetApp1.rev6
4. DeploymentApp1 scales down ReplicaSetApp1.rev6
5. ReplicaSetApp1.rev5 becomes ReplicaSetApp1.rev7 with 3 pods running with no change to pod status/age.
6. Events/logs show you a successful restart.

I’ve investigated leads where too many replicasets exist which in some past meant things got confused, but we use the default of 10 and none are over the limit. Killing all the replicasets to start back at one did not influence the behavior.

• Deleting the deployment and allowing ArgoCD to put it back did not change the behavior on subsequent rollout restarts. It did of course result in everything being destroyed and spinning back up.
• Deleting a single pod results in a new pod spinning up as expected.
• Having ArgoCD send the restart duplicates the unexpected behavior. My testing is all native otherwise.
• Labels and annotations seem to match up at every level.
• We don’t set maxSurge or maxAvailable

[RickArroues]

I recently stumbled across https://stackoverflow.com/questions/59050709/how-to-rollout-restart-deployment-through-the-api which caused me to look at our Helm configuration. Figured out the deployments having the issue all referenced a dependent chart for annotations. That dependent chart had no annotations listed which resulted in the manifest having a value of 'null'. Null in Helm means remove.

So our configuration was set to remove annotations which impacted kubectl.kubernetes.io/restartedAt from being applied and resulted in our weird behavior.

[Avi-Gupta1]

I am a newbie with ArgoCD, I also came across this issue but even after adding that in annotaion, i had to specifically update my applicationset to add that particular annotation in IgnoreDifferences hope that helps.
e.g.

ignoreDifferences:
    - group: apps
      jsonPointers:
        - /spec/template/metadata/annotations/kubectl.kubernetes.io~1restartedAt
      kind: Deployment

[anhqqt]

I am a newbie with ArgoCD, I also came across this issue but even after adding that in annotaion, i had to specifically update my applicationset to add that particular annotation in IgnoreDifferences hope that helps. e.g.

ignoreDifferences:
    - group: apps
      jsonPointers:
        - /spec/template/metadata/annotations/kubectl.kubernetes.io~1restartedAt
      kind: Deployment

Thank you so much. You saved me. I find another approach that is applying the ignoreDifferences globally in argocd-cm

data:
  resource.customizations.ignoreDifferences.apps_Deployment: |
    jqPathExpressions:
      - .spec.template.metadata.annotations."kubectl.kubernetes.io/restartedAt"
      - if (.spec.template.metadata.annotations | keys | length) == 0 then .spec.template.metadata.annotations else empty end