Skip to content

Conversation

dylantangredi-jakala
Copy link
Collaborator

@dylantangredi-jakala dylantangredi-jakala commented Oct 6, 2025

List of Changes

  • upgraded helm chart to v8.0.2
  • added rolling update strategy (maxUnavailable, maxSurge)
  • overridden deployment replicas to match HPA minReplica config

NOTE: upgraded in

  • DEV -> OK
  • UAT -> OK
  • PROD -> OK

Motivation and Context

This upgrade was required to address an issue with the rollout strategy and the autoscaler behaviour, where the pods would scale down to 1 during deploy instead of keeping the same number of pods and doing +1 -1 to avoid traffic congestion. Also replicas are set to match the autoscaling config during deploy, to avoid starting the service with 1 pod when there is relevant traffic on the service.
This also ensures zero-downtime deployments by keeping all existing pods running until new versions are healthy and ready, then replacing them one at a time.

How Has This Been Tested?

❯ helm dependency update
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://pagopa.github.io/aks-microservice-chart-blueprint" chart repository
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading microservice-chart from repo https://pagopa.github.io/aks-microservice-chart-blueprint
Deleting outdated charts
❯ cat ./Chart.lock | grep -A 2 microservice-chart
- name: microservice-chart
  repository: https://pagopa.github.io/aks-microservice-chart-blueprint
  version: 8.0.2
digest: sha256:5ba3e752f711c1065e0e2eaacb1a94110f5def30971c19ef0759ac6b77de1d88

VERY IMPORTANT
The scripts needs to be adapted for every service and launched on each environment before actually upgrading through helm upgrade (ref. https://github.com/pagopa/aks-microservice-chart-blueprint/blob/main/scripts/migrate_from_v7_to_v8.sh)

❯ ../migrate_from_v7_to_v8.sh pagopaapiconfig-postgresql pagopaapiconfig apiconfig
scaledobject.keda.sh/pagopaapiconfig-postgresql not labeled
scaledobject.keda.sh/pagopaapiconfig-postgresql not labeled
scaledobject.keda.sh/pagopaapiconfig-postgresql annotated
scaledobject.keda.sh/pagopaapiconfig-postgresql annotated
❯ kubectl get scaledobject pagopaapiconfig-postgresql -n apiconfig -o json | jq '{
    labels: {
      "app.kubernetes.io/instance": .metadata.labels."app.kubernetes.io/instance",
      "app.kubernetes.io/managed-by": .metadata.labels."app.kubernetes.io/managed-by"
    },
    annotations: {
      "meta.helm.sh/release-name": .metadata.annotations."meta.helm.sh/release-name",
      "meta.helm.sh/release-namespace": .metadata.annotations."meta.helm.sh/release-namespace"
    }
  }'
{
  "labels": {
    "app.kubernetes.io/instance": "pagopaapiconfig",
    "app.kubernetes.io/managed-by": "Helm"
  },
  "annotations": {
    "meta.helm.sh/release-name": "pagopaapiconfig",
    "meta.helm.sh/release-namespace": "apiconfig"
  }
}
❯ helm upgrade --namespace apiconfig \
    --install --values ./values-dev.yaml \
    --set postgresql.azure.workloadIdentityClientId=xxxxx \
    --set postgresql.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --wait --timeout 15m0s \
    pagopaapiconfig .
Release "pagopaapiconfig" has been upgraded. Happy Helming!
NAME: pagopaapiconfig
LAST DEPLOYED: Wed Oct 15 09:42:01 2025
NAMESPACE: apiconfig
STATUS: deployed
REVISION: 11
TEST SUITE: None
❯ kubectl get scaledobject pagopaapiconfig-postgresql -n apiconfig -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
7.5.0%

NOTE: still on 7.5.0 so we delete the scaled object and redeploy

❯ kubectl delete scaledobject pagopaapiconfig-postgresql -n apiconfig
scaledobject.keda.sh "pagopaapiconfig-postgresql" deleted from apiconfig namespace
❯ helm upgrade --namespace apiconfig \
    --install --values ./values-dev.yaml \
    --set postgresql.azure.workloadIdentityClientId=f0cbdf46-9ee9-4ac8-a908-9f25502c91c9 \
    --set postgresql.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --wait --timeout 15m0s \
    pagopaapiconfig .
Release "pagopaapiconfig" has been upgraded. Happy Helming!
NAME: pagopaapiconfig
LAST DEPLOYED: Wed Oct 15 09:48:31 2025
NAMESPACE: apiconfig
STATUS: deployed
REVISION: 12
TEST SUITE: None
❯ kubectl get scaledobject pagopaapiconfig-postgresql -n apiconfig -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
8.0.2%

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@dylantangredi-jakala dylantangredi-jakala added the ignore-for-release Ignore pull request from the release label Oct 15, 2025
@dylantangredi-jakala dylantangredi-jakala marked this pull request as draft October 15, 2025 07:58
@dylantangredi-jakala dylantangredi-jakala marked this pull request as ready for review October 17, 2025 08:14
Copy link

@dylantangredi-jakala dylantangredi-jakala added patch and removed ignore-for-release Ignore pull request from the release skip labels Oct 17, 2025
Copy link
Contributor

@FedericoRuzzier FedericoRuzzier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@FedericoRuzzier FedericoRuzzier merged commit cba6c11 into main Oct 17, 2025
22 of 24 checks passed
@FedericoRuzzier FedericoRuzzier deleted the PIDM-504-set-min-pod-replicas branch October 17, 2025 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants