feat: [PIDM-504] update deploy config #32

dylantangredi-jakala · 2025-10-07T07:56:15Z

List of Changes

upgraded helm chart to v8.0.2
added rolling update strategy (maxUnavailable, maxSurge)
overridden deployment replicas to match HPA minReplica config

NOTE: upgraded in

DEV -> OK
UAT -> OK
PROD -> OK

Motivation and Context

This upgrade was required to address an issue with the rollout strategy and the autoscaler behaviour, where the pods would scale down to 1 during deploy instead of keeping the same number of pods and doing +1 -1 to avoid traffic congestion. Also replicas are set to match the autoscaling config during deploy, to avoid starting the service with 1 pod when there is relevant traffic on the service.
This also ensures zero-downtime deployments by keeping all existing pods running until new versions are healthy and ready, then replacing them one at a time.

How Has This Been Tested?

❯ helm dependency update
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://pagopa.github.io/aks-microservice-chart-blueprint" chart repository
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 3 charts
Downloading microservice-chart from repo https://pagopa.github.io/aks-microservice-chart-blueprint
Already downloaded microservice-chart from repo https://pagopa.github.io/aks-microservice-chart-blueprint
Already downloaded microservice-chart from repo https://pagopa.github.io/aks-microservice-chart-blueprint
Deleting outdated charts

VERY IMPORTANT
The scripts needs to be adapted for every service and launched on each environment before actually upgrading through helm upgrade (ref. https://github.com/pagopa/aks-microservice-chart-blueprint/blob/main/scripts/migrate_from_v7_to_v8.sh)

❯ ../../migrate_from_v7_to_v8.sh pagopafdrtoeventhub-fdr1-blobtrigger pagopafdrtoeventhub fdr
scaledobject.keda.sh/pagopafdrtoeventhub-fdr1-blobtrigger not labeled
scaledobject.keda.sh/pagopafdrtoeventhub-fdr1-blobtrigger not labeled
scaledobject.keda.sh/pagopafdrtoeventhub-fdr1-blobtrigger annotated
scaledobject.keda.sh/pagopafdrtoeventhub-fdr1-blobtrigger annotated

❯ ../../migrate_from_v7_to_v8.sh pagopafdrtoeventhub-fdr3-blobtrigger pagopafdrtoeventhub fdr
scaledobject.keda.sh/pagopafdrtoeventhub-fdr3-blobtrigger not labeled
scaledobject.keda.sh/pagopafdrtoeventhub-fdr3-blobtrigger not labeled
scaledobject.keda.sh/pagopafdrtoeventhub-fdr3-blobtrigger annotated
scaledobject.keda.sh/pagopafdrtoeventhub-fdr3-blobtrigger annotated

❯ ../../migrate_from_v7_to_v8.sh pagopafdrtoeventhub-recovery pagopafdrtoeventhub fdr
scaledobject.keda.sh/pagopafdrtoeventhub-recovery not labeled
scaledobject.keda.sh/pagopafdrtoeventhub-recovery not labeled
scaledobject.keda.sh/pagopafdrtoeventhub-recovery annotated
scaledobject.keda.sh/pagopafdrtoeventhub-recovery annotated

❯ helm upgrade --namespace fdr \
    --install --values ./helm/values-dev.yaml \
    --set fdr1-blobtrigger.azure.workloadIdentityClientId=xxxxx \
    --set fdr3-blobtrigger.azure.workloadIdentityClientId=xxxxx \
    --set recovery.azure.workloadIdentityClientId=xxxxx \
    --set fdr1-blobtrigger.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --set fdr3-blobtrigger.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --set recovery.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --wait --timeout 15m0s \
    pagopafdrtoeventhub ./helm
Release "pagopafdrtoeventhub" has been upgraded. Happy Helming!
NAME: pagopafdrtoeventhub
LAST DEPLOYED: Thu Oct 16 10:11:28 2025
NAMESPACE: fdr
STATUS: deployed
REVISION: 27
TEST SUITE: None

❯  kubectl get scaledobject pagopafdrtoeventhub-fdr1-blobtrigger -n fdr -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
7.5.0%
❯ kubectl get scaledobject pagopafdrtoeventhub-fdr3-blobtrigger -n fdr -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
7.5.0%
❯  kubectl get scaledobject pagopafdrtoeventhub-recovery -n fdr -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
7.5.0%

NOTE: still on 7.5.0 so we delete the scaled objects and redeploy

❯ kubectl delete scaledobject pagopafdrtoeventhub-fdr1-blobtrigger -n fdr
scaledobject.keda.sh "pagopafdrtoeventhub-fdr1-blobtrigger" deleted from fdr namespace
❯ kubectl delete scaledobject pagopafdrtoeventhub-fdr3-blobtrigger -n fdr
scaledobject.keda.sh "pagopafdrtoeventhub-fdr3-blobtrigger" deleted from fdr namespace
❯ kubectl delete scaledobject pagopafdrtoeventhub-recovery -n fdr
scaledobject.keda.sh "pagopafdrtoeventhub-recovery" deleted from fdr namespace

❯ helm upgrade --namespace fdr \
    --install --values ./helm/values-dev.yaml \
    --set fdr1-blobtrigger.azure.workloadIdentityClientId=xxxxx \
    --set fdr3-blobtrigger.azure.workloadIdentityClientId=xxxxx \
    --set recovery.azure.workloadIdentityClientId=xxxxx \
    --set fdr1-blobtrigger.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --set fdr3-blobtrigger.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --set recovery.podAnnotations.force-rollout="rollout-$(date +%s)" \
    --wait --timeout 15m0s \
    pagopafdrtoeventhub ./helm
Release "pagopafdrtoeventhub" has been upgraded. Happy Helming!
NAME: pagopafdrtoeventhub
LAST DEPLOYED: Thu Oct 16 10:17:53 2025
NAMESPACE: fdr
STATUS: deployed
REVISION: 28
TEST SUITE: None

❯   kubectl get scaledobject pagopafdrtoeventhub-fdr1-blobtrigger -n fdr -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
8.0.2%
❯   kubectl get scaledobject pagopafdrtoeventhub-fdr3-blobtrigger -n fdr -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
8.0.2%
❯   kubectl get scaledobject pagopafdrtoeventhub-recovery -n fdr -o jsonpath='{.metadata.labels.helm\.sh/blueprint-version}'
8.0.2%

❯   kubectl get pods -n fdr -l app.kubernetes.io/instance=pagopafdrtoeventhub
NAME                                                    READY   STATUS    RESTARTS   AGE
pagopafdrtoeventhub-fdr1-blobtrigger-7585c8f9ff-s4zx4   1/1     Running   0          4m14s
pagopafdrtoeventhub-fdr3-blobtrigger-5cc88c898b-7rrvt   1/1     Running   0          4m14s
pagopafdrtoeventhub-recovery-694d9f9dd7-kkvrv           1/1     Running   0          4m14s

❯   kubectl get pods,hpa,scaledobject -n fdr -l app.kubernetes.io/instance=pagopafdrtoeventhub
NAME                                                        READY   STATUS    RESTARTS   AGE
pod/pagopafdrtoeventhub-fdr1-blobtrigger-7585c8f9ff-s4zx4   1/1     Running   0          4m21s
pod/pagopafdrtoeventhub-fdr3-blobtrigger-5cc88c898b-7rrvt   1/1     Running   0          4m21s
pod/pagopafdrtoeventhub-recovery-694d9f9dd7-kkvrv           1/1     Running   0          4m21s

NAME                                                                                REFERENCE                                         TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/keda-hpa-pagopafdrtoeventhub-fdr1-blobtrigger   Deployment/pagopafdrtoeventhub-fdr1-blobtrigger   cpu: 5%/75%   1         1         1          4m20s
horizontalpodautoscaler.autoscaling/keda-hpa-pagopafdrtoeventhub-fdr3-blobtrigger   Deployment/pagopafdrtoeventhub-fdr3-blobtrigger   cpu: 7%/75%   1         1         1          4m20s
horizontalpodautoscaler.autoscaling/keda-hpa-pagopafdrtoeventhub-recovery           Deployment/pagopafdrtoeventhub-recovery           cpu: 3%/75%   1         1         1          4m20s

NAME                                                        SCALETARGETKIND      SCALETARGETNAME                        MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED    TRIGGERS   AUTHENTICATIONS                        AGE
scaledobject.keda.sh/pagopafdrtoeventhub-fdr1-blobtrigger   apps/v1.Deployment   pagopafdrtoeventhub-fdr1-blobtrigger   1     1     True    True     False      Unknown   cpu        pagopafdrtoeventhub-fdr1-blobtrigger   4m20s
scaledobject.keda.sh/pagopafdrtoeventhub-fdr3-blobtrigger   apps/v1.Deployment   pagopafdrtoeventhub-fdr3-blobtrigger   1     1     True    True     False      Unknown   cpu        pagopafdrtoeventhub-fdr3-blobtrigger   4m20s
scaledobject.keda.sh/pagopafdrtoeventhub-recovery           apps/v1.Deployment   pagopafdrtoeventhub-recovery           1     1     True    True     False      Unknown   cpu        pagopafdrtoeventhub-recovery           4m20s

❯   kubectl get scaledobject pagopafdrtoeventhub-fdr1-blobtrigger -n fdr -o jsonpath='{.metadata.labels}' | jq
{
  "app.kubernetes.io/instance": "pagopafdrtoeventhub",
  "app.kubernetes.io/managed-by": "Helm",
  "app.kubernetes.io/name": "fdr1-blobtrigger",
  "app.kubernetes.io/version": "1.0.15",
  "azure.workload.identity/use": "true",
  "canaryDelivery": "false",
  "helm.sh/blueprint-version": "8.0.2",
  "helm.sh/chart": "fdr1-blobtrigger-1.0.15",
  "scaledobject.keda.sh/name": "pagopafdrtoeventhub-fdr1-blobtrigger"
}

❯   kubectl get scaledobject pagopafdrtoeventhub-fdr1-blobtrigger -n fdr -o jsonpath='{.metadata.annotations}' | jq
{
  "meta.helm.sh/release-name": "pagopafdrtoeventhub",
  "meta.helm.sh/release-namespace": "fdr"
}

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

sonarqubecloud · 2025-10-16T07:55:41Z

Quality Gate failed

Failed conditions
3 Security Hotspots

See analysis details on SonarQube Cloud

FedericoRuzzier

LGTM 👍

feat: [PIDM-504] update deploy config

40bc8d4

dylantangredi-jakala self-assigned this Oct 7, 2025

dylantangredi-jakala requested a review from a team as a code owner October 7, 2025 07:56

dylantangredi-jakala added the skip label Oct 7, 2025

github-actions bot added size/small and removed size/small labels Oct 7, 2025

feat: [PIDM-504] upgrade helm charts and move deploy strategy sections

3304b1b

github-actions bot added size/small and removed size/small labels Oct 16, 2025

dylantangredi-jakala marked this pull request as draft October 16, 2025 08:25

dylantangredi-jakala added the ignore-for-release label Oct 16, 2025

github-actions bot added size/small and removed size/small labels Oct 16, 2025

dylantangredi-jakala marked this pull request as ready for review October 17, 2025 07:40

FedericoRuzzier approved these changes Oct 17, 2025

View reviewed changes

FedericoRuzzier merged commit f9a35a5 into main Oct 17, 2025
24 of 25 checks passed

FedericoRuzzier deleted the PIDM-504-upd-deploy-config branch October 17, 2025 07:45

FedericoRuzzier temporarily deployed to dev October 17, 2025 07:46 — with GitHub Actions Inactive

dylantangredi-jakala added patch and removed skip ignore-for-release patch labels Oct 17, 2025

dylantangredi-jakala added skip ignore-for-release labels Oct 17, 2025

github-actions bot added size/small and removed size/small labels Oct 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: [PIDM-504] update deploy config #32

feat: [PIDM-504] update deploy config #32

Uh oh!

dylantangredi-jakala commented Oct 7, 2025 •

edited

Loading

Uh oh!

sonarqubecloud bot commented Oct 16, 2025

Uh oh!

FedericoRuzzier left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: [PIDM-504] update deploy config #32

feat: [PIDM-504] update deploy config #32

Uh oh!

Conversation

dylantangredi-jakala commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

sonarqubecloud bot commented Oct 16, 2025

Quality Gate failed

Uh oh!

FedericoRuzzier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dylantangredi-jakala commented Oct 7, 2025 •

edited

Loading