Skip to content

Header Route Is Being Removed During Canary Deployment #4486

@pravikmr

Description

@pravikmr

Checklist:

  • I've included steps to reproduce the bug.
  • I've included the version of argo rollouts.

Describe the bug

We are using ALB ingress controller with SetHeaderRoute configuration. with canary deployment strategy

we are running Rollout v1.8.3 and AWS load balancer controller v1.12.0

In the previous version of rollout(1.7.2) when we configure the SetheaderRoute in canary steps it would create the listener rule on the ALB level..so we can reach the preview/canary pod using the Header value to run tests and this Header rule would be available through out the canary deployment.

But in the recent version of the rollout is seems to ignores this SetHeaderroute configuration.

This is our config:

strategy:
    canary:
      canaryService: starfish-new
      dynamicStableScale: true
      stableService: starfish
      steps:
        - setHeaderRoute:
            match:
              - headerName: x-sm-preview
                headerValue:
                  exact: starfish
            name: canary-header-route
        - setCanaryScale:
            replicas: 1
        - pause: {}
        - setCanaryScale:
            matchTrafficWeight: true
        - setWeight: 10
        - pause:
            duration: 15s
        - setWeight: 30
        - pause:
            duration: 15s
        - setWeight: 60
        - pause:
            duration: 15s
        - setWeight: 100

And initially it does set the canary route and it create the ALB listener rule on the ALB level

time="2025-10-09T12:43:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"cloud-solutions-starfish\", Name:\"starfish\", UID:\"110dab0a-b3f7-4bc9-812c-034ddb9dec00\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"763214433\", FieldPath:\"\"}): type: 'Normal' reason: 'TrafficWeightUpdated' Traffic weight updated

time="2025-10-09T12:43:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"cloud-solutions-starfish\", Name:\"starfish\", UID:\"110dab0a-b3f7-4bc9-812c-034ddb9dec00\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"763214433\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 1/11 completed"

But right after sometime that Header listener rule on the ALB is getting removed.

time="2025-07-15T12:43:57Z" level=info msg="delaying service switch from 5958ccb5b7 to 7d567fff75: ReplicaSet has zero availability" namespace=cloud-solutions-starfish rollout=starfish service=starfish-new

time="2025-10-09T12:43:57Z" level=info msg="Found 1 TrafficRouting Reconcilers" namespace=cloud-solutions-starfish rollout=starfish

time="2025-10-09T12:43:57Z" level=info msg="Reconciling TrafficRouting with type 'ALB'" namespace=cloud-solutions-starfish rollout=starfish

time="2025-10-09T12:43:57Z" level=info msg="Updating Ingress `cloud-solutions-sandbox-starfish` removing managed routes" event_reason=PatchingALBIngress namespace=cloud-solutions-starfish rollout=starfish

time="2025-10-09T12:43:57Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"cloud-solutions-starfish\", Name:\"starfish\", UID:\"110dab0a-b3f7-4bc9-812c-034ddb9dec00\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"763214439\", FieldPath:\"\"}): type: 'Normal' reason: 'PatchingALBIngress' Updating Ingress `cloud-solutions-sandbox-starfish` removing managed routes"

time="2025-10-09T12:43:57Z" level=info msg="no changes to the ALB Ingress" ingress=cloud-solutions-sandbox-starfish namespace=cloud-solutions-starfish rollout=starfish 

And with that manged route/listener rule begin removed on the ALB level. we are unable to reach the canary/preview pod for further testing of our application.

This was not the case in the previous version of rollout. in the previous version once the Header route/ Listener rule is set on the ALB level by rollout controller that listener rule would available during the entire canary deployment

To Reproduce

  1. Define Canary Rollout using SetHeaderRoute configuration.
  2. when it reaches the setHeaderRoute step listener rule gets added on the ALB. which will help us reaching canary pod using the Header value.
  3. Wait for few seconds and we will see rollout will remove managed route which it previously added.

Expected behavior

Once the HeaderRoute listener rule is configured on the ALB that Header listener rule should be available through out the canary deployment. So we can run/reach the canary pods during the canary deployment when we gradually shift traffic from old version to new version.

Screenshots

Image

Version

  • Rollouts : v1.8.3
  • AWS load balancer controller: 1.12.0

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions