handle cross zone disabled for alb

[shuqz]

Issue

#4477

Description

For ALB, if cross_zone=disabled and AZ is either not specified or all, we get error from api

ValidationError: When cross-zone load balancing is disabled on this target group, you must specify an Availability Zone for IP target '172.16.0.108' since it is outside of the VPC

it only succeed when az is specified. this does not apply to NLB
Fix implemented:

• introduced a cache: needsPodAZCache tracks TGBs that require pod availability zones during target registration.
• catch error from registerTarget, if it is validationError, try register with pod AZ, if still fails, fail with error

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
  created peered VPC, trying to register pod from VPC 1 to target group in VPC2. Before fix, no target is registered. After fix, target is registered. tested with yaml (load_balancing.cross_zone.enabled=false)

---
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 5
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
        - image: public.ecr.aws/l6m2t8p7/docker-2048:latest
          imagePullPolicy: Always
          name: app-2048
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: game-2048
  name: ingress-2048
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/target-group-attributes: load_balancing.cross_zone.enabled=false
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-2048
                port:
                  number: 80
---
apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: peered-tgb
  namespace: game-2048
spec:
  serviceRef:
    name: service-2048
    port: 80
  targetGroupARN: arn:aws:elasticloadbalancing:us-west-2::targetgroup/test-tg/
  vpcID: vpc- # The peered VPC ID (different from cluster VPC)

• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: shuqz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [shuqz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[zac-nixon]

This might not always work. Customers aren't required to add ingress annotations to their Services.

It's more reliable to use the TG protocol: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/apis/elbv2/v1beta1/targetgroupbinding_types.go#L143-L146

[zac-nixon]

I realized that this only works for the Ingress API. We will want a solution that works ALB Gateways too.

[shuqz]

i thought about TG protocol too, but did not realize they are not required to add annotation.
yea i forgot gateway api integrations, i will include it

[zac-nixon]

If the protocol isn't added by user, then we infer and inject it.
https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/webhooks/elbv2/targetgroupbinding_mutator.go#L82-L84

[zac-nixon]

For Ingress objects, this could go on the Ingress or Service, meaning that this could be incorrect

[zac-nixon]

See my other comment, this will never work for ALB Gateways.

[shuqz]

the other more striaghtforward way is to use describe api get cross-zone value.

[zac-nixon]

This will block reconciles on non-EKS clusters. I suspect that's not what we want.

[shuqz]

i can remove this fallback part

[zac-nixon]

What do you think about using a node-az cache? That way we don't have to recalculate the node AZ each reconcile loop iteration.

[shuqz]

i am not sure if they will change error message but checking ValidationError alone might not enough