Regression when using ExternalName services

[philpep]

After upgrade from chart 4.11.3 to 4.12.1 my ingresses using ExternalName services aren't working anymore (http 503 Service Temporarily Unavailable).

Logs of the controller shows:

2025/03/25 10:22:17 [error] 26#26: *33847 lua entry thread aborted: runtime error: /etc/nginx/lua/balancer.lua:78: bad argument #1 to 'ipairs' (table expected, got nil)
stack traceback:
coroutine 0:
        [C]: in function 'ipairs'
        /etc/nginx/lua/balancer.lua:78: in function 'resolve_external_names'
        /etc/nginx/lua/balancer.lua:114: in function 'sync_backend'
        /etc/nginx/lua/balancer.lua:148: in function </etc/nginx/lua/balancer.lua:146>, context: ngx.timer

Example:

apiVersion: v1
kind: Service
metadata:
  name: example
spec:
  type: ExternalName
  externalName: internal.example.com
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example.com
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: example
                port:
                  number: 443
  tls:
    - hosts:
      - example.com

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[philpep]

It might relate to #13076 but seems a different issue.

[philpep]

Seems the bug was introduced in 4.11.5 release (4.11.4 is OK).

[philpep]

I think issue is from this commit c6c5b48

[tolix1]

same issue here. 4.11.5 and 4.12.1 impacted

[Gacko]

Please do not override the issue template and instead fill it as requested. This is important for reproducing your issue.

Also please add information about what internal.example.com is pointing at. Are these IP addresses? Is it a CNAME?

[philpep]

Please do not override the issue template and instead fill it as requested. This is important for reproducing your issue.

Sorry will do better next time.

Also please add information about what internal.example.com is pointing at. Are these IP addresses? Is it a CNAME?

Yes it's a resolvable CNAME.

[sepich]

Another possibly related issue is that nginx.ingress.kubernetes.io/default-backend stopped working for externalName Ingresses.

Here is example yamls

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - image: inanimate/echo-server
          name: echo
      enableServiceLinks: false
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: google.com
    nginx.ingress.kubernetes.io/custom-http-errors: "301"
    nginx.ingress.kubernetes.io/default-backend: echo
    prometheus.io/probe: "false"
  name: test
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: ext
      port:
        number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: ext
spec:
  type: ExternalName
  externalName: google.com

So in example we trying to use custom errors:
https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/
In this case we access google.com via http/80 and get redirect to https:

$ curl -si google.com | head -1
HTTP/1.1 301 Moved Permanently

Then we try to forward this 301 to be served from out service echo.
It works in v1.11.4:

127.0.0.1 - - [26/Mar/2025:15:22:12 +0000] "GET / HTTP/1.1" 200 1774 "-" "curl/8.7.1" 67 0.033 [custom-default-backend-default-echo] [] 142.250.186.174:80 : 10.244.0.8:8080 0 : 1774 0.030 : 0.003 301 : 200 0cc21efb7ceba0b69745bc9afc359738

but broken in v1.11.5:

127.0.0.1 - - [26/Mar/2025:15:15:00 +0000] "GET / HTTP/1.1" 502 150 "-" "curl/8.7.1" 67 0.036 [custom-default-backend-default-echo] [] 216.58.206.78:80 : 0.0.0.1:80 0 : 0 0.035 : 0.001 301 : 502 6958aeaac151beea2b82922fd8da30fd
2025/03/26 15:15:00 [error] 38#38: *24724 connect() failed (113: Host is unreachable) while connecting to upstream, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "test"
2025/03/26 15:15:00 [warn] 38#38: *24724 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "test"

[Gacko]

Yes it's a resolvable CNAME.

Out of curiosity: Can you make it an A / AAAA record? Just wanna see if it's related to the record type.

[philpep]

Yes it's a resolvable CNAME.

Out of curiosity: Can you make it an A / AAAA record? Just wanna see if it's related to the record type.

In my case it's a A record (without AAAA) resolving into a RFC1918 reserved ip address, outside of the k8s cluster (e.g. 192.168.42.12)

[vasili439]

Yes it's a resolvable CNAME.

Out of curiosity: Can you make it an A / AAAA record? Just wanna see if it's related to the record type.

In my case it's a A record (without AAAA) resolving into a RFC1918 reserved ip address, outside of the k8s cluster (e.g. 192.168.42.12)

in my case the same: I've tried as a quick fix to replace RFC1918 IP with temp domain name (A record) but no luck. HTTP503 with plain IP address and with DNS A record.