PR #23959 breaks consul ipv6 registration address

[42wim]

Nomad version

Nomad v1.9.7
BuildDate 2025-03-11T09:07:15Z
Revision f869597+CHANGES

Operating system and Environment details

Linux 4.18.0-553.46.1.el8_10.x86_64

Issue

After an upgrade from 1.7.7 to 1.9.7 we got an issue with our CNI containers and consul address registration.

#23959 changes the content of netstatus.Address, in our setup this was an IPv6 address inside the CNI bridge and it got registered correctly in consul.

Now this is an IPv4 address as the PR has split it into a netstatus.Address and netstatus.AddressIPv6.

And GetAddress with AddressModeAlloc now returns the ipv4 address

nomad/client/serviceregistration/address.go

Line 150 in 45dab2d

return netStatus.Address, 0, nil

Maybe there should be an option to specify to only use the IPv6 Address like with the docker driver

[jrasell]

Hi @42wim and thanks for raising this issue and the linked PR. I'll get this and the PR onto our backlog for review.

[jrasell]

Hi @42wim would it be possible to share the configuration you are using to generate the issue you're seeing (CNI plugins, client relevant config)? We had a quick internal chat about this and there was a question of whether client.preferred_address_family= ipv6 would achieve what you're looking for?

[42wim]

@jrasell I'll share the CNI config below.

Basically the container gets an ipv4 and ipv6 address.

2: eth0@if116: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 6a:89:e1:82:57:fa brd ff:ff:ff:ff:ff:ff
    inet 172.17.242.1/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 1800:2d30:0:51:144:b:0:1/80 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::6779:e1ff:ee82:58ba/64 scope link
       valid_lft forever preferred_lft forever

This hasn't changed, but instead of registering 1800:2d30:0:51:144:b:0:1 like before, the PR #23959 change now registers 172.17.242.1

The original code always returned the first CNI address (which always happened to be ipv6)

nomad/client/allocrunner/networking_cni.go

Lines 406 to 410 in 2f5cf8e

	if iface.Sandbox != "" && len(iface.IPConfigs) > 0 {
	netStatus.Address = iface.IPConfigs[0].IP.String()
	netStatus.InterfaceName = name
	break
	}

The changed code now loops all addresses and specifically returns the IPv4 address in netStatus.Address and the IPv6 address (if it exists) in netStatus.AddressIPv6

nomad/client/allocrunner/networking_cni.go

Lines 473 to 481 in 21fd0bb

	for _, ipConfig := range iface.IPConfigs {
	isIP4 := ipConfig.IP.To4() != nil
	if netStatus.Address == "" && isIP4 {
	netStatus.Address = ipConfig.IP.String()
	}
	if netStatus.AddressIPv6 == "" && !isIP4 {
	netStatus.AddressIPv6 = ipConfig.IP.String()
	}
	}

And the consul registration only looks at netstatus.Address for registration (which is now ipv4 only)

nomad/client/serviceregistration/address.go

Lines 154 to 159 in 21fd0bb

	if port, ok := ports.Get(portLabel); ok {
	// Use port.To value unless not set
	if port.To > 0 {
	return netStatus.Address, port.To, nil
	}
	return netStatus.Address, port.Value, nil

Client.preferred_address_family hasn't got any impact on this.

Here is my CNI config

{
  "cniVersion": "0.4.0",
  "name": "lnx-traefik",
  "plugins": [ {
  "type": "bridge",
  "bridge": "docker0",
  "isDefaultGateway": true,
  "ipMasq": false,
  "ipam": {
    "type": "static",
    "addresses": [
      {
        "address": "1800:2d30:0:51:144:b::1/80",
        "gateway": "1800:2d30:0:51:144::1"
      },
      {
        "address": "172.17.242.1/16",
        "gateway": "172.17.0.1"
      }
    ],
    "dns": {
      "nameservers": [
        "1800:2d30:0:23::a:1782",
        "10.11.66.1"
      ]
    },
    "routes": [
      {
        "dst": "::/0",
        "gw": "1800:2d30:0:51:144::1"
      },
      {
        "dst": "0.0.0.0/0",
        "gw": "172.17.0.1"
      }
   ]
  },
  "dns": {
    "nameservers": [
      "1800:2d30:0:23::a:1782",
      "10.11.66.1"
    ]
  }
  }]
}