Use VM with multiple network interfaces

[yellowhat]

Hi,

I am trying to use talos on an OVH VM with 2 network interfaces (public and private):

terraform VM

resource "openstack_compute_instance_v2" "main" {
  name            = "node-${count.index + 1}"
  ...

  network {
    name = "Ext-Net"
  }

  network {
    name = openstack_networking_network_v2.main.name
  }
}

I am using the talos terraform provider to configure the instance:

Talos

resource "talos_machine_secrets" "main" {}

data "talos_client_configuration" "main" {
  cluster_name         = "my-k8s"
  client_configuration = talos_machine_secrets.main.client_configuration
  nodes                = openstack_compute_instance_v2.main[*].access_ip_v4
  endpoints            = openstack_compute_instance_v2.main[*].access_ip_v4
}

data "talos_machine_configuration" "controlplane" {
  cluster_name       = "my-k8s"
  machine_type       = "controlplane"
  cluster_endpoint   = "https://${openstack_compute_instance_v2.main[0].network[1].fixed_ip_v4}:6443"
  talos_version      = "1.11.6"
  kubernetes_version = "1.34.3"
  machine_secrets    = talos_machine_secrets.main.machine_secrets
}

resource "talos_machine_configuration_apply" "controlplane" {
  client_configuration        = talos_machine_secrets.main.client_configuration
  machine_configuration_input = data.talos_machine_configuration.controlplane.machine_configuration
  node                        = openstack_compute_instance_v2.main[0].access_ip_v4
  config_patches = [
    yamlencode({
      machine = {
        install = {
          disk = "/dev/sda"
          wipe = true
        }
        network = {
          hostname = openstack_compute_instance_v2.main[0].name
          interfaces = [
            {
              interface = "ens3"
              dhcp = true
            },
            {
              interface = "ens4"
              dhcp      = false
              addresses = ["${openstack_compute_instance_v2.main[0].network[1].fixed_ip_v4}/24"]
            }
          ]
          nameservers = ["1.1.1.1", "9.9.9.9"]
        }
        kubelet = {
          nodeIP = {
            validSubnets = [openstack_networking_subnet_v2.main.cidr]
          }
        }
      }
    })
  ]
}

On applying the config I struggle as ens4 (private interface):

• via DHCP it also returns DNS nameservers but they cannot resolve as this is only for private traffic
• therefore I need to disable DHCP and use static IP

still it is the default gateway therefore it cannot reach the internet.

I managed to make it work by:

            {
              interface = "ens3"
              dhcp      = false
              addresses = ["${openstack_compute_instance_v2.main[0].network[0].fixed_ip_v4}/32"]
              routes = [
                {
                  # Make gateway reachable (on-link) for /32 address
                  network = "135.125.216.1/32"
                },
                {
                  # Default route via gateway
                  network = "0.0.0.0/0"
                  gateway = "135.125.216.1"
                  metric  = 10
                }
              ]
            },

If I understand it correctly I am forcing the ens3 interface to have higher priority.

But it is a bit challenging to get the public gateway via terraform, I have to hardcode it.

Any suggestion?

Thanks

[smira]

I know nothing about OVH to be specific, but machine config DNS settings should override DHCP DNS, so you can keep DHCP on.

If both interfaces announce default gateway, you can per-DHCP metric to control the priority.

[yellowhat]

I was not able to found the DHCP metric.

Could you point me in that direction?

[smira]

Look for dhcpOptions in the doc here https://docs.siderolabs.com/talos/v1.11/reference/configuration/v1alpha1/config#dhcpoptions

[yellowhat]

I saw it, but then I forgot about it, anyway it works:

          interfaces = [
            {
              interface = "ens3"
              dhcp = true
              dhcpOptions = {
                 routeMetric = 1024
              }
            },
            {
              interface = "ens4"
              dhcp = true
              dhcpOptions = {
                 routeMetric = 2048
              }
            }
          ]

Thank you very much