Announce individual instance IP addresses over BGP

[max107]

Is there an existing issue for this?

• There is no existing issue for this feature

What are you currently unable to do

In a small home-cloud with mikrotik as bgp peer all incus servers promote same /24 subnet via bgp. Possible to announce /32 (every vm ip)?

What do you think would need to be added

to announce /32 (every vm ip)

[max107]

Similar question https://discuss.linuxcontainers.org/t/how-can-i-use-bgp-with-incus-cluster/20917

[stgraber]

Closing as not an issue, will answer on forum.

[max107]

Can you reopen and investigate please? 😅

[max107]

@stgraber https://discuss.linuxcontainers.org/t/how-can-i-use-bgp-with-incus-cluster/20917/22 thread continue

An error occurred: We appreciate your enthusiasm, keep it up! That said, for the safety of our community, you’ve reached the maximum number of replies a new user can create on their first day. Please wait 21 hours and you’ll be able to create more replies.

+-------+---------+-----------------------+------+-----------------+-----------+----------+
| NAME  |  STATE  |         IPV4          | IPV6 |      TYPE       | SNAPSHOTS | LOCATION |
+-------+---------+-----------------------+------+-----------------+-----------+----------+
| debug | RUNNING | 10.35.28.245 (enp5s0) |      | VIRTUAL-MACHINE | 0         | mini2    |
+-------+---------+-----------------------+------+-----------------+-----------+----------+

vm in mini2 server

ansible@mini2 ~> sudo incus network show incusbr0
config:
  bgp.peers.debug.address: 192.168.20.51
  bgp.peers.debug.asn: "64512"
  bgp.peers.mikrotik.address: 192.168.20.1
  bgp.peers.mikrotik.asn: "65540"
  ipv4.address: 10.35.28.1/24
  ipv4.nat: "false"
  ipv6.address: none
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/instances/debug
- /1.0/profiles/default
- /1.0/profiles/packer
managed: true
status: Created
locations:
- mini1
- mini3
- worker1
- mini2
- mini5
- mini4
- lb
project: default

ansible@mini2 ~> sudo incus config show --expanded debug
architecture: x86_64
config:
  cluster.evacuate: auto
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (cloud) (20251126_07:42)
  image.name: ubuntu-noble-amd64-cloud-20251126_07:42
  image.os: ubuntu
  image.release: noble
  image.serial: "20251126_07:42"
  image.variant: cloud
  limits.cpu: "2"
  limits.memory: 2GiB
  migration.stateful: "true"
  volatile.base_image: 8d3e6ff9770f2d2da831982c06967600e963ca0eb58353b2bf1f6c3b8b736b24
  volatile.cloud-init.instance-id: 517f6db5-b2df-4f1b-a90b-5f5a092822c2
  volatile.eth0.host_name: tap1367dc15
  volatile.eth0.hwaddr: 10:66:6a:25:38:bb
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: eefc17b4-cd07-4feb-942b-79c23c664df2
  volatile.uuid.generation: eefc17b4-cd07-4feb-942b-79c23c664df2
  volatile.vm.definition: pc-q35-10.1
  volatile.vm.rtc_adjustment: "-1"
  volatile.vm.rtc_offset: "0"
  volatile.vsock_id: "1410789558"
devices:
  eth0:
    ipv4.routes.external: 10.35.28.218/32
    network: incusbr0
    type: nic
  root:
    path: /
    pool: local
    size: 20GiB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

ansible@mini2 ~> sudo incus query /internal/debug/bgp
{
	"peers": [
		{
			"address": "192.168.20.1",
			"asn": 65540,
			"count": 1,
			"holdtime": 0,
			"password": ""
		},
		{
			"address": "192.168.20.51",
			"asn": 64512,
			"count": 1,
			"holdtime": 0,
			"password": ""
		}
	],
	"prefixes": [
		{
			"nexthop": "0.0.0.0",
			"owner": "network_2",
			"prefix": "10.35.28.0/24"
		},
		{
			"nexthop": "0.0.0.0",
			"owner": "instance_19_eth0",
			"prefix": "10.35.28.218/32"
		}
	],
	"server": {
		"address": "0.0.0.0:179",
		"asn": 65536,
		"router_id": "192.168.20.22",
		"running": true
	}
}

/32 prefix exists.

One last problem - is how to automate it?

[stgraber]

Okay, so that's looking good. By the way, not that the ipv4.routes.external doesn't match the instance's address so that is likely broken right now.

It should be pretty easy to script it, effectively:

• Get the instance list
• Check if the instance NIC already has ipv4.address and ipv4.routes.external set
• If not, get the instance current IPv4 address and put it in both config keys

We can't do too much more than that automatically in Incus as we're not in charge of the instance's IP address, it's automatically allocated by dnsmasq and we're not getting notified or anything when the lease is made, or changed.

If you're using tooling like Terraform/OpenTofu to create the instances, you could have that set both config keys for you as part of creation.

[max107]

Thanks @stgraber

[max107]

Okay, so that's looking good.

And yes and no.

By the way, not that the ipv4.routes.external doesn't match the instance's address so that is likely broken right now.

I know, it's just a demo / example. Correct address is not important here if we talking about bgp announces.

It should be pretty easy to script it, effectively:
Get the instance list
Check if the instance NIC already has ipv4.address and ipv4.routes.external set
If not, get the instance current IPv4 address and put it in both config keys

Nope, that ^ step is incorrect or incomplete, because im talking about full automatic process (not semi automatically).

We should manually set ipv4.routes.external, but all incus cluster members know ip assigned to vm / lxc in dnsmasq.

# incus ls
+-------+---------+----------------------+------+-----------+-----------+----------+
| NAME  |  STATE  |         IPV4         | IPV6 |   TYPE    | SNAPSHOTS | LOCATION |
+-------+---------+----------------------+------+-----------+-----------+----------+
| minio | RUNNING | 172.17.0.1 (docker0) |      | CONTAINER | 0         | worker1  |
|       |         | 10.35.28.189 (eth0)  |      |           |           |          |
+-------+---------+----------------------+------+-----------+-----------+----------+

• If we disable ip4.nat - we should set ip manually via ipv4.routes.external
• If we enable ip4.nat - incus know vm ip, but dont show it in config.

architecture: x86_64
config:
  cluster.evacuate: auto
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (cloud) (20251126_07:42)
  image.name: ubuntu-noble-amd64-cloud-20251126_07:42
  image.os: ubuntu
  image.release: noble
  image.serial: "20251126_07:42"
  image.variant: cloud
  limits.cpu: "2"
  limits.memory: 2GiB
  migration.stateful: "true"
  volatile.base_image: 8e50820aa2dd1e2229cc6525a8285b5cd61c3a94e0f45a23f27ca0982d787097
  volatile.cloud-init.instance-id: e16b1303-8844-465f-9430-6b91c3a843b5
  volatile.eth0.host_name: vethaa8b8dce
  volatile.eth0.hwaddr: 10:66:6a:40:5f:d2
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 4e782d53-e07d-4c15-bc26-61da44cc35c2
  volatile.uuid.generation: 4e782d53-e07d-4c15-bc26-61da44cc35c2
devices:
  eth0:
    network: incusbr0
    type: nic
  root:
    path: /
    pool: local
    size: 20GiB
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

But we can ask dnsmasq directly about instance ip address. Right?

cat /var/lib/incus/networks/incusbr0/dnsmasq.leases
1764315163 10:66:6a:40:5f:d2 10.35.28.189 minio ff:3b:84:34:59:00:02:00:00:ab:11:29:10:24:3a:82:b2:b3:df

and here is vm ip.

So technically incus can return vm ip and announce /32.

[max107]

Simple solution can be like you describe above. Just wait for some events in dnsmasq, set ipv4.routes.external for fully automate this process.

type Event struct {
	Name string
	IP   string
}

func parse(path string) ([]*Event, error) {
	f, err := os.OpenFile(path, os.O_RDONLY, 0)
	if err != nil {
		return nil, werr.Wrap(err)
	}

	defer f.Close()

	scanner := bufio.NewScanner(f)

	events := make([]*Event, 0, 10)

	for scanner.Scan() {
		parts := strings.Split(scanner.Text(), " ")
		events = append(events, &Event{
			Name: parts[3],
			IP:   parts[2] + "/32",
		})
	}

	return events, nil
}

func Watch(ctx context.Context, path string) (<-chan *Event, error) {
	l := log.Ctx(ctx)

	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		return nil, werr.Wrap(err)
	}

	if err := watcher.Add(path); err != nil {
		return nil, werr.Wrap(err)
	}

	ch := make(chan *Event)

	go func() {
		defer watcher.Close()
		defer close(ch)

		for {
			select {
			case <-ctx.Done():
				return
			case event, ok := <-watcher.Events:
				if !ok {
					return
				}

				if event.Op&fsnotify.Write == fsnotify.Write {
					ipList, err := parse(event.Name)
					if err != nil {
						l.Err(err).Msg("failed to parse event")
						continue
					}

					for _, ip := range ipList {
						ch <- ip
					}
				}
			case err, ok := <-watcher.Errors:
				if !ok {
					return
				}

				l.Err(err).Send()
			}
		}
	}()

	return ch, nil
}

And use something like this

func TestWatch(t *testing.T) {
	t.Parallel()

	crt, err := os.ReadFile("/Users/max/.config/incus/client.crt")
	require.NoError(t, err)
	key, err := os.ReadFile("/Users/max/.config/incus/client.key")
	require.NoError(t, err)

	// Connect to Incus over the Unix socket
	c, err := incus.ConnectIncusWithContext(t.Context(), "https://192.168.20.21:8443", &incus.ConnectionArgs{
		InsecureSkipVerify: true,
		TLSClientCert:      string(crt),
		TLSClientKey:       string(key),
	})
	require.NoError(t, err)

	ch, err := dnsmasq.Watch(t.Context(), "./testdata")
	require.NoError(t, err)

	for event := range ch {
		_, err := c.UpdateInstance(event.Name, api.InstancePut{
			Config: map[string]string{
				"ipv4.routes.external": event.IP,
			},
		}, "")
		require.NoError(t, err)
	}
}

But i think will be great if some similar solution can be a part of incus project.

[max107]

If i understand incus code correctly, we should add some logic here #2710 and this feature can be available in incus generally without any external ugly hacks.