Add option to disable entry in main routing table if `host_table` is set for routed NIC

[ibot3]

For routed NICs, the host_table parameter can be specified to create a route in the specified routing table.
However, the same route is always installed in the main routing table too:

incus/internal/server/device/nic_routed.go

Lines 410 to 416 in fc4d0ca

	// Apply host-side static routes to main routing table.
	r := ip.Route{
	DevName: saveData["host_name"],
	Route: fmt.Sprintf("%s/%d", addrStr, subnetSize),
	Table: "main",
	Family: ipFamilyArg,
	}

It would be great if this behavior could be changed by another config option so that the route is only present in the specified routing table.

Our scenario:
We want to have VMs in different VRFs on the incus host.
But for routed NICs to work, ip_forwarding must be enabled on the host.
This causes that from the main VRF, all VMs with routed NICs can be reached, what we don't want, because they should only be reachable within their VRF.

[stgraber]

Hmm, I don't recall how we settled on this behavior for host_table...

In hindsight, we'd have been better served by a host_tables config key taking a comma separated list of table names and not having that default behavior of always putting in the main table too.

I think the best path forward here is to implement ipv4.host_tables and ipv6.host_tables defined as a comma separated list of table IDs and whose default is 254. Making them conflict with ipv4.host_table and ipv6.host_table respectively and then mark the old keys as deprecated in documentation.

[AbhinavTiruvee]

@stgraber Hi, I'm part of a group at UT Austin taking a class on virtualization. Could my partner and I take a look at this issue and try to solve it?

[stgraber]

Assigned it to you. I'll be able to assign your partner once they also comment in here.

[stgraber]

So for this one, we'll want to implement the behavior I described in the comment above (#1559 (comment))

[avipatel805]

Hello, I am the partner mentioned above. Could you please add me to the issue as well?

[avipatel805]

Hello @stgraber ,

We wanted to check if our current approach for implementing host_tables support looks good before moving on to testing and documentation. Here's what we've done so far:

1. Field Addition and Validation

• Added ipv4.host_tables and ipv6.host_tables to optional_fields.

• Added a validator to ensure list items are within the 0–255 range.

2. Table Retrieval Logic

• Created a get_tables function:

• If host_tables is specified, it returns those values.

• If the (now deprecated) host_table is specified, it returns that value.
  → Should we also return the default value 254 in this case, or only what's provided?

• If neither is provided, it defaults to 254.

3. Conditional Routing

• Ensured that routes are only installed into the user-specified tables.

4. Route Installation

• Loop through the specified tables and install routes in the appropriate places.

Here’s the current code for reference:
AbhinavTiruvee@5754613

We’d appreciate any feedback on whether we’re headed in the right direction. Also, we’re a bit unsure how to approach testing—there don't seem to be any existing test cases for routed NICs. Should we create new ones, or are there some we might be missing?

Thanks!