feat(netd): report what netd implements and what it still holds

netd creates persistent TAPs, so an interface outlives the QEMU that used
it. A VMM that died between creating one and recording it leaves an
interface nothing can name again: `check` and `remove` both answer about
an identity, and the identity is exactly what was lost. Nothing could
enumerate them, so the leak was invisible until someone went looking at
`ip link`.

Add `list` and `remove_tap` to close that, and run them from the VMM at
startup next to the cleanup it already does for orphaned supervisor
processes. Listing is scoped to an instance namespace: one host runs
several VMMs against one netd, and a caller reconciling its own VMs must
not mistake another's interfaces for orphans. netd records the namespace
in the TAP's `ifalias` rather than in a state file, so a crash leaves
nothing to repair, and records only the namespace because IFALIAS_MAX is
255 bytes while two 128-byte identifiers are not -- a caller derives every
TAP name it owns anyway. Reconciliation over-approximates which TAPs are
still wanted, since keeping a leak is the cheaper mistake than deleting a
live interface out from under a running QEMU.

`remove_tap` is the one operation that names an interface instead of an
identity, because the identity is what the caller no longer has. The name
must be one `tap_name` could have produced, so it reaches nothing netd did
not create.

Add `hello` alongside it. Support has so far been detected by a field's
own absence from a response -- `queues` is documented that way -- which
works once per field, only for fields that are echoed back, and says
nothing until something has already failed. netd ships as its own package
on its own release cadence and a host may run an implementation that is
not this one, so version skew is the normal case; asking once at startup
beats inferring repeatedly. A netd that predates the operation rejects it,
and that rejection is the answer rather than an error.

Add `workdir` to both prepare operations. It is untrusted and never read
for a decision, and exists so an operator reading netd's log can get from
an opaque TAP name back to the VM that asked for it.

Author: Kevin Wang

docs/libvirt-network-filter.md

@@ -101,9 +101,55 @@ Deploy VMM instances that do not share this trust boundary with dedicated netd
 sockets and distinct filesystem permissions.
 
 `netd` invokes fixed absolute `ip` and `virsh` executables with separate
-arguments. It never accepts a command, executable path, TAP name, or raw XML
-from a client. Filter XML is generated internally with XML escaping and is
-validated by libvirt.
+arguments. It never accepts a command, executable path, or raw XML from a
+client. Filter XML is generated internally with XML escaping and is validated by
+libvirt.
+
+### Asking what a netd implements
+
+`netd` ships as its own package on its own release cadence, and a host may run
+an implementation that is not this one, so a VMM newer than its `netd` is the
+normal case rather than the exception. `hello` answers with a protocol version
+and a list of feature names, which the VMM asks for once at startup instead of
+inferring support one failed operation at a time. A `netd` that predates the
+operation rejects it, and that rejection *is* the answer: it claims nothing.
+
+### Finding interfaces whose VM is gone
+
+A `netd`-created TAP outlives the QEMU that used it. A VMM that died between
+creating one and recording it leaves an interface nothing can name again —
+`check` and `remove` both answer about an identity, and the identity is exactly
+what was lost. Two operations close that:
+
+- `list` enumerates the interfaces `netd` holds for one instance namespace.
+- `remove_tap` deletes one by name. The name must be one `tap_name` could have
+  produced, so it reaches nothing `netd` did not create, and any binding is
+  cleared best-effort because the caller cannot say whether there was one.
+
+The VMM runs this at startup, alongside the cleanup it already does for
+supervisor processes: anything `netd` holds that maps to no VM it knows is
+removed. It over-approximates which TAPs are still wanted, since keeping a leak
+is the cheaper mistake.
+
+Listing is scoped to an instance namespace because one host runs several VMMs
+against one `netd`, and a caller reconciling its own VMs must not mistake
+another's interfaces for orphans. `netd` records the namespace in the TAP's
+`ifalias` rather than in a state file, so a crash leaves nothing to repair. The
+identity itself is not recorded: `IFALIAS_MAX` is 255 bytes and two 128-byte
+identifiers do not fit, and a caller can already derive every TAP name it owns.
+The namespace is the only part it cannot.
+
+An interface created by an older `netd` carries no stamp and is therefore listed
+by nobody. It is left in place rather than attributed to whoever asks, which
+keeps reconciliation from deleting an interface it cannot prove is an orphan.
+
+### Fields that only travel
+
+`workdir` on both prepare operations names the VM's directory on the host. It is
+untrusted and never read for a decision — any process that can reach the socket
+can assert anything — and exists so that an operator reading `netd`'s log can get
+from an opaque TAP name back to the VM that asked for it without going through
+the VMM.
 
 ## Deployment modes
 

dstack/vmm/src/app.rs

@@ -314,6 +314,12 @@ pub struct App {
 
 const GUEST_AGENT_RPC_TIMEOUT: Duration = Duration::from_secs(30);
 
+/// How many NIC indices per VM startup reconciliation treats as possibly still
+/// in use. Deliberately larger than any VM the VMM builds: naming a TAP that no
+/// longer belongs to anything keeps a leak, while failing to name a live one
+/// would delete it.
+const RECONCILE_MAX_NICS: usize = 16;
+
 impl App {
     pub(crate) fn lock(&self) -> MutexGuard<'_, AppState> {
         self.state.lock().or_panic("mutex poisoned")
@@ -564,6 +570,13 @@ impl App {
             return Ok(());
         }
         let qemu_uid = Uid::effective().as_raw();
+        // Only ever read back out of a log line. netd is told where the VM
+        // lives so an operator holding an opaque TAP name can get to the VM
+        // without going through the VMM first.
+        let workdir = self
+            .work_dir(&vm.manifest.id)
+            .map(|dir| dir.path().display().to_string())
+            .unwrap_or_default();
         let mut prepared = Vec::new();
         for (nic_index, network) in networks.iter_mut().enumerate() {
             if !needs_netd_interface(network, &self.config.cvm) {
@@ -593,6 +606,7 @@ impl App {
                     // libvirt at all.
                     filtered,
                     queues,
+                    workdir: workdir.clone(),
                 }),
                 NetworkingMode::Macvtap => NetdRequest::PrepareMacvtap(PrepareMacvtapRequest {
                     identity: identity.clone(),
@@ -601,6 +615,7 @@ impl App {
                     qemu_uid,
                     mode: network.macvtap_mode.clone(),
                     queues,
+                    workdir: workdir.clone(),
                 }),
                 NetworkingMode::User | NetworkingMode::Custom => continue,
             };
@@ -1067,9 +1082,75 @@ impl App {
             }
         }
 
+        self.reconcile_netd_interfaces(&loaded_vm_ids).await;
+
         Ok(())
     }
 
+    /// Deletes host interfaces netd still holds for VMs this VMM no longer has.
+    ///
+    /// The same cleanup the loop above does for supervisor processes, for the
+    /// other resource a crash can strand. A netd-created TAP outlives the QEMU
+    /// that used it, so a VMM that died between creating one and recording it
+    /// leaves an interface nothing can find again: `Check` answers about an
+    /// identity, and the identity is exactly what was lost.
+    ///
+    /// Failure is never fatal. Most nodes run no netd at all, and a leaked TAP
+    /// costs a name and an ifindex -- not a reason to refuse to start.
+    async fn reconcile_netd_interfaces(&self, loaded_vm_ids: &HashSet<String>) {
+        let socket = &self.config.netd.socket;
+        let instance_id = &self.config.cvm.instance_id;
+        // Unreachable is the ordinary case: most nodes run no netd at all.
+        let Ok(capabilities) = netd::capabilities(socket).await else {
+            debug!("no netd on this node, so nothing to reconcile");
+            return;
+        };
+        info!(
+            version = capabilities.version,
+            features = ?capabilities.features,
+            "netd is available"
+        );
+        if !capabilities.has("list") {
+            // Asking anyway would log a failure on every start of a node whose
+            // netd is simply older than this VMM.
+            info!("netd cannot list its interfaces, so orphans are left in place");
+            return;
+        }
+        let held = match netd::list(socket, instance_id).await {
+            Ok(held) => held,
+            Err(error) => {
+                warn!(%error, "failed to list netd interfaces");
+                return;
+            }
+        };
+        if held.is_empty() {
+            return;
+        }
+        // Over-approximate on purpose. A manifest edited while the VMM was down
+        // can leave a TAP at a NIC index the VM no longer has, and treating
+        // that as expected only means a leak survives -- the opposite mistake
+        // would delete a live interface out from under a running QEMU.
+        let mut expected = HashSet::new();
+        for vm_id in loaded_vm_ids {
+            for nic_index in 0..RECONCILE_MAX_NICS {
+                expected.insert(netd::tap_name(&InterfaceIdentity {
+                    instance_id: instance_id.clone(),
+                    vm_id: vm_id.clone(),
+                    nic_index,
+                }));
+            }
+        }
+        for interface in held {
+            if expected.contains(&interface.tap) {
+                continue;
+            }
+            info!(tap = %interface.tap, kind = %interface.kind, "removing orphaned netd interface");
+            if let Err(error) = netd::remove_tap(socket, &interface.tap).await {
+                warn!(tap = %interface.tap, %error, "failed to remove orphaned netd interface");
+            }
+        }
+    }
+
     /// Reload VMs directory and sync with memory state while preserving statistics
     pub async fn reload_vms_sync(&self) -> Result<ReloadVmsResponse> {
         let vm_path = self.vm_dir();