Skip to content

Commit 062dfb9

Browse files
oshovaloshoval
committed
VEP-183: NetworkDevicesWithDRA API design
Document support for DRA-provisioned network devices, specifically SR-IOV NICs. Key additions: - ResourceClaimNetworkSource API for specifying DRA networks in spec.networks - SR-IOV integration details: device allocation via DRA, configuration via existing KubeVirt networks API. - Custom MAC address support through kubevirt.io/dra-networks. - NetworkDevicesWithDRA feature gate (Alpha). - Example VMI YAML with DRA SR-IOV network configuration. Network DRA maintains mutual exclusivity with traditional Multus-based SR-IOV per VM. The existing Multus SR-IOV API remains fully supported and unchanged. Assisted-by: claude-4.5-sonnet Signed-off-by: Or Shoval <oshoval@redhat.com>
1 parent c64c5f2 commit 062dfb9

File tree

1 file changed

+292
-0
lines changed
  • veps/sig-network/183-dra-network

1 file changed

+292
-0
lines changed

‎veps/sig-network/183-dra-network/vep.md‎

Lines changed: 292 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,292 @@
1+
# VEP #183: SR-IOV Network DRA Support
2+
3+
## Release Signoff Checklist
4+
5+
Items marked with (R) are required *prior to targeting to a milestone / release*.
6+
7+
- [x] (R) Enhancement issue created, which links to VEP dir in [kubevirt/enhancements] (not the initial VEP PR)
8+
9+
## Overview
10+
11+
This proposal adds support for DRA (Dynamic Resource Allocation) provisioned SR-IOV network devices in KubeVirt.
12+
It extends the existing KubeVirt networks API with a new `ResourceClaimNetworkSource` type, allowing SR-IOV NICs to be allocated via DRA while maintaining compatibility with the existing Multus-based SR-IOV approach.
13+
14+
This VEP builds upon the core DRA infrastructure defined in VEP #10 ([kubevirt/enhancements/pull/11](https://github.com/kubevirt/enhancements/pull/11)) to add support for network devices, specifically SR-IOV NICs.
15+
16+
## Motivation
17+
18+
DRA adoption for network devices is important for KubeVirt so that network device vendors can expect
19+
the same level of control when using Virtual Machines as they have with Containers.
20+
DRA allows network device vendors fine-grained control over device allocation and topology.
21+
22+
## Goals
23+
24+
- Align on the API changes needed to consume DRA-enabled SR-IOV network devices in KubeVirt
25+
- Align on how KubeVirt will consume SR-IOV devices via external DRA drivers
26+
- Enable DRA SR-IOV use cases available to containers to work seamlessly with KubeVirt VMIs
27+
- Support custom MAC addresses for DRA-based SR-IOV networks
28+
29+
## Non Goals
30+
31+
- Replace existing Multus-based SR-IOV network integration (remains fully supported)
32+
- Deploy DRA SR-IOV driver (handled by sriov-network-operator)
33+
- Support coexistence of DRA SR-IOV and device-plugin SR-IOV
34+
- Live migration of VMs with DRA network devices
35+
- Align on what drivers KubeVirt will support in tree
36+
37+
## Definition of Users
38+
39+
- **User**: A person who wants to attach SR-IOV network devices to a VM
40+
- **Admin**: A person who manages infrastructure and configures DRA device classes and drivers
41+
- **Developer**: A person familiar with CNCF ecosystem who develops automation using these APIs
42+
43+
## User Stories
44+
45+
- As a user, I want to consume SR-IOV network devices via DRA in my VMs
46+
- As a user, I want to specify custom MAC addresses for DRA-provisioned SR-IOV interfaces
47+
- As an admin, I want to use DRA drivers to manage SR-IOV device allocation with fine-grained control
48+
- As a developer, I want extensible APIs to build automation for DRA-based networking
49+
50+
## Use Cases
51+
52+
### Supported Use Cases
53+
54+
1. SR-IOV network devices where the DRA driver publishes required attributes in device metadata files:
55+
- `resources.kubernetes.io/pciBusID` for SR-IOV VF passthrough
56+
57+
### Future Use Cases
58+
1. Scalable Functions network devices
59+
2. Live migration of VMIs using DRA network devices (will have a VEP amendment)
60+
61+
## Repos
62+
63+
kubevirt/kubevirt
64+
65+
## Design
66+
67+
This design introduces a new feature gate: `NetworkDevicesWithDRA`.
68+
All the API changes will be gated behind this feature gate so as not to break existing functionality.
69+
70+
### API Changes
71+
72+
A new network source type `ResourceClaimNetworkSource` is added to the existing `NetworkSource` type:
73+
74+
```go
75+
// Represents the source resource that will be connected to the vm.
76+
// Only one of its members may be specified.
77+
type NetworkSource struct {
78+
Pod *PodNetwork `json:"pod,omitempty"`
79+
Multus *MultusNetwork `json:"multus,omitempty"`
80+
ResourceClaim *ResourceClaimNetworkSource `json:"resourceClaim,omitempty"`
81+
}
82+
83+
// ResourceClaimNetworkSource represents a network resource requested
84+
// via a Kubernetes ResourceClaim.
85+
type ResourceClaimNetworkSource struct {
86+
// ClaimName references the name of a ResourceClaim in the
87+
// VMI's namespace that provides the network resource.
88+
// +kubebuilder:validation:MinLength=1
89+
ClaimName string `json:"claimName"`
90+
91+
// RequestName specifies which request from the
92+
// ResourceClaim.spec.devices.requests array this network
93+
// source corresponds to.
94+
// +kubebuilder:validation:MinLength=1
95+
RequestName string `json:"requestName"`
96+
}
97+
```
98+
99+
The VMI must also include the resource claim in `spec.resourceClaims[]` (consistent with GPU and HostDevice DRA usage).
100+
101+
### Status Reporting
102+
103+
For consistency with GPUs and HostDevices, DRA-provisioned network devices populate the same `vmi.status.deviceStatus.hostDeviceStatuses[]` array. The DRA controller in virt-controller:
104+
105+
1. Identifies networks with `resourceClaim` source type
106+
2. Extracts device information from the allocated ResourceClaim and ResourceSlice
107+
3. Populates `hostDeviceStatuses` with network name and allocated device attributes (PCI address)
108+
109+
The status entry name matches the network name from `spec.networks[].name`, allowing virt-launcher to correlate the network configuration with its allocated DRA device.
110+
111+
The detailed mechanism for extracting device information from Pod status, ResourceClaim, and ResourceSlice follows the same approach described in VEP #10.
112+
113+
### SR-IOV Integration
114+
115+
When a network interface has `sriov` binding and references a network with `resourceClaim` source:
116+
117+
1. The network admitter validates that exactly one network source type (pod, multus, or resourceClaim) is specified
118+
2. Virt-controller adds the resource claim to the virt-launcher pod spec via `WithNetworksDRA()` render option
119+
3. The DRA controller populates device status with the PCI address from the ResourceSlice
120+
4. Virt-launcher's SR-IOV implementation (`CreateDRAHostDevices()`) reads the PCI address from device status and generates the appropriate libvirt hostdev XML, identical to traditional Multus-based SR-IOV
121+
122+
This approach provides clean separation: DRA handles device provisioning, KubeVirt networks API handles configuration.
123+
124+
**Important:** Traditional Multus-based SR-IOV (using `multus` network source) and DRA-based SR-IOV (using `resourceClaim` network source) are **mutually exclusive per VM**. A single VMI should not mix both approaches. The existing Multus-based SR-IOV API remains fully supported and unchanged.
125+
126+
### Custom MAC Address Support
127+
128+
To support custom MAC addresses for DRA-based SR-IOV networks, KubeVirt annotates the virt-launcher pod with requested MAC addresses. The MAC address is taken from the existing `spec.domain.devices.interfaces[].macAddress` field:
129+
130+
```
131+
kubevirt.io/dra-networks: '[{"claimName":"sriov","requestName":"vf","mac":"de:ad:00:00:be:ef"}]'
132+
```
133+
134+
This preserves the structure of `k8s.v1.cni.cncf.io/networks`, but for claimName/requestName instead of NAD.
135+
136+
The SR-IOV DRA driver reads this annotation and passes the claim/request identifier along with the MAC address to the SR-IOV CNI, ensuring the network interface is configured with the specified MAC address.
137+
138+
**Design Rationale:** The annotation-based approach was chosen because it solves the case where ResourceClaim/ResourceClaimTemplate is created by the admin (not by KubeVirt). Since this approach handles the more complex admin-created claim scenario, it naturally also works for the general case where KubeVirt creates the claims ("auto" mode), providing a unified solution for both scenarios.
139+
140+
### Validation
141+
142+
Webhook validations ensure:
143+
1. Networks with `resourceClaim` source have corresponding `sriov` binding interfaces
144+
2. Each network must reference a unique `claimName` + `requestName` combination. No two DRA entities (networks, hostDevices, or GPUs) can share the same tuple, as each interface+network pair must map to exactly one device allocation
145+
3. No mixing of Multus-based and DRA-based SR-IOV in the same VMI.
146+
147+
### Component Changes
148+
149+
**Virt-Controller:**
150+
- Renders virt-launcher pod spec with resource claims from `vmi.spec.resourceClaims[]` referenced by `vmi.spec.networks[].resourceClaim`
151+
- Annotates virt-launcher pod with `kubevirt.io/dra-networks` containing MAC addresses from `spec.domain.devices.interfaces[].macAddress`
152+
153+
**Virt-Launcher:**
154+
- For SR-IOV networks with DRA, virt-launcher uses `vmi.status.deviceStatus` to generate the domain XML instead of Kubevirt's downwardAPI file as in the case of device-plugins
155+
- The `CreateDRAHostDevices()` function generates hostdev XML by:
156+
- Filtering interfaces with SRIOV binding that reference networks with resourceClaim source
157+
- Looking up the corresponding device status entry by network name
158+
- Extracting the PCI address from device status attributes
159+
- Generating standard libvirt hostdev XML
160+
- **Note:** If the ResourceClaim/ResourceClaimTemplate has `count != 1`, KubeVirt will consume the first device from the allocated devices
161+
162+
## API Examples
163+
164+
### VMI with DRA SR-IOV Network
165+
166+
```yaml
167+
---
168+
apiVersion: resource.k8s.io/v1
169+
kind: DeviceClass
170+
metadata:
171+
name: sriov.network.example.com
172+
spec:
173+
selectors:
174+
- cel:
175+
expression: device.driver == 'sriov.network.example.com'
176+
---
177+
apiVersion: resource.k8s.io/v1
178+
kind: ResourceClaimTemplate
179+
metadata:
180+
name: sriov-network-claim-template
181+
namespace: default
182+
spec:
183+
spec:
184+
devices:
185+
requests:
186+
- name: sriov-nic-request
187+
exactly:
188+
deviceClassName: sriov.network.example.com
189+
---
190+
apiVersion: kubevirt.io/v1
191+
kind: VirtualMachineInstance
192+
metadata:
193+
name: vmi-sriov-dra
194+
namespace: default
195+
spec:
196+
domain:
197+
devices:
198+
interfaces:
199+
- name: sriov-net
200+
sriov: {}
201+
macAddress: "de:ad:00:00:be:ef"
202+
networks:
203+
- name: sriov-net
204+
resourceClaim:
205+
claimName: sriov-network-claim
206+
requestName: sriov-nic-request
207+
resourceClaims:
208+
- name: sriov-network-claim
209+
resourceClaimTemplateName: sriov-network-claim-template
210+
status:
211+
deviceStatus:
212+
hostDeviceStatuses:
213+
- name: sriov-net
214+
deviceResourceClaimStatus:
215+
name: 0000-05-00-1
216+
resourceClaimName: virt-launcher-vmi-sriov-dra-sriov-network-claim-abc123
217+
attributes:
218+
pciAddress: 0000:05:00.1
219+
---
220+
apiVersion: v1
221+
kind: Pod
222+
metadata:
223+
name: virt-launcher-vmi-sriov-dra
224+
namespace: default
225+
annotations:
226+
kubevirt.io/dra-networks: '[{"claimName":"sriov-network-claim","requestName":"sriov-nic-request","mac":"de:ad:00:00:be:ef"}]'
227+
spec:
228+
containers:
229+
- name: compute
230+
image: virt-launcher
231+
resources:
232+
claims:
233+
- name: sriov-network-claim
234+
request: sriov-nic-request
235+
resourceClaims:
236+
- name: sriov-network-claim
237+
resourceClaimTemplateName: sriov-network-claim-template
238+
status:
239+
resourceClaimStatuses:
240+
- name: sriov-network-claim
241+
resourceClaimName: virt-launcher-vmi-sriov-dra-sriov-network-claim-abc123
242+
```
243+
244+
## Scalability
245+
246+
The DRA controller in virt-controller uses existing shared informers (no additional watch calls) and filters events to relevant status sections. See [VEP #10](../../sig-compute/10-dra-devices/vep.md#scalability) for detailed scalability analysis.
247+
248+
## Update/Rollback Compatibility
249+
250+
- Changes are upgrade compatible
251+
- Rollback works as long as feature gate is disabled
252+
- If the feature is enabled, VMIs using DRA network devices must be deleted and feature gate disabled before attempting rollback
253+
254+
## Functional Testing Approach
255+
256+
- Unit tests with optimum coverage for new code
257+
- New e2e test lane with all current SR-IOV tests using the new API
258+
(excluding migration tests, which will be added when migration is supported)
259+
260+
## Implementation History
261+
262+
- 2026-01-20: Initial design/VEP proposal for SR-IOV Network DRA support
263+
264+
## Graduation Requirements
265+
266+
### Alpha
267+
268+
- Code changes behind `NetworkDevicesWithDRA` feature gate
269+
- Unit tests
270+
- E2E tests with SR-IOV DRA driver (excluding migration)
271+
272+
### Beta
273+
274+
- Evaluate user and driver author experience
275+
- Consider additional use cases if any
276+
- Work with Kubernetes community on standardizing device information injection
277+
- Live migration support for DRA network devices
278+
- Live migration will use CDI/NRI to inject device information as files into each pod (mappings of request/claim to PCI addresses)
279+
- Each virt-launcher reads its pod-specific device file, avoiding conflicts in VMI status
280+
- Might be initially implemented by SR-IOV DRA driver; future Kubernetes support may generalize this (see [kubernetes/enhancements#5606](https://github.com/kubernetes/enhancements/pull/5606))
281+
- Details: https://github.com/k8snetworkplumbingwg/dra-driver-sriov/pull/62
282+
283+
### GA
284+
285+
- Upgrade/downgrade testing
286+
287+
## References
288+
289+
- DRA: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/
290+
- SR-IOV DRA driver: https://github.com/k8snetworkplumbingwg/dra-driver-sriov
291+
- VEP #10 (DRA devices): /veps/sig-compute/10-dra-devices/vep.md
292+
- Kubernetes DRA device information injection: https://github.com/kubernetes/enhancements/pull/5606

0 commit comments

Comments
 (0)