You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+92-6Lines changed: 92 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -13,9 +13,14 @@ PCI Devices Controller is a **Kubernetes controller** that:
13
13
This operator introduces these CRDs:
14
14
- PCIDevice
15
15
- PCIDeviceClaim
16
+
- USBDevice
17
+
- USBDeviceClaim
18
+
19
+
It introduces different types of devices that can be used here. In general, when a device is claimed, it is marked as "healthy" and is ready to be used. Each device type has own custom device plugin to manage the device lifecycle.
20
+
21
+
- PCI Device: Each resource name might have multiple PCI devices.
22
+
- USB Device: Each resource name only has one USB device.
16
23
17
-
It also introduces a custom PCIDevicePlugin. The way the deviceplugin works is by storing all
18
-
PCIDevices with the same resourceName. Then when one is claimed, the deviceplugin marks that device state as "healthy".
19
24
20
25
## PCIDevice
21
26
@@ -68,7 +73,6 @@ status:
68
73
```
69
74
70
75
71
-
72
76
## PCIDeviceClaim
73
77
74
78
This custom resource is created to store the request to prepare a device for
@@ -101,9 +105,67 @@ then `true` when it is bound to the `vfio-pci` driver.
101
105
The `status.kernelDriverToUnbind` is stored so that deleting the claim
102
106
can re-bind the device to the original driver.
103
107
108
+
## USBDevice
109
+
110
+
Collect all USB devices on all nodes, provided to users, so they know how many USB devices they could use.
111
+
112
+
Considering the same vendor/product case, we should add the bus and device number as a suffix to avoid misunderstanding. It's 002/004 in this case.
When users decide which USB devices they want to use, they will create a `usbdeviceclaim`. After `usbdeviceclaim` is created, the controller will update spec.configuration.permittedHostDevices.usb in the kubevirt resource.
137
+
138
+
We use the pciAddress field to prevent users from enabling the specified PCI device.
139
+
140
+
### CRD
141
+
142
+
```yaml
143
+
apiVersion: devices.harvesterhci.io/v1beta1
144
+
kind: USBDeviceClaim
145
+
metadata:
146
+
name: jacknode-0951-1666-002004
147
+
ownerReferences:
148
+
- apiVersion: devices.harvesterhci.io/v1beta1
149
+
kind: USBDevice
150
+
name: jacknode-0951-1666-002004
151
+
uid: b584d7d0-0820-445e-bc90-4ffbfe82d63b
152
+
spec: {}
153
+
status:
154
+
userName: ""
155
+
nodeName: jacknode
156
+
pciAddress: "0000:02:01.0"
157
+
```
158
+
104
159
# Controllers
105
160
106
-
There is be a DaemonSet that runs the PCIDevice controller on each node. The controller reconciles the stored list of PCI Devices for that node to the actual current list of PCI devices for that node.
161
+
There is a DaemonSet that runs the device controller on each node. The controller reconciles the stored list of devices for that node to the actual current list of devices for that node.
162
+
163
+
164
+
## PCIDevice Controller
165
+
166
+
The PCIDevice controller will pick up on the new currently active driver automatically, as part of its normal operation.
167
+
168
+
## PCIDeviceClaim Controller
107
169
108
170
The PCIDeviceClaim controller will process the requests by attempting to set up devices for PCI Passthrough. The steps involved are:
109
171
- Load `vfio-pci` kernel module
@@ -113,7 +175,31 @@ The PCIDeviceClaim controller will process the requests by attempting to set up
113
175
114
176
Once the device is confirmed to have been bound to `vfio-pci`, the PCIDeviceClaim controller will delete the request.
115
177
116
-
The PCIDevice controller will pick up on the new currently active driver automatically, as part of it's normal operation.
178
+
## USBDevice Controller
179
+
180
+
The USBDevice Controller triggers the collection of USB devices on the host at two specific times:
181
+
182
+
- When the pod starts.
183
+
- When there is a change in the USB devices (including plugin and removal), primarily detected through fsnotify.
184
+
185
+
Once the USBDevice Controller detects the USB devices on the host, it converts them into the USBDevice CRD format and creates the corresponding resources, ignoring any devices that have already been created.
186
+
187
+
## USBDeviceClaim Controller
188
+
189
+
The USBDeviceClaim Controller is triggered when a user decides which USB device to use and creates a USB device claim. Upon the creation of a USBDeviceClaim, this controller performs two main actions:
190
+
191
+
- It activates the device plugin.
192
+
- It updates the kubevirt resource.
193
+
194
+
This controller primarily manages the lifecycle of the device plugin, ensuring that the claimed USB device is properly integrated and available for use.
195
+
196
+
# Limitation
197
+
198
+
## USB Device
199
+
200
+
- Don't support live migration
201
+
- Don't support hot-plug (including re-plug)
202
+
- Require re-creating a USBDeviceClaim to enable the USB device after re-plugging, as re-plugging will change the device path.
117
203
118
204
# Daemon
119
205
@@ -134,7 +220,7 @@ This only detects the presence or absence of device, not the number of them.
134
220
Another reason not to use these simple labels is that we want to be able to allow our customers to set custom RBAC rules that restrict who can use which device in the cluster. We can do that with a custom `PCIDevice` CRD, but it's not clear how to do that with node labels.
0 commit comments