Skip to content

Commit 6a90910

Browse files
authored
readme: add usbdevice related content (#83)
Signed-off-by: Jack Yu <[email protected]>
1 parent 9c9e373 commit 6a90910

File tree

1 file changed

+92
-6
lines changed

1 file changed

+92
-6
lines changed

README.md

Lines changed: 92 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,14 @@ PCI Devices Controller is a **Kubernetes controller** that:
1313
This operator introduces these CRDs:
1414
- PCIDevice
1515
- PCIDeviceClaim
16+
- USBDevice
17+
- USBDeviceClaim
18+
19+
It introduces different types of devices that can be used here. In general, when a device is claimed, it is marked as "healthy" and is ready to be used. Each device type has own custom device plugin to manage the device lifecycle.
20+
21+
- PCI Device: Each resource name might have multiple PCI devices.
22+
- USB Device: Each resource name only has one USB device.
1623

17-
It also introduces a custom PCIDevicePlugin. The way the deviceplugin works is by storing all
18-
PCIDevices with the same resourceName. Then when one is claimed, the deviceplugin marks that device state as "healthy".
1924

2025
## PCIDevice
2126

@@ -68,7 +73,6 @@ status:
6873
```
6974
7075
71-
7276
## PCIDeviceClaim
7377
7478
This custom resource is created to store the request to prepare a device for
@@ -101,9 +105,67 @@ then `true` when it is bound to the `vfio-pci` driver.
101105
The `status.kernelDriverToUnbind` is stored so that deleting the claim
102106
can re-bind the device to the original driver.
103107

108+
## USBDevice
109+
110+
Collect all USB devices on all nodes, provided to users, so they know how many USB devices they could use.
111+
112+
Considering the same vendor/product case, we should add the bus and device number as a suffix to avoid misunderstanding. It's 002/004 in this case.
113+
114+
### CRD
115+
116+
```yaml
117+
apiVersion: devices.harvesterhci.io/v1beta1
118+
kind: USBDevice
119+
metadata:
120+
labels:
121+
nodename: jacknode
122+
name: jacknode-0951-1666-002004
123+
status:
124+
description: DataTraveler 100 G3/G4/SE9 G2/50 Kyson (Kingston Technology)
125+
devicePath: /dev/bus/usb/002/004
126+
enabled: false
127+
nodeName: jacknode
128+
pciAddress: "0000:02:01.0"
129+
productID: "1666"
130+
resourceName: kubevirt.io/jacknode-0951-1666-002004
131+
vendorID: "0951"
132+
```
133+
134+
## USBDeviceClaim
135+
136+
When users decide which USB devices they want to use, they will create a `usbdeviceclaim`. After `usbdeviceclaim` is created, the controller will update spec.configuration.permittedHostDevices.usb in the kubevirt resource.
137+
138+
We use the pciAddress field to prevent users from enabling the specified PCI device.
139+
140+
### CRD
141+
142+
```yaml
143+
apiVersion: devices.harvesterhci.io/v1beta1
144+
kind: USBDeviceClaim
145+
metadata:
146+
name: jacknode-0951-1666-002004
147+
ownerReferences:
148+
- apiVersion: devices.harvesterhci.io/v1beta1
149+
kind: USBDevice
150+
name: jacknode-0951-1666-002004
151+
uid: b584d7d0-0820-445e-bc90-4ffbfe82d63b
152+
spec: {}
153+
status:
154+
userName: ""
155+
nodeName: jacknode
156+
pciAddress: "0000:02:01.0"
157+
```
158+
104159
# Controllers
105160

106-
There is be a DaemonSet that runs the PCIDevice controller on each node. The controller reconciles the stored list of PCI Devices for that node to the actual current list of PCI devices for that node.
161+
There is a DaemonSet that runs the device controller on each node. The controller reconciles the stored list of devices for that node to the actual current list of devices for that node.
162+
163+
164+
## PCIDevice Controller
165+
166+
The PCIDevice controller will pick up on the new currently active driver automatically, as part of its normal operation.
167+
168+
## PCIDeviceClaim Controller
107169

108170
The PCIDeviceClaim controller will process the requests by attempting to set up devices for PCI Passthrough. The steps involved are:
109171
- Load `vfio-pci` kernel module
@@ -113,7 +175,31 @@ The PCIDeviceClaim controller will process the requests by attempting to set up
113175

114176
Once the device is confirmed to have been bound to `vfio-pci`, the PCIDeviceClaim controller will delete the request.
115177

116-
The PCIDevice controller will pick up on the new currently active driver automatically, as part of it's normal operation.
178+
## USBDevice Controller
179+
180+
The USBDevice Controller triggers the collection of USB devices on the host at two specific times:
181+
182+
- When the pod starts.
183+
- When there is a change in the USB devices (including plugin and removal), primarily detected through fsnotify.
184+
185+
Once the USBDevice Controller detects the USB devices on the host, it converts them into the USBDevice CRD format and creates the corresponding resources, ignoring any devices that have already been created.
186+
187+
## USBDeviceClaim Controller
188+
189+
The USBDeviceClaim Controller is triggered when a user decides which USB device to use and creates a USB device claim. Upon the creation of a USBDeviceClaim, this controller performs two main actions:
190+
191+
- It activates the device plugin.
192+
- It updates the kubevirt resource.
193+
194+
This controller primarily manages the lifecycle of the device plugin, ensuring that the claimed USB device is properly integrated and available for use.
195+
196+
# Limitation
197+
198+
## USB Device
199+
200+
- Don't support live migration
201+
- Don't support hot-plug (including re-plug)
202+
- Require re-creating a USBDeviceClaim to enable the USB device after re-plugging, as re-plugging will change the device path.
117203

118204
# Daemon
119205

@@ -134,7 +220,7 @@ This only detects the presence or absence of device, not the number of them.
134220
Another reason not to use these simple labels is that we want to be able to allow our customers to set custom RBAC rules that restrict who can use which device in the cluster. We can do that with a custom `PCIDevice` CRD, but it's not clear how to do that with node labels.
135221

136222
## License
137-
Copyright (c) 2023 [Rancher Labs, Inc.](http://rancher.com)
223+
Copyright (c) 2024 [Rancher Labs, Inc.](http://rancher.com)
138224

139225
Licensed under the Apache License, Version 2.0 (the "License");
140226
you may not use this file except in compliance with the License.

0 commit comments

Comments
 (0)