|
| 1 | +# Infiniband VF GUID Static Configuration |
| 2 | + |
| 3 | +SR-IOV Network Operator is able to use a static configuration file from the host filesystem to assign GUIDs to IB VFs. |
| 4 | + |
| 5 | +## Prerequisites |
| 6 | + |
| 7 | +- Infiniband NICs |
| 8 | +- Static GUID configuration file on the host filesystem |
| 9 | + |
| 10 | +### Configuration file |
| 11 | + |
| 12 | +Config file should be stored at `/etc/sriov-operator/infiniband/guids`. This location is writable across most cloud platforms. |
| 13 | + |
| 14 | +VF to GUID assignment, based on this file, is ordered. VF0 takes the GUID0, VF1 takes the GUID1 etc. |
| 15 | + |
| 16 | +Each PF has its own set of GUIDs. |
| 17 | + |
| 18 | +Example of the config file: |
| 19 | + |
| 20 | +```json |
| 21 | +[ |
| 22 | + { |
| 23 | + "pci_address": "<pci_address_1>", |
| 24 | + "guids": [ |
| 25 | + "02:00:00:00:00:00:00:00", |
| 26 | + "02:00:00:00:00:00:00:01" |
| 27 | + ] |
| 28 | + }, |
| 29 | + { |
| 30 | + "pf_guid": "<pf_guid_2>", |
| 31 | + "guidsRange": { |
| 32 | + "start": "02:00:00:00:00:aa:00:02", |
| 33 | + "end": "02:00:00:00:00:aa:00:0a" |
| 34 | + } |
| 35 | + } |
| 36 | +] |
| 37 | +``` |
| 38 | + |
| 39 | +Config file parameters: |
| 40 | + |
| 41 | +* `pci_address` is a PCI address of a PF |
| 42 | +* `pf_guid` is a GUID of a PF. Can be obtained with `ibstat` command. |
| 43 | +* `guids` is an array of VF GUID strings |
| 44 | +* `guidsRange` is an object representing the start and end of a VF GUID range. It has two fields: |
| 45 | + * `start` is a VF GUID range start |
| 46 | + * `end` is a VF GUID range end |
| 47 | + |
| 48 | +Requirements for the config file: |
| 49 | + |
| 50 | +* `pci_address` and `pf_guid` cannot be set at the same time for a single device - should return an error |
| 51 | +* if the list contains multiple entries for the same device, the first one shall be taken |
| 52 | +* `rangeStart` and `rangeEnd` are both included in the range |
| 53 | +* `guids` list and range cannot be both set at the same time for a single device - should return an error |
| 54 | +* GUIDs are assigned once and not change throughout the lifecycle of the host |
| 55 | + |
| 56 | +### Deploy SriovNetworkNodePolicy |
| 57 | + |
| 58 | +```yaml |
| 59 | +apiVersion: sriovnetwork.openshift.io/v1 |
| 60 | +kind: SriovNetworkNodePolicy |
| 61 | +metadata: |
| 62 | + name: ib-policy |
| 63 | + namespace: sriov-network-operator |
| 64 | +spec: |
| 65 | + nodeSelector: |
| 66 | + feature.node.kubernetes.io/network-sriov.capable: "true" |
| 67 | + resourceName: ib_vfs |
| 68 | + priority: 10 |
| 69 | + numVfs: 2 |
| 70 | + nicSelector: |
| 71 | + vendor: "15b3" |
| 72 | + rootDevices: |
| 73 | + - 0000:d8.00.0 |
| 74 | + deviceType: netdevice |
| 75 | +``` |
| 76 | +
|
| 77 | +## Verify assigned GUIDs |
| 78 | +
|
| 79 | +Run ip link tool to verify assigned GUIDs: |
| 80 | +
|
| 81 | +```bash |
| 82 | +ip link |
| 83 | + |
| 84 | +... |
| 85 | +6: ibp216s0f0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state UP mode DEFAULT group default qlen 256 |
| 86 | + link/infiniband 00:00:05:a9:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:f9:8f:86 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff |
| 87 | + vf 0 link/infiniband ... NODE_GUID 02:00:00:00:00:00:00:00, PORT_GUID 02:00:00:00:00:00:00:00, link-state enable, trust off, query_rss off |
| 88 | + vf 1 link/infiniband ... NODE_GUID 02:00:00:00:00:00:00:01, PORT_GUID 02:00:00:00:00:00:00:01, link-state enable, trust off, query_rss off |
| 89 | +``` |
| 90 | +
|
0 commit comments