Skip to content

Commit 6f1fe62

Browse files
committed
Update README.md; add Contributor License Agreement, update License Agreement, remove vgpu support, add iommufd support
Signed-off-by: Rajat Chopra <rajatc@nvidia.com>
1 parent 23a92da commit 6f1fe62

File tree

128 files changed

+2920
-12878
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

128 files changed

+2920
-12878
lines changed

.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
_out
22
vendor/**/*_test.go
33
coverage.out
4-
nvidia-kubevirt-gpu-device-plugin
4+
nvidia-sandbox-device-plugin

Dockerfile

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ FROM nvcr.io/nvidia/cuda:12.9.1-base-ubi9 as builder
2828

2929
RUN yum install -y wget make gcc
3030

31-
ARG GOLANG_VERSION=1.24.0
31+
ARG GOLANG_VERSION=1.24.7
3232
RUN wget -nv -O - https://storage.googleapis.com/golang/go${GOLANG_VERSION}.linux-amd64.tar.gz \
3333
| tar -C /usr/local -xz
3434

@@ -38,7 +38,7 @@ ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
3838
ENV GOOS=linux\
3939
GOARCH=amd64
4040

41-
WORKDIR /go/src/kubevirt-gpu-device-plugin
41+
WORKDIR /go/src/sandbox-device-plugin
4242

4343
COPY . .
4444

@@ -48,17 +48,17 @@ FROM nvcr.io/nvidia/distroless/go:v3.1.11
4848

4949
ARG VERSION
5050

51-
LABEL io.k8s.display-name="NVIDIA KubeVirt GPU Device Plugin"
52-
LABEL name="NVIDIA KubeVirt GPU Device Plugin"
51+
LABEL io.k8s.display-name="NVIDIA Sandbox Device Plugin"
52+
LABEL name="NVIDIA Sandbox Device Plugin"
5353
LABEL vendor="NVIDIA"
5454
LABEL version="${VERSION}"
5555
LABEL release="N/A"
56-
LABEL summary="NVIDIA device plugin for KubeVirt"
56+
LABEL summary="NVIDIA Sandbox Device Plugin"
5757
LABEL description="See summary"
5858

59-
COPY --from=builder /go/src/kubevirt-gpu-device-plugin/nvidia-kubevirt-gpu-device-plugin /usr/bin/
60-
COPY --from=builder /go/src/kubevirt-gpu-device-plugin/utils/pci.ids /usr/pci.ids
59+
COPY --from=builder /go/src/sandbox-device-plugin/nvidia-sandbox-device-plugin /usr/bin/
60+
COPY --from=builder /go/src/sandbox-device-plugin/utils/pci.ids /usr/pci.ids
6161

6262
USER 0:0
6363

64-
CMD ["nvidia-kubevirt-gpu-device-plugin"]
64+
CMD ["nvidia-sandbox-device-plugin"]

LICENSE

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
1+
Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22

33
Redistribution and use in source and binary forms, with or without
44
modification, are permitted provided that the following conditions

Makefile

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,19 +24,19 @@
2424
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
2525
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
2626

27-
DOCKER_REPO ?= "nvcr.io/nvidia/kubevirt-gpu-device-plugin"
27+
DOCKER_REPO ?= "nvcr.io/nvidia/sandbox-device-plugin"
2828
DOCKER_TAG ?= v1.4.0
2929

3030
PCI_IDS_URL ?= https://pci-ids.ucw.cz/v2.2/pci.ids
3131

3232
build:
33-
go build -o nvidia-kubevirt-gpu-device-plugin kubevirt-gpu-device-plugin/cmd
33+
go build -o nvidia-sandbox-device-plugin ./cmd
3434
test:
3535
go test ./... -coverprofile=coverage.out -v
3636
test-coverage:
3737
go tool cover -html=coverage.out
3838
clean:
39-
rm -rf nvidia-kubevirt-gpu-device-plugin && rm -rf coverage.out
39+
rm -f nvidia-sandbox-device-plugin && rm -rf coverage.out
4040
build-image:
4141
docker build . -t $(DOCKER_REPO):$(DOCKER_TAG)
4242
push-image: build-image

README.md

Lines changed: 8 additions & 78 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
# NVIDIA K8s Device Plugin to assign GPUs and vGPUs to KubeVirt VMs
2-
3-
> Starting from v1.1.0, we will only be supporting KubeVirt v0.36.0 or newer. Please use v1.0.1 for compatibility with older KubeVirt versions.
1+
# NVIDIA K8s Device Plugin to assign Passthrough GPUs to Kata VMs for Confidential Containers
42

53
## Table of Contents
64
- [About](#about)
@@ -10,26 +8,21 @@
108
- [Docs](#docs)
119

1210
## About
13-
This is a kubernetes device plugin that can discover and expose GPUs and vGPUs on a kubernetes node. This device plugin will enable to launch GPU attached [KubeVirt](https://github.com/kubevirt/kubevirt/blob/master/README.md) VMs in your kubernetes cluster. Its specifically developed to serve KubeVirt workloads in a Kubernetes cluster.
11+
This is a kubernetes device plugin that can discover and expose GPUs for passthrough on a kubernetes node. This device plugin will enable to launch GPU attached [Kata](https://katacontainers.io/) VM based containers in your kubernetes cluster. Its specifically developed to serve Kata workloads in a Kubernetes cluster.
1412

1513

1614
## Features
1715
- Discovers Nvidia GPUs which are bound to VFIO-PCI driver and exposes them as devices available to be attached to VM in pass through mode.
18-
- Discovers Nvidia vGPUs configured on a kubernetes node and exposes them to be attached to KubeVirt VMs
1916
- Performs basic health check on the GPU on a kubernetes node.
2017

2118
## Prerequisites
22-
- Need to have Nvidia GPU configured for GPU passthrough or vGPU. Quickstart section provides details about this
19+
- Need to have Nvidia GPU configured for GPU passthrough. Quickstart section provides details about this
2320
- Kubernetes version >= v1.11
24-
- KubeVirt release >= v0.36.0
25-
- KubeVirt GPU feature gate should be enabled and permitted devices should be whitelisted. Feature gate is enabled by creating a ConfigMap. ConfigMap yaml can be found under `/examples`.
21+
- Kata release >= v3.23.0
2622

2723
## Quick Start
2824

29-
Before starting the device plug, the GPUs on a kubernetes node need to configured to be in GPU pass through mode or vGPU mode
30-
31-
### Whitelist GPU and vGPU in KubeVirt CR
32-
GPUs and vGPUs should be allowlisted in the KubeVirt CR following the instructions outlined [here](https://kubevirt.io/user-guide/virtual_machines/host-devices/#listing-permitted-devices). An example KubeVirt CR can be found under `/examples`.
25+
Before starting the device plug, the GPUs on a kubernetes node need to configured to be in GPU pass through mode.
3326

3427
### Preparing a GPU to be used in pass through mode
3528
GPU needs to be loaded with VFIO-PCI driver to be used in pass through mode
@@ -107,75 +100,12 @@ $ lspci -nnk -d 10de:
107100
Kernel driver in use: vfio-pci
108101
Kernel modules: nouveau
109102
```
110-
--------------------------------------------------------------
111-
### Preparing a GPU to be used in vGPU mode
112-
Nvidia Virtual GPU manager needs to be installed on the host to configure GPUs in vGPU mode.
113-
114-
##### 1. Change to the mdev_supported_types directory for the physical GPU.
115-
```shell
116-
$ cd /sys/class/mdev_bus/domain\:bus\:slot.function/mdev_supported_types/
117-
```
118-
This example changes to the mdev_supported_types directory for the GPU with the domain 0000 and PCI device BDF 06:00.0.
119-
```shell
120-
$ cd /sys/bus/pci/devices/0000\:06\:00.0/mdev_supported_types/
121-
```
122-
##### 2. Find out which subdirectory of mdev_supported_types contains registration information for the vGPU type that you want to create.
123-
```shell
124-
$ grep -l "vgpu-type" nvidia-*/name
125-
vgpu-type
126-
```
127-
The vGPU type, for example, M10-2Q.
128-
This example shows that the registration information for the M10-2Q vGPU type is contained in the nvidia-41 subdirectory of mdev_supported_types.
129-
```shell
130-
$ grep -l "M10-2Q" nvidia-*/name
131-
nvidia-41/name
132-
```
133-
##### 3. Confirm that you can create an instance of the vGPU type on the physical GPU.
134-
```shell
135-
$ cat subdirectory/available_instances
136-
```
137-
**subdirectory** -- The subdirectory that you found in the previous step, for example, nvidia-41.
138-
139-
The number of available instances must be at least 1. If the number is 0, either an instance of another vGPU type already exists on the physical GPU, or the maximum number of allowed instances has already been created.
140-
141-
This example shows that four more instances of the M10-2Q vGPU type can be created on the physical GPU.
142-
```shell
143-
$ cat nvidia-41/available_instances
144-
4
145-
```
146-
##### 4. Generate a correctly formatted universally unique identifier (UUID) for the vGPU.
147-
```shell
148-
$ uuidgen
149-
aa618089-8b16-4d01-a136-25a0f3c73123
150-
```
151-
##### 5. Write the UUID that you obtained in the previous step to create the file in the registration information directory for the vGPU type that you want to create.
152-
```shell
153-
$ echo "uuid"> subdirectory/create
154-
```
155-
**uuid** -- The UUID that you generated in the previous step, which will become the UUID of the vGPU that you want to create.
156-
157-
**subdirectory** -- The registration information directory for the vGPU type that you want to create, for example, nvidia-41.
158-
159-
This example creates an instance of the M10-2Q vGPU type with the UUID aa618089-8b16-4d01-a136-25a0f3c73123.
160-
```shell
161-
$ echo "aa618089-8b16-4d01-a136-25a0f3c73123" > nvidia-41/create
162-
```
163-
An mdev device file for the vGPU is added to the parent physical device directory of the vGPU. The vGPU is identified by its UUID.
164-
165-
The /sys/bus/mdev/devices/ directory contains a symbolic link to the mdev device file.
166-
167-
##### 6. Confirm that the vGPU was created.
168-
```shell
169-
$ ls -l /sys/bus/mdev/devices/
170-
total 0
171-
lrwxrwxrwx. 1 root root 0 Nov 24 13:33 aa618089-8b16-4d01-a136-25a0f3c73123 -> ../../../devices/pci0000:00/0000:00:03.0/0000:03:00.0/0000:04:09.0/0000:06:00.0/aa618089-8b16-4d01-a136-25a0f3c73123
172-
```
173103

174104
## Docs
175105
### Deployment
176106
The daemon set creation yaml can be used to deploy the device plugin.
177107
```
178-
kubectl apply -f nvidia-kubevirt-gpu-device-plugin.yaml
108+
kubectl apply -f nvidia-sandbox-device-plugin.yaml
179109
```
180110

181111
Example YAMLs for creating VMs with GPU/vGPU are in the `examples` folder
@@ -185,7 +115,7 @@ Example YAMLs for creating VMs with GPU/vGPU are in the `examples` folder
185115
Change to proper DOCKER_REPO and DOCKER_TAG env before building images
186116
e.g.
187117
```shell
188-
export DOCKER_REPO="quay.io/nvidia/kubevirt-gpu-device-plugin"
118+
export DOCKER_REPO="quay.io/nvidia/nvidia-sandbox-device-plugin"
189119
export DOCKER_TAG=devel
190120
```
191121

@@ -203,5 +133,5 @@ make push-image DOCKER_REPO=<docker-repo-url> DOCKER_TAG=<image-tag>
203133
```
204134
### To Do
205135
- Improve the healthcheck mechanism for GPUs with VFIO-PCI drivers
206-
- Support GetPreferredAllocation API of DevicePluginServer. It returns a preferred set of devices to allocate from a list of available ones. The resulting preferred allocation is not guaranteed to be the allocation ultimately performed by the devicemanager. It is only designed to help the devicemanager make a more informed allocation decision when possible. It has not been implemented in kubevirt-gpu-device-plugin.
136+
- Support GetPreferredAllocation API of DevicePluginServer. It returns a preferred set of devices to allocate from a list of available ones. The resulting preferred allocation is not guaranteed to be the allocation ultimately performed by the devicemanager. It is only designed to help the devicemanager make a more informed allocation decision when possible. It has not been implemented in sandbox-device-plugin.
207137
--------------------------------------------------------------

cmd/main.go

Lines changed: 39 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,39 @@
1-
/*
2-
* Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
3-
*
4-
* Redistribution and use in source and binary forms, with or without
5-
* modification, are permitted provided that the following conditions
6-
* are met:
7-
* * Redistributions of source code must retain the above copyright
8-
* notice, this list of conditions and the following disclaimer.
9-
* * Redistributions in binary form must reproduce the above copyright
10-
* notice, this list of conditions and the following disclaimer in the
11-
* documentation and/or other materials provided with the distribution.
12-
* * Neither the name of NVIDIA CORPORATION nor the names of its
13-
* contributors may be used to endorse or promote products derived
14-
* from this software without specific prior written permission.
15-
*
16-
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17-
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18-
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19-
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20-
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21-
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22-
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23-
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24-
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25-
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26-
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27-
*/
28-
29-
package main
30-
31-
import "kubevirt-gpu-device-plugin/pkg/device_plugin"
32-
33-
func main() {
34-
device_plugin.InitiateDevicePlugin()
35-
}
1+
/*
2+
* Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
*
4+
* Redistribution and use in source and binary forms, with or without
5+
* modification, are permitted provided that the following conditions
6+
* are met:
7+
* * Redistributions of source code must retain the above copyright
8+
* notice, this list of conditions and the following disclaimer.
9+
* * Redistributions in binary form must reproduce the above copyright
10+
* notice, this list of conditions and the following disclaimer in the
11+
* documentation and/or other materials provided with the distribution.
12+
* * Neither the name of NVIDIA CORPORATION nor the names of its
13+
* contributors may be used to endorse or promote products derived
14+
* from this software without specific prior written permission.
15+
*
16+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17+
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19+
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20+
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21+
* EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22+
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23+
* PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24+
* OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25+
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26+
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27+
*/
28+
29+
package main
30+
31+
import (
32+
"github.com/nvidia/sandbox-device-plugin/pkg/device_plugin"
33+
"os"
34+
)
35+
36+
func main() {
37+
device_plugin.PGPUAlias = os.Getenv("P_GPU_ALIAS")
38+
device_plugin.InitiateDevicePlugin()
39+
}

go.mod

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,14 @@
1-
module kubevirt-gpu-device-plugin
1+
module github.com/nvidia/sandbox-device-plugin
22

3-
go 1.23.0
3+
go 1.24.6
44

5-
toolchain go1.23.5
5+
toolchain go1.24.7
66

77
require (
8-
github.com/NVIDIA/gpu-monitoring-tools v0.0.0-20211102125545-5a2c58442e48
9-
github.com/fsnotify/fsnotify v1.8.0
8+
github.com/fsnotify/fsnotify v1.9.0
109
github.com/onsi/ginkgo/v2 v2.22.2
1110
github.com/onsi/gomega v1.36.2
12-
google.golang.org/grpc v1.70.0
11+
google.golang.org/grpc v1.72.0
1312
k8s.io/klog/v2 v2.130.1
1413
k8s.io/kubelet v0.32.2
1514
)
@@ -18,13 +17,13 @@ require (
1817
github.com/go-logr/logr v1.4.2 // indirect
1918
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
2019
github.com/gogo/protobuf v1.3.2 // indirect
21-
github.com/google/go-cmp v0.6.0 // indirect
20+
github.com/google/go-cmp v0.7.0 // indirect
2221
github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad // indirect
23-
golang.org/x/net v0.38.0 // indirect
24-
golang.org/x/sys v0.31.0 // indirect
25-
golang.org/x/text v0.23.0 // indirect
22+
golang.org/x/net v0.40.0 // indirect
23+
golang.org/x/sys v0.33.0 // indirect
24+
golang.org/x/text v0.25.0 // indirect
2625
golang.org/x/tools v0.28.0 // indirect
27-
google.golang.org/genproto/googleapis/rpc v0.0.0-20241202173237-19429a94021a // indirect
28-
google.golang.org/protobuf v1.36.1 // indirect
26+
google.golang.org/genproto/googleapis/rpc v0.0.0-20250313205543-e70fdf4c4cb4 // indirect
27+
google.golang.org/protobuf v1.36.6 // indirect
2928
gopkg.in/yaml.v3 v3.0.1 // indirect
3029
)

0 commit comments

Comments
 (0)