Skip to content

Commit c613eca

Browse files
MondayChaMondayCha
committed
Enable resource naming in config
1 parent cb6e45e commit c613eca

File tree

3 files changed

+230
-2
lines changed

3 files changed

+230
-2
lines changed

‎README.md

+4
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
[![End-to-end Tests](https://github.com/NVIDIA/k8s-device-plugin/actions/workflows/e2e.yaml/badge.svg)](https://github.com/NVIDIA/k8s-device-plugin/actions/workflows/e2e.yaml) [![Go Report Card](https://goreportcard.com/badge/github.com/NVIDIA/k8s-device-plugin)](https://goreportcard.com/report/github.com/NVIDIA/k8s-device-plugin) [![Latest Release](https://img.shields.io/github/v/release/NVIDIA/k8s-device-plugin)](https://github.com/NVIDIA/k8s-device-plugin/releases/latest)
44

5+
> This fork of the NVIDIA device plugin for Kubernetes has a `release-1.1` branch that is based on the `v0.16.1` tag of the original NVIDIA repository. This version includes all the features and updates available in the `v0.16.1` tag, along with any additional modifications specific to this fork.
6+
>
7+
> For more details on the changes and updates in this release, please refer to the [Documentation](docs/resource-naming/README.md).
8+
59
## Table of Contents
610

711
- [About](#about)

‎cmd/nvidia-device-plugin/main.go

+8-2
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ import (
3434

3535
spec "github.com/NVIDIA/k8s-device-plugin/api/config/v1"
3636
"github.com/NVIDIA/k8s-device-plugin/internal/info"
37-
"github.com/NVIDIA/k8s-device-plugin/internal/logger"
3837
"github.com/NVIDIA/k8s-device-plugin/internal/plugin"
3938
"github.com/NVIDIA/k8s-device-plugin/internal/rm"
4039
"github.com/NVIDIA/k8s-device-plugin/internal/watch"
@@ -280,7 +279,14 @@ func startPlugins(c *cli.Context, flags []cli.Flag) ([]plugin.Interface, bool, e
280279
if err != nil {
281280
return nil, false, fmt.Errorf("unable to load config: %v", err)
282281
}
283-
spec.DisableResourceNamingInConfig(logger.ToKlog, config)
282+
283+
// This block has been commented out due to issue #69.
284+
// Date: 2024-08-07
285+
// Reason: Commenting out this block allows for the configuration of resource naming.
286+
// This enables the setting of different quotas for various GPU types.
287+
// For more details, see the GitHub issue: https://github.com/volcano-sh/devices/issues/69
288+
289+
// spec.DisableResourceNamingInConfig(logger.ToKlog, config)
284290

285291
driverRoot := root(*config.Flags.Plugin.ContainerDriverRoot)
286292
// We construct an NVML library specifying the path to libnvidia-ml.so.1

‎docs/resource-naming/README.md

+218
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
2+
# Configuring the NVIDIA Device Plugin to report Different GPU Models with Volcano Capacity Scheduling
3+
4+
Volcano v1.9.0 introduces capacity scheduling capabilities, allowing you to configure different quotas for various GPU types (essential in production environments). For example:
5+
6+
```yaml
7+
apiVersion: scheduling.volcano.sh/v1beta1
8+
kind: Queue
9+
metadata:
10+
name: queue1
11+
spec:
12+
reclaimable: true
13+
deserved: # set the deserved field.
14+
cpu: 2
15+
memory: 8Gi
16+
nvidia.com/t4: 40
17+
nvidia.com/a100: 20
18+
```
19+
20+
However, the default Nvidia Device Plugin reports GPU resources as `nvidia.com/gpu`, which does not support reporting different GPU models as shown in the example.
21+
22+
This guide walks you through configuring the NVIDIA Device Plugin to report different GPU models and integrating it with Volcano's capacity scheduling.
23+
24+
## Install a Customized Device Plugin
25+
26+
In this section, we will cover how to install a customized Device Plugin using Helm. Instructions for installing `helm` can be found [here](https://helm.sh/docs/intro/install/). For prerequisites to install the Device Plugin, refer to the [documentation](../../README.md#prerequisites) for detailed instructions.
27+
28+
If you are using the GPU Operator to manage all GPU-related components, there are two options. One option is to disable the GPU Operator's management of the Device Plugin and follow the steps in this section. The other option is to modify the GPU Operator configuration accordingly.
29+
30+
### 1.1 Install a Custom Device Plugin
31+
32+
Begin by setting up the plugin's `helm` repository and updating it as follows:
33+
34+
```shell
35+
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
36+
helm repo update
37+
```
38+
39+
Then verify that the modified release (`v0.16.1`) of the plugin is available:
40+
41+
```shell
42+
$ helm search repo nvdp --version 0.16.1
43+
NAME CHART VERSION APP VERSION DESCRIPTION
44+
nvdp/nvidia-device-plugin 0.16.1 0.16.1 A Helm chart for ...
45+
```
46+
47+
Next, prepare a ConfigMap file, assuming it's named `config.yaml`. A typical configuration is as follows:
48+
49+
```yaml
50+
version: v1
51+
flags:
52+
migStrategy: "none"
53+
failOnInitError: true
54+
nvidiaDriverRoot: "/"
55+
plugin:
56+
passDeviceSpecs: false
57+
deviceListStrategy: "envvar"
58+
deviceIDStrategy: "uuid"
59+
```
60+
61+
Once this repo is updated and the ConfigMap is prepared, you can begin installing packages from it to deploy the `nvidia-device-plugin` helm chart.
62+
63+
```shell
64+
helm upgrade -i nvdp nvdp/nvidia-device-plugin \
65+
--version=0.16.1 \
66+
--namespace nvidia-device-plugin \
67+
--create-namespace \
68+
--set gfd.enabled=true \
69+
--set image.repository=volcanosh/volcano-device-plugin \
70+
--set image.tag=${TODO_VOLCANO_SH_DOCKER_IMAGE} \
71+
--set config.default=default-config \
72+
--set-file config.map.default-config=config.yaml
73+
```
74+
75+
Here is a brief explanation of some of the configuration parameters:
76+
77+
| Command | Usage |
78+
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
79+
| `--set gfd.enabled=true` | Enables the GFD feature. If you already have NFD deployed on your cluster and do not wish for it to be pulled in by this installation, you can disable it with `nfd.enabled=false`. |
80+
| `--set image.repository`<br>`--set image.tag` | Replaces the official image with the _Volcano_ community-modified version. |
81+
| `--set config.default`<br>`--set-file config.map` | Sets the ConfigMap for the Device Plugin to configure resource renaming rules. |
82+
83+
If you wish to set multiple ConfigMaps to allow different policies for different nodes or configure Time Slicing, refer to the [documentation](../../README.md#multiple-config-file-example) for more information.
84+
85+
The _Volcano_ community provides an image based on NVIDIA Device Plugin 0.16.1. If this version does not meet your needs, you can refer to the [discussion](https://github.com/NVIDIA/k8s-device-plugin/issues/424) and create your custom image that supports Resource Naming.
86+
87+
### 1.2 Set Resource Renaming Rules
88+
89+
Once the Device Plugin is deployed, to report resources like `nvidia.com/a100`, you need to update the ConfigMap to provide renaming rules, for example:
90+
91+
```yaml
92+
version: v1
93+
flags:
94+
migStrategy: "none"
95+
failOnInitError: true
96+
nvidiaDriverRoot: "/"
97+
plugin:
98+
passDeviceSpecs: false
99+
deviceListStrategy: envvar
100+
deviceIDStrategy: uuid
101+
# new
102+
resources:
103+
gpus:
104+
- pattern: "A100-*-40GB"
105+
name: a100-40gb
106+
- pattern: "*A100*"
107+
name: a100
108+
```
109+
110+
The `pattern` field can be checked on your node by running the command:
111+
112+
```shell
113+
$ nvidia-smi --query-gpu=name --format=csv,noheader
114+
A100-SXM4-40GB
115+
...
116+
```
117+
118+
Patterns can include wildcards (i.e. ‘*’) to match against multiple devices with similar names. Additionally, the order of the entries under `resources.gpus` and `resources.mig` matters. Entries earlier in the list will be matched before entries later in the list. MIG devices can also be renamed. For more information, you can read the [documentation](https://docs.google.com/document/d/1dL67t9IqKC2-xqonMi6DV7W2YNZdkmfX7ibB6Jb-qmk).
119+
120+
After modifying the ConfigMap, you should observe that the node is now reporting `nvidia.com/a100` resources, indicating that the configuration was successful.
121+
122+
### 1.3 Clean Up Stale Resources
123+
124+
If you previously installed the Device Plugin and reported `nvidia.com/gpu` type resources, you might notice that `nvidia.com/gpu` is still retained after reconfiguring the Device Plugin. For a detailed discussion on this issue, refer to [here](https://github.com/NVIDIA/k8s-device-plugin/issues/240).
125+
126+
```yaml
127+
allocatable:
128+
nvidia.com/gpu: 0
129+
nvidia.com/a100: 8
130+
```
131+
132+
To clean up stale resources, you can start `kubectl proxy` in one terminal:
133+
134+
```shell
135+
$ kubectl proxy
136+
Starting to serve on 127.0.0.1:8001
137+
```
138+
139+
And in another terminal, run the cleanup script (note `/` needs to be escaped as `~1`):
140+
141+
```bash
142+
#!/bin/bash
143+
144+
# Check if at least one node name is provided
145+
if [ "$#" -lt 1 ]; then
146+
echo "Usage: $0 <node-name> [<node-name>...]"
147+
exit 1
148+
fi
149+
150+
# Prepare the JSON patch data
151+
PATCH_DATA=$(cat <<EOF
152+
[
153+
{"op": "remove", "path": "/status/capacity/nvidia.com~1gpu"}
154+
]
155+
EOF
156+
)
157+
158+
# Iterate over each node name provided as an argument
159+
for NODE_NAME in "$@"
160+
do
161+
# Execute the PATCH request
162+
curl --header "Content-Type: application/json-patch+json" \
163+
--request PATCH \
164+
--data "$PATCH_DATA" \
165+
http://127.0.0.1:8001/api/v1/nodes/$NODE_NAME/status
166+
167+
echo "Patch request sent for node $NODE_NAME"
168+
done
169+
```
170+
171+
Pass the node name and clean up:
172+
173+
```shell
174+
chmod +x ./patch_node_gpu.sh
175+
./patch_node_gpu.sh node1 node2
176+
```
177+
178+
## 2. Configure DCGM Exporter for Pod-Level Monitoring
179+
180+
After changing the GPU resource name, the DCGM Exporter cannot obtain Pod-level GPU usage metrics. The reason is that DCGM Exporter must fully match the resource name `nvidia.com/gpu` or those with the prefix `nvidia.com/mig-`. For more details, refer to [here](https://github.com/NVIDIA/dcgm-exporter/issues/314).
181+
182+
To resolve this issue, you need to replace the DCGM Exporter image, for example:
183+
184+
```shell
185+
kubectl patch daemonset nvidia-dcgm-exporter -n gpu-operator --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"volcanosh/volcano-dcgm-exporter:todo"}]'
186+
```
187+
188+
The _volcano_ community provides an image based on NVIDIA DCGM Exporter x.xx.xx. If this version does not meet your needs, you can refer to the [discussion](https://github.com/NVIDIA/dcgm-exporter/issues/314) and create your custom image.
189+
190+
## 3. Configure Volcano to Use the Capacity Scheduling Plugin
191+
192+
After completing the above configuration, you can edit the `volcano-scheduler-configmap` to enable the capacity plugin:
193+
194+
```yaml
195+
kind: ConfigMap
196+
apiVersion: v1
197+
metadata:
198+
name: volcano-scheduler-configmap
199+
namespace: volcano-system
200+
data:
201+
volcano-scheduler.conf: |
202+
actions: "enqueue, allocate, backfill, reclaim"
203+
tiers:
204+
- plugins:
205+
- name: priority
206+
- name: gang
207+
enablePreemptable: false
208+
- name: conformance
209+
- plugins:
210+
- name: drf
211+
enablePreemptable: false
212+
- name: predicates
213+
- name: capacity # add this field and remove proportion plugin.
214+
- name: nodeorder
215+
- name: binpack
216+
```
217+
218+
You can customize your configuration according to your needs. For more information, refer to the [documentation](https://github.com/volcano-sh/volcano/blob/release-1.10/docs/user-guide/how_to_use_capacity_plugin.md).

0 commit comments

Comments
 (0)