Skip to content

Commit c635e80

Browse files
committed
Update README file
Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>
1 parent 41473f2 commit c635e80

1 file changed

Lines changed: 78 additions & 0 deletions

File tree

docs/gpu-sharing/hami/README.md

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,3 +144,81 @@ curl {pod-ip}:9394/metrics
144144
### Memory value precision
145145

146146
The `gpu-memory` annotation accepts an **integer in MiB** (no unit suffix). Internally, KAI-Scheduler converts this to a GPU fraction with 2-decimal precision, which is then multiplied against the total GPU memory to compute the actual limit. As a result, the value seen in `nvidia-smi` may differ slightly from the requested value. For example, requesting `4096` MiB on a `15360` MiB GPU (T4) rounds to a `0.27` fraction, yielding `4147m` as the enforced limit.
147+
148+
## Local e2e testing
149+
150+
The HAMi-core e2e suite lives at `test/e2e/suites/integrations/third_party/hamicore/`. It checks KAI’s isolation contract (`CUDA_DEVICE_MEMORY_LIMIT` injection and limited `nvidia-smi` visible memory). It is **not wired into CI**: KAI PR e2e runs on kind with the fake GPU operator and has no real GPUs, so these specs soft-skip unless the `hamicore` binder plugin and the `kai-resource-isolator` mutating webhook are present.
151+
152+
Use a machine with a real NVIDIA GPU (for example minikube with `--gpus=all`). Docker must be able to run `docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi` before you start.
153+
154+
### 1. Start minikube with GPU access
155+
156+
```bash
157+
minikube start --driver=docker --gpus=all --cpus=6 --memory=12288
158+
kubectl config use-context minikube
159+
```
160+
161+
### 2. NVIDIA device plugin + node labels
162+
163+
The binder and e2e helpers read `nvidia.com/gpu.memory` (MiB per GPU). Label the node with the card’s total from `nvidia-smi` (example below is an 11264 MiB 2080 Ti):
164+
165+
```bash
166+
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml
167+
168+
kubectl label node minikube \
169+
nvidia.com/gpu.present=true \
170+
nvidia.com/gpu.memory=11264 \
171+
--overwrite
172+
173+
kubectl -n kube-system rollout status ds/nvidia-device-plugin-daemonset --timeout=180s
174+
kubectl get node minikube -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}{.metadata.labels.nvidia\.com/gpu.memory}{"\n"}'
175+
```
176+
177+
### 3. Install KAI with GPU sharing + hamicore
178+
179+
If the cluster has no `RuntimeClass` named `nvidia` (common on bare device-plugin minikube), clear the default runtime-class settings so admission does not reject fraction pods:
180+
181+
```bash
182+
helm upgrade -i kai-scheduler \
183+
oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
184+
--namespace kai-scheduler --create-namespace \
185+
--version v0.17.0 \
186+
--set global.gpuSharing=true \
187+
--set binder.plugins.hamicore.enabled=true \
188+
--set binder.resourceReservation.runtimeClassName="" \
189+
--set admission.gpuFractionRuntimeClassName="" \
190+
--wait
191+
```
192+
193+
### 4. Install kai-resource-isolator
194+
195+
Prefer the helper under `hack/hami/` (also used by `--test-hami` in kind cluster setup):
196+
197+
```bash
198+
# from the KAI-Scheduler repo root
199+
./hack/hami/deploy_isolator.sh
200+
201+
# or a local isolator chart checkout:
202+
# ISOLATOR_CHART_REF=/path/to/KAI-resource-isolator/chart/kai-resource-isolator \
203+
# ./hack/hami/deploy_isolator.sh
204+
```
205+
206+
Confirm the webhook exists:
207+
208+
```bash
209+
kubectl get mutatingwebhookconfiguration kai-resource-isolator-mutating
210+
kubectl -n kai-resource-isolator get deploy,ds,pods
211+
```
212+
213+
### 5. Run the suite
214+
215+
```bash
216+
export PATH="$(go env GOPATH)/bin:$PATH"
217+
go install github.com/onsi/ginkgo/v2/ginkgo@latest
218+
219+
ginkgo -v --trace ./test/e2e/suites/integrations/third_party/hamicore/
220+
```
221+
222+
### Optional: kind helper flag
223+
224+
`hack/setup-e2e-cluster.sh --test-hami` (and `hack/run-e2e-kind.sh --test-hami`) enables `binder.plugins.hamicore.enabled=true` and runs `hack/hami/deploy_isolator.sh`. That is useful for install plumbing on kind, but the hamicore specs still need a real GPU and will skip or fail against the fake GPU operator alone.

0 commit comments

Comments
 (0)