Skip to content

README.md

Latest commit

 

History

History
146 lines (114 loc) · 7.88 KB

README.md

File metadata and controls

146 lines (114 loc) · 7.88 KB
Raw

HAMi Resource Isolation

KAI Scheduler's GPU sharing feature allows multiple pods to share a single GPU, but by default it does not enforce memory limits at the CUDA level — a container requesting 2000 MiB could still see (and use) the full GPU memory via nvidia-smi and CUDA APIs.

kai-resource-isolator solves this by deploying HAMi-core, a CNCF-incubated CUDA interception library. HAMi-core hooks CUDA memory allocation calls via LD_PRELOAD and enforces per-container GPU memory limits, ensuring each container can only allocate up to its requested amount.

Architecture

┌─────────────────────────────────────────────────────────┐
│                     KAI-Scheduler                        │
│                                                         │
│  1. Schedules pod to a GPU node                         │
│  2. Injects CUDA_DEVICE_MEMORY_LIMIT env var             │
│     based on gpu-fraction or gpu-memory annotation       │
└─────────────────────────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────┐
│                kai-resource-isolator                     │
│                                                         │
│  3. Mutating webhook injects:                            │
│     - hostPath volume mount (/usr/local/vgpu)            │
│     - /etc/ld.so.preload → libvgpu.so                    │
│     - POD_UID / CONTAINER_NAME / CONTAINER_VGPU_MOUNT    │
│       (for per-container VRAM metrics)                   │
└─────────────────────────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────┐
│                     Container                            │
│                                                         │
│  4. libvgpu.so intercepts CUDA memory allocation calls   │
│  5. Enforces limit set by CUDA_DEVICE_MEMORY_LIMIT       │
└─────────────────────────────────────────────────────────┘

kai-resource-isolator combines:

Component Role
DaemonSet (libsync) Copies libvgpu.so (HAMi-core) to /usr/local/vgpu on every GPU node
Mutating webhook Injects the libvgpu hostPath volume and ld.so.preload into pods that request GPU sharing
DaemonSet (monitor, optional) Reads per-container shared-memory caches and exposes HAMi-compatible VRAM metrics on :9394

Architecture

Prerequisites

  • KAI-Scheduler version: ≥ v0.17.0

  • KAI-Scheduler deployed with GPU sharing and the hamicore plugin enabled:

    helm install kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  --set global.gpuSharing=true \
  --set binder.plugins.hamicore.enabled=true \
  --namespace kai-scheduler --create-namespace \
  --version v0.17.0

Installation

Deploy kai-resource-isolator (with optional per-container VRAM metrics):

helm install kai-resource-isolator oci://docker.io/projecthami/kai-resource-isolator \
  --namespace kai-resource-isolator --create-namespace \
  --set monitor.enabled=true \
  --set monitor.serviceMonitor.enabled=true \
  --version 1.1.0-chart

Chart versions carry a -chart suffix (e.g. 1.1.0-chart). Available versions are listed on Docker Hub.

The default monitor.nodeSelector is nvidia.com/gpu.present: "true" (NVIDIA GPU feature discovery). Set monitor.runtimeClassName=nvidia if NVML is only available through the NVIDIA runtime handler in your cluster.

For customization or more detail, see kai-resource-isolator.

Usage

Once both KAI-Scheduler and kai-resource-isolator are deployed, any pod requesting GPU sharing via gpu-fraction or gpu-memory annotations will automatically receive memory isolation:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-sharing-with-isolation
  labels:
    kai.scheduler/queue: default-queue
  annotations:
    gpu-memory: "4096"  # in MiB, no suffix
spec:
  schedulerName: kai-scheduler
  containers:
    - name: gpu-workload
      image: nvidia/cuda:12.9.2-base-ubuntu24.04
      command: ["sleep", "infinity"]

After the pod starts, nvidia-smi inside the container will show only the allocated memory instead of the full GPU memory:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       On  |   00000000:00:04.0 Off |                    0 |
| N/A   43C    P8             16W /   70W |       0MiB /   4147MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Per-container VRAM metrics

When monitor.enabled=true, kai-vgpu-monitor reads the shared-memory cache that libvgpu.so writes for each GPU container and exposes HAMi-compatible gauges such as:

  • hami_vgpu_memory_used_bytes
  • hami_vgpu_memory_limit_bytes
  • hami_container_device_utilization_ratio

Scrape them with:

curl {pod-ip}:9394/metrics

Opt-out

  • Per pod: add annotation kai-resource-isolator.io/inject: "false"
  • Per namespace: add label kai-resource-isolator.io/webhook=ignore

Memory value precision

The gpu-memory annotation accepts an integer in MiB (no unit suffix). Internally, KAI-Scheduler converts this to a GPU fraction with 2-decimal precision, which is then multiplied against the total GPU memory to compute the actual limit. As a result, the value seen in nvidia-smi may differ slightly from the requested value. For example, requesting 4096 MiB on a 15360 MiB GPU (T4) rounds to a 0.27 fraction, yielding 4147m as the enforced limit.