feat(e2e): Under testing kai hack deployments, Add hami testing option, installing kai-resource-isolator by dttung2905 · Pull Request #2033 · kai-scheduler/KAI-Scheduler

dttung2905 · 2026-08-08T08:42:15Z

Description

As the final part of this #1821, I'm working on adding e2e test to stand up kai-resource-isolator vgpu monitor to scrape and expose container VRAM metrics

Dependent on Project-HAMi/KAI-resource-isolator#22 to be merged before we can run it on CI

What I ran locally

# --- 0) Start minikube with GPU ---
minikube start --driver=docker --gpus=all --cpus=6 --memory=12288
kubectl config use-context minikube

# --- 1) NVIDIA device plugin + node labels ---
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml
kubectl label node minikube \
  nvidia.com/gpu.present=true \
  nvidia.com/gpu.memory=11264 \
  --overwrite
kubectl -n kube-system rollout status ds/nvidia-device-plugin-daemonset --timeout=180s
kubectl get node minikube -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}{.metadata.labels.nvidia\.com/gpu.memory}{"\n"}'
# expect: 1 and 11264

# --- 2) KAI-Scheduler (GPU sharing + hamicore, no nvidia RuntimeClass) ---
helm upgrade -i kai-scheduler \
  oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  --namespace kai-scheduler --create-namespace \
  --version v0.17.0 \
  --set global.gpuSharing=true \
  --set binder.plugins.hamicore.enabled=true \
  --set binder.resourceReservation.runtimeClassName="" \
  --set admission.gpuFractionRuntimeClassName="" \
  --wait

# --- 3) Isolator from LOCAL chart (monitor.enabled=true) ---
cd /home/tung/go/src/KAI-Scheduler
ISOLATOR_CHART_REF=/home/tung/go/src/KAI-resource-isolator/chart/kai-resource-isolator \
  ./hack/third_party_integrations/deploy_isolator.sh

kubectl -n kai-resource-isolator delete pod -l app.kubernetes.io/component=kai-libsync
kubectl -n kai-resource-isolator rollout status ds/kai-resource-isolator-libsync --timeout=120s
kubectl -n kai-resource-isolator get deploy,ds,pods
minikube ssh -- 'stat -c "%a %n" /usr/local/vgpu/containers /tmp/vgpulock'
# expect: 1777 for both

# --- 4) Run hamicore e2e ---
export PATH="$(go env GOPATH)/bin:$PATH"
ginkgo -v --trace ./test/e2e/suites/integrations/third_party/hamicore/

Then I checked the daemonset kai-resource-isolator-monitor /metrics endpoints

curl -s localhost:9394/metrics

# HELP hami_container_device_memory_bytes Container device memory usage breakdown in bytes (The label "context_size", "module_size", "buffer_size" and "offset" will be deprecated in v2.10.0, use hami_vgpu_memory_context_bytes, hami_vgpu_memory_module_bytes and hami_vgpu_memory_buffer_bytes instead)
# TYPE hami_container_device_memory_bytes gauge
hami_container_device_memory_bytes{buffer_size="67108864",container="ubuntu-container",context_size="163577856",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",module_size="0",namespace="kai-zjldfiaddb",offset="0",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 2.3068672e+08
# HELP hami_container_device_utilization_ratio Container device SM utilization ratio
# TYPE hami_container_device_utilization_ratio gauge
hami_container_device_utilization_ratio{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 0
# HELP hami_container_last_kernel_elapsed_seconds Seconds since last kernel execution in container
# TYPE hami_container_last_kernel_elapsed_seconds gauge
hami_container_last_kernel_elapsed_seconds{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 2
# HELP hami_host_gpu_memory_used_bytes GPU device memory usage in bytes
# TYPE hami_host_gpu_memory_used_bytes gauge
hami_host_gpu_memory_used_bytes{device_index="0",device_type="NVIDIA-NVIDIA GeForce RTX 2080 Ti",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",zone="vGPU"} 3.5586048e+09
# HELP hami_host_gpu_utilization_ratio GPU core utilization ratio (0-100)
# TYPE hami_host_gpu_utilization_ratio gauge
hami_host_gpu_utilization_ratio{device_index="0",device_type="NVIDIA-NVIDIA GeForce RTX 2080 Ti",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",zone="vGPU"} 16
# HELP hami_vgpu_memory_buffer_bytes Container device memory buffer size in bytes
# TYPE hami_vgpu_memory_buffer_bytes gauge
hami_vgpu_memory_buffer_bytes{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 6.7108864e+07
# HELP hami_vgpu_memory_context_bytes Container device memory context size in bytes
# TYPE hami_vgpu_memory_context_bytes gauge
hami_vgpu_memory_context_bytes{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 1.63577856e+08
# HELP hami_vgpu_memory_limit_bytes vGPU device memory limit in bytes
# TYPE hami_vgpu_memory_limit_bytes gauge
hami_vgpu_memory_limit_bytes{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 2.125463552e+09
# HELP hami_vgpu_memory_module_bytes Container device memory module size in bytes
# TYPE hami_vgpu_memory_module_bytes gauge
hami_vgpu_memory_module_bytes{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 0
# HELP hami_vgpu_memory_used_bytes vGPU device memory usage in bytes
# TYPE hami_vgpu_memory_used_bytes gauge
hami_vgpu_memory_used_bytes{container="ubuntu-container",device_uuid="GPU-066f734b-226c-30ba-4c7b-8d0c3221093f",namespace="kai-zjldfiaddb",pod="ccdeemaoeo",vdevice_index="0",zone="vGPU"} 2.3068672e+08

Local test output

Running Suite: HAMi-core Resource Isolation Suite - /home/tung/go/src/KAI-Scheduler/test/e2e/suites/integrations/third_party/hamicore
=====================================================================================================================================
Random Seed: 1786176994

Will run 4 of 4 specs
------------------------------
HAMi-core resource isolation gpu-memory: CUDA_DEVICE_MEMORY_LIMIT is injected and bounded [reservationPod]
/home/tung/go/src/KAI-Scheduler/test/e2e/suites/integrations/third_party/hamicore/hamicore_test.go:131
  "level"=0 "msg"="GPU info" "node"="minikube" "totalGPUMemMiB"=11264
  "level"=0 "msg"="CUDA_DEVICE_MEMORY_LIMIT" "value"="2027m"
• [15.369 seconds]
------------------------------
HAMi-core resource isolation gpu-memory: nvidia-smi reports limited GPU memory matching CUDA_DEVICE_MEMORY_LIMIT [reservationPod]
/home/tung/go/src/KAI-Scheduler/test/e2e/suites/integrations/third_party/hamicore/hamicore_test.go:159

  === nvidia-smi output inside pod kai-zjldfiaddb/rsmbrkvnam ===
  Sat Aug  8 08:16:57 2026       
  +-----------------------------------------------------------------------------------------+
  | NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
  +-----------------------------------------+------------------------+----------------------+
  | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
  |                                         |                        |               MIG M. |
  |=========================================+========================+======================|
  |   0  NVIDIA GeForce RTX 2080 Ti     Off |   00000000:09:00.0  On |                  N/A |
  | 46%   50C    P3             72W /  250W |       0MiB /   2027MiB |     30%      Default |
  |                                         |                        |                  N/A |
  +-----------------------------------------+------------------------+----------------------+

  +-----------------------------------------------------------------------------------------+
  | Processes:                                                                              |
  |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
  |        ID   ID                                                               Usage      |
  |=========================================================================================|
  |  No running processes found                                                             |
  +-----------------------------------------------------------------------------------------+

  ===
  "level"=0 "msg"="GPU info" "node"="minikube" "totalGPUMemMiB"=11264
  "level"=0 "msg"="nvidia-smi inside container" "memory.total (MiB)"=2027
  "level"=0 "msg"="CUDA_DEVICE_MEMORY_LIMIT" "value"="2027m" "parsedMiB"=2027
• [4.281 seconds]
------------------------------
HAMi-core resource isolation gpu-fraction: CUDA_DEVICE_MEMORY_LIMIT is injected and proportional [reservationPod]
/home/tung/go/src/KAI-Scheduler/test/e2e/suites/integrations/third_party/hamicore/hamicore_test.go:195

  === nvidia-smi output inside pod kai-zjldfiaddb/seuoipcnfq ===
  Sat Aug  8 08:17:01 2026       
  +-----------------------------------------------------------------------------------------+
  | NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
  +-----------------------------------------+------------------------+----------------------+
  | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
  |                                         |                        |               MIG M. |
  |=========================================+========================+======================|
  |   0  NVIDIA GeForce RTX 2080 Ti     Off |   00000000:09:00.0  On |                  N/A |
  | 46%   50C    P3             70W /  250W |       0MiB /   2816MiB |     11%      Default |
  |                                         |                        |                  N/A |
  +-----------------------------------------+------------------------+----------------------+

  +-----------------------------------------------------------------------------------------+
  | Processes:                                                                              |
  |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
  |        ID   ID                                                               Usage      |
  |=========================================================================================|
  |  No running processes found                                                             |
  +-----------------------------------------------------------------------------------------+

  ===
  "level"=0 "msg"="GPU info" "node"="minikube" "totalGPUMemMiB"=11264 "requestedFraction"=0.25
  "level"=0 "msg"="CUDA_DEVICE_MEMORY_LIMIT" "value"="2816m" "parsedMiB"=2816
  "level"=0 "msg"="nvidia-smi inside container" "memory.total (MiB)"=2816
• [4.065 seconds]
------------------------------
HAMi-core resource isolation gpu-memory: kai-vgpu-monitor reports hami_vgpu_memory_used_bytes > 0 [reservationPod]
/home/tung/go/src/KAI-Scheduler/test/e2e/suites/integrations/third_party/hamicore/hamicore_test.go:239
  STEP: waiting for kai-vgpu-monitor to expose hami_vgpu_memory_used_bytes for the workload @ 08/08/26 09:20:26.034
  "level"=0 "msg"="hami_vgpu_memory_used_bytes" "namespace"="kai-zjldfiaddb" "pod"="ccdeemaoeo" "container"="ubuntu-container" "bytes"=230686720
• [215.095 seconds]
------------------------------

Ran 4 of 4 Specs in 238.811 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Related Issues

Related to: #1821

Checklist

Note: Ensure your PR title follows the Conventional Commits format (e.g., feat(scheduler): add new feature)

Self-reviewed
Added/updated tests (if needed)
Updated documentation (if needed)
Added a changelog fragment via make changelog (or applied the skip-changelog label). Do not edit CHANGELOG.md directly — pending fragments are folded into it at release time.

Breaking Changes

N/A

Additional Notes

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

davidLif · 2026-08-09T20:01:57Z

Hello @dttung2905,
1- I don't think that the kai e2e need to have tests that check Hami metrics, as kai doesn't really have an api/connection to the metrics themselves
2- As the integration between kai/hami is deeper then just "installing an operator for spesific kind of crds", I don't think the hack deployment should be under the third party integration, but rather have a sepearte folder and flag in the hack cluster setup. When the "test hami" flag is on, it should also set the values in the helm install accordingly

dttung2905 · 2026-08-09T20:50:56Z

Thanks @davidLif . I think both of your points make sense.

1- I don't think that the kai e2e need to have tests that check Hami metrics, as kai doesn't really have an api/connection to the metrics themselves

I totally agree that hami_* isn’t a KAI API surface. I'm happy to drop the metrics scrape assert from this PR and keep the existing hamicore coverage focused on KAI’s contract (CUDA_DEVICE_MEMORY_LIMIT / visible memory).

2- As the integration between kai/hami is deeper then just "installing an operator for spesific kind of crds", I don't think the hack deployment should be under the third party integration, but rather have a sepearte folder and flag in the hack cluster setup. When the "test hami" flag is on, it should also set the values in the helm install accordingly

I’ll move the isolator deploy out of hack/third_party_integrations into a dedicated folder (e.g. hack/hami/) and add a separate --test-hami flag in cluster setup. When that flag is set, the KAI helm install will also enable the hamicore-related values (not only global.gpuSharing). wdyt about this?

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

davidLif · 2026-08-10T06:59:57Z

Thanks @davidLif . I think both of your points make sense.

1- I don't think that the kai e2e need to have tests that check Hami metrics, as kai doesn't really have an api/connection to the metrics themselves

I totally agree that hami_* isn’t a KAI API surface. I'm happy to drop the metrics scrape assert from this PR and keep the existing hamicore coverage focused on KAI’s contract (CUDA_DEVICE_MEMORY_LIMIT / visible memory).

2- As the integration between kai/hami is deeper then just "installing an operator for spesific kind of crds", I don't think the hack deployment should be under the third party integration, but rather have a sepearte folder and flag in the hack cluster setup. When the "test hami" flag is on, it should also set the values in the helm install accordingly

I’ll move the isolator deploy out of hack/third_party_integrations into a dedicated folder (e.g. hack/hami/) and add a separate --test-hami flag in cluster setup. When that flag is set, the KAI helm install will also enable the hamicore-related values (not only global.gpuSharing). wdyt about this?

Sounds good to me.

github-actions · 2026-08-10T07:36:08Z

Total coverage: 54.5% -> 54.5% (delta 0.00%)

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

mesutoezdil · 2026-08-10T12:52:07Z

+set -euo pipefail
+
+ISOLATOR_CHART_REF="${ISOLATOR_CHART_REF:-oci://docker.io/projecthami/kai-resource-isolator}"
+ISOLATOR_CHART_VERSION="${ISOLATOR_CHART_VERSION:-1.1.0-chart}"


chart.yaml on main is still 0.1.0, not 1.1.0-chart. needs a bump after isolator pr 22 merges?

yes you are right. PR 22 needs to be merged first. I will then update the chart number

mesutoezdil · 2026-08-10T12:52:12Z

+# operators: paired with binder hamicore helm values above. Soft-gated in
+# hamicore e2e when the webhook / monitor are absent.
+if [ "$TEST_HAMI" = "true" ]; then
+    ${REPO_ROOT}/hack/hami/deploy_isolator.sh


dont see --test-hami wired into ci yet. so hamicore e2e still skips in ci for now, right?

Yes, I intentionally not wired into CI for now. Wiring them to CI will make the e2e failed because we are running against fake GPU, not real one

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

Add hami metrics e2e test

ab2f2a7

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

Move hami deploy_isolator.sh to hack/hami folder

91021da

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

davidLif reviewed Aug 10, 2026

View reviewed changes

Comment thread hack/setup-e2e-cluster.sh Outdated

Comment thread test/e2e/suites/integrations/third_party/hamicore/hamicore_test.go Outdated

remove tests for kai-resource-isolator metrics

41473f2

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

davidLif changed the title ~~feat(e2e): Add hami metrics e2e test from kai-resource-isolator~~ feat(e2e): Under testing kai hack deployments, Add hami testing option, installing kai-resource-isolator Aug 10, 2026

davidLif added the skip-changelog label Aug 10, 2026

davidLif previously approved these changes Aug 10, 2026

View reviewed changes

mesutoezdil reviewed Aug 10, 2026

View reviewed changes

Update README file

c635e80

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

dttung2905 dismissed davidLif’s stale review via c635e80 August 12, 2026 20:54

Navigation Menu

Uh oh!

feat(e2e): Under testing kai hack deployments, Add hami testing option, installing kai-resource-isolator - #2033

feat(e2e): Under testing kai hack deployments, Add hami testing option, installing kai-resource-isolator#2033
dttung2905 wants to merge 4 commits into
kai-scheduler:mainfrom
dttung2905:add-hami-metrics-e2e-test

dttung2905 commented Aug 8, 2026

Uh oh!

davidLif commented Aug 9, 2026

Uh oh!

dttung2905 commented Aug 9, 2026

Uh oh!

davidLif commented Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

mesutoezdil Aug 10, 2026 •

edited

Loading

Uh oh!

dttung2905 Aug 12, 2026

Uh oh!

mesutoezdil Aug 10, 2026 •

edited

Loading

Uh oh!

dttung2905 Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dttung2905 commented Aug 8, 2026

Description

Related Issues

Checklist

Breaking Changes

Additional Notes

Uh oh!

davidLif commented Aug 9, 2026

Uh oh!

dttung2905 commented Aug 9, 2026

Uh oh!

davidLif commented Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mesutoezdil Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dttung2905 Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

mesutoezdil Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dttung2905 Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Aug 10, 2026 •

edited

Loading

mesutoezdil Aug 10, 2026 •

edited

Loading

mesutoezdil Aug 10, 2026 •

edited

Loading