Skip to content

feat(e2e): Under testing kai hack deployments, Add hami testing option, installing kai-resource-isolator #2033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dttung2905 wants to merge 4 commits into
base: main
Choose a base branch
from
+163 −4
Open

feat(e2e): Under testing kai hack deployments, Add hami testing option, installing kai-resource-isolator #2033

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ab2f2a7
Add hami metrics e2e test
dttung2905 Aug 8, 2026
91021da
Move hami deploy_isolator.sh to hack/hami folder
dttung2905 Aug 9, 2026
41473f2
remove tests for kai-resource-isolator metrics
dttung2905 Aug 10, 2026
c635e80
Update README file
dttung2905 Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions hack/hami/deploy_isolator.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash
# Copyright 2026 NVIDIA CORPORATION
# SPDX-License-Identifier: Apache-2.0
#
# Installs kai-resource-isolator with kai-vgpu-monitor enabled for HAMi/hamicore
# e2e (see --test-hami in hack/setup-e2e-cluster.sh).
#
# Overrides (optional):
# ISOLATOR_CHART_REF OCI ref or local chart path
# (default: oci://docker.io/projecthami/kai-resource-isolator)
# ISOLATOR_CHART_VERSION Chart version when using OCI (default: 1.1.0-chart)
# ISOLATOR_NAMESPACE Install namespace (default: kai-resource-isolator)
# ISOLATOR_RELEASE Helm release name (default: kai-resource-isolator)
# ISOLATOR_HELM_EXTRA_ARGS Extra args appended to helm upgrade (word-split)
set -euo pipefail

ISOLATOR_CHART_REF="${ISOLATOR_CHART_REF:-oci://docker.io/projecthami/kai-resource-isolator}"
ISOLATOR_CHART_VERSION="${ISOLATOR_CHART_VERSION:-1.1.0-chart}"

@mesutoezdil mesutoezdil Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chart.yaml on main is still 0.1.0, not 1.1.0-chart. needs a bump after isolator pr 22 merges?

@dttung2905 dttung2905 Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes you are right. PR 22 needs to be merged first. I will then update the chart number

ISOLATOR_NAMESPACE="${ISOLATOR_NAMESPACE:-kai-resource-isolator}"
ISOLATOR_RELEASE="${ISOLATOR_RELEASE:-kai-resource-isolator}"

HELM_ARGS=(
upgrade --install "${ISOLATOR_RELEASE}" "${ISOLATOR_CHART_REF}"
--namespace "${ISOLATOR_NAMESPACE}"
--create-namespace
--set monitor.enabled=true
--wait
--timeout 5m
)

# --version only applies to OCI/repo charts, not a local filesystem chart path.
if [[ "${ISOLATOR_CHART_REF}" == oci://* ]] || [[ "${ISOLATOR_CHART_REF}" == *://* ]]; then
HELM_ARGS+=(--version "${ISOLATOR_CHART_VERSION}")
fi

# shellcheck disable=SC2206
if [[ -n "${ISOLATOR_HELM_EXTRA_ARGS:-}" ]]; then
EXTRA=( ${ISOLATOR_HELM_EXTRA_ARGS} )
HELM_ARGS+=("${EXTRA[@]}")
fi

echo "Installing kai-resource-isolator from ${ISOLATOR_CHART_REF} (monitor.enabled=true)..."
helm "${HELM_ARGS[@]}"

echo "Waiting for isolator webhook deployment..."
kubectl -n "${ISOLATOR_NAMESPACE}" rollout status \
"deployment/${ISOLATOR_RELEASE}-webhook" --timeout=180s

echo "kai-resource-isolator installed (mutating webhook + monitor chart resources)."
11 changes: 10 additions & 1 deletion hack/run-e2e-kind.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ GOBIN=${GOPATH}/bin

# Parse named parameters
TEST_THIRD_PARTY_INTEGRATIONS="false"
TEST_HAMI="false"
LOCAL_IMAGES_BUILD="false"
PRESERVE_CLUSTER="false"

Expand All @@ -20,6 +21,10 @@ while [[ $# -gt 0 ]]; do
TEST_THIRD_PARTY_INTEGRATIONS="true"
shift
;;
--test-hami)
TEST_HAMI="true"
shift
;;
--local-images-build)
LOCAL_IMAGES_BUILD="true"
shift
Expand All @@ -29,8 +34,9 @@ while [[ $# -gt 0 ]]; do
shift
;;
-h|--help)
echo "Usage: $0 [--test-third-party-integrations] [--local-images-build] [--preserve-cluster]"
echo "Usage: $0 [--test-third-party-integrations] [--test-hami] [--local-images-build] [--preserve-cluster]"
echo " --test-third-party-integrations: Install third party operators for compatibility testing"
echo " --test-hami: Enable HAMi/hamicore integration (binder hamicore plugin + kai-resource-isolator)"
echo " --local-images-build: Build and use local images instead of pulling from registry"
echo " --preserve-cluster: Keep the kind cluster after running the test suite"
exit 0
Expand All @@ -48,6 +54,9 @@ SETUP_ARGS=""
if [ "$TEST_THIRD_PARTY_INTEGRATIONS" = "true" ]; then
SETUP_ARGS="$SETUP_ARGS --test-third-party-integrations"
fi
if [ "$TEST_HAMI" = "true" ]; then
SETUP_ARGS="$SETUP_ARGS --test-hami"
fi
if [ "$LOCAL_IMAGES_BUILD" = "true" ]; then
SETUP_ARGS="$SETUP_ARGS --local-images-build"
fi
Expand Down
29 changes: 26 additions & 3 deletions hack/setup-e2e-cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ trap cleanup EXIT

# Parse named parameters
TEST_THIRD_PARTY_INTEGRATIONS=${TEST_THIRD_PARTY_INTEGRATIONS:-"false"}
TEST_HAMI=${TEST_HAMI:-"false"}
LOCAL_IMAGES_BUILD=${LOCAL_IMAGES_BUILD:-"false"}
INSTALL_VPA=${INSTALL_VPA:-"false"}
SKIP_KAI_INSTALL=${SKIP_KAI_INSTALL:-"false"}
Expand All @@ -41,6 +42,10 @@ while [[ $# -gt 0 ]]; do
TEST_THIRD_PARTY_INTEGRATIONS="true"
shift
;;
--test-hami)
TEST_HAMI="true"
shift
;;
--local-images-build)
LOCAL_IMAGES_BUILD="true"
shift
Expand All @@ -62,8 +67,9 @@ while [[ $# -gt 0 ]]; do
shift 2
;;
-h|--help)
echo "Usage: $0 [--test-third-party-integrations] [--local-images-build] [--install-vpa] [--skip-kai-install] [--feature-config <config>] [--kind-config <path>]"
echo "Usage: $0 [--test-third-party-integrations] [--test-hami] [--local-images-build] [--install-vpa] [--skip-kai-install] [--feature-config <config>] [--kind-config <path>]"
echo " --test-third-party-integrations: Install third party operators for compatibility testing"
echo " --test-hami: Enable HAMi/hamicore integration (binder hamicore plugin + kai-resource-isolator)"
echo " --local-images-build: Build and use local images instead of pulling from registry"
echo " --install-vpa: Install Vertical Pod Autoscaler and metrics-server"
echo " --skip-kai-install: Prepare the cluster (and images/chart with --local-images-build) without installing KAI (e.g. for gitops e2e tests)"
Expand All @@ -79,6 +85,14 @@ while [[ $# -gt 0 ]]; do
esac
done

# Extra helm --set flags for HAMi/hamicore when --test-hami is set.
HAMI_HELM_SETS=()
if [ "$TEST_HAMI" = "true" ]; then
HAMI_HELM_SETS+=(
--set "binder.plugins.hamicore.enabled=true"
)
fi

if [[ -n "$KIND_CONFIG" && "$FEATURE_CONFIG" != "default" ]]; then
echo "--feature-config cannot be used together with --kind-config"
exit 1
Expand Down Expand Up @@ -204,7 +218,8 @@ if [ "$LOCAL_IMAGES_BUILD" = "true" ]; then
else
helm upgrade -i kai-scheduler ./charts/kai-scheduler-$PACKAGE_VERSION.tgz -n kai-scheduler --create-namespace \
--values ${REPO_ROOT}/hack/kai-scheduler-fake-npe-values.yaml \
--set "global.gpuSharing=true" --set "global.registry=localhost:30100" --set "prometheus.enabled=true" --debug --wait
--set "global.gpuSharing=true" --set "global.registry=localhost:30100" --set "prometheus.enabled=true" \
"${HAMI_HELM_SETS[@]}" --debug --wait
rm -rf ./charts/kai-scheduler-$PACKAGE_VERSION.tgz
fi
cd ${REPO_ROOT}/hack
Expand All @@ -213,7 +228,8 @@ elif [ "$SKIP_KAI_INSTALL" = "true" ]; then
else
helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler -n kai-scheduler --create-namespace \
--values ${REPO_ROOT}/hack/kai-scheduler-fake-npe-values.yaml \
--set "global.gpuSharing=true" --set "prometheus.enabled=true" --wait --version "$PACKAGE_VERSION"
--set "global.gpuSharing=true" --set "prometheus.enabled=true" \
"${HAMI_HELM_SETS[@]}" --wait --version "$PACKAGE_VERSION"
fi

if [ "$SKIP_KAI_INSTALL" != "true" ]; then
Expand All @@ -222,4 +238,11 @@ if [ "$SKIP_KAI_INSTALL" != "true" ]; then
kubectl create rolebinding fake-status-updater --clusterrole=pods-patcher --serviceaccount=gpu-operator:status-updater -n kai-resource-reservation
fi

# HAMi resource isolation (kai-resource-isolator). Deeper than third-party CRD
# operators: paired with binder hamicore helm values above. Soft-gated in
# hamicore e2e when the webhook / monitor are absent.
if [ "$TEST_HAMI" = "true" ]; then
${REPO_ROOT}/hack/hami/deploy_isolator.sh

@mesutoezdil mesutoezdil Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont see --test-hami wired into ci yet. so hamicore e2e still skips in ci for now, right?

@dttung2905 dttung2905 Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I intentionally not wired into CI for now. Wiring them to CI will make the e2e failed because we are running against fake GPU, not real one

fi

echo "Cluster setup complete. Cluster name: $CLUSTER_NAME"
Loading