README.md

Scheduling GPU Workloads with KAI Scheduler

Table of Contents

• Prerequisites
• Queue Configuration
• Initial Cluster State
• Scheduling Actions after New Workload Submission
• Demo Flow
  • Step 1: Setup the Initial Cluster State
  • Step 2: Submit New Workloads
  • Step 3: Monitor Scheduling
• Troubleshooting

Prerequisites

• Kubernetes cluster with KAI Scheduler installed (version 1.0 or later)
• 4 nodes with 4 GPUs each (16 total GPUs)
• NVIDIA device plugin installed and configured
• Kubeflow training operator installed (required for distributed training)

Queue Configuration

Department 1 (Higher Priority - 200)

• Project A (Priority: 150)
  • 4 GPU quota (25% of cluster)
• Project B (Priority: 100)
  • 4 GPU quota (25% of cluster)

Department 2 (Lower Priority - 100)

• Project C (Priority: 150)
  • 4 GPU quota (25% of cluster)
• Project D (Priority: 100)
  • 4 GPU quota (25% of cluster)

Initial Cluster State

• Training Job 1 (2 GPUs)
• Training Job 2 (1 GPUs)
• Distributed Training Job 3 (3 GPUs)
• Interactive Job (4 GPUs)

Step 1: Setup the Initial Cluster State

kubectl apply -f 1-queues.yaml
kubectl apply -f 2-initial-state.yaml

Step 2: Submit New Workloads

kubectl apply -f 3-new-workloads.yaml

Cleanup

kubectl delete -f 3-new-workloads.yaml
kubectl delete -f 2-initial-state.yaml
kubectl delete -f 1-queues.yaml

Troubleshooting

If pods are not being scheduled, check:

1. Namespace exists and matches queue name
2. runai/queue label is correctly set
3. schedulerName is set to kai-scheduler
4. Resource requests are within queue quotas
5. Node resources are available
6. For distributed training make sure that sufficient resources are available on the nodes