Node Label Updates (#66)

Signed-off-by: Claudia <[email protected]>

Author: cmisale

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,6 +1,6 @@
 <!-- 
 
--=-=-=-= Replace the itallic content with your own. -=-=-=-=
+-=-=-=-= Replace the italic content with your own. -=-=-=-=
 -=-=-=-=  Don't forget to update the GitHub Issue   -=-=-=-=
 -=-=-=-=      your Pull-Request pertains to!        -=-=-=-=
 

HEALTH_CHECKS.md

@@ -6,6 +6,8 @@ Here is a breakdown of the existing health checks:
     - Description  : Host-to-device connection speeds, one measurement per GPU. Codebase in tag [v12.4.1](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/bandwidthTest)
     - Outputs: Pass/fail results based on PCIe bandwidth thresholds.
     - Implementation: Compares bandwidth results to a threshold (e.g., 8 GB/s). If the measured bandwidth falls below the threshold, it triggers a failure.
+    - It is recommended to set a threshold that is 25% or lower of the expected peak PCIe bandwidth capability, which maps to maximum peak from 16 lanes to 4 lanes. For example, for a PCIe Gen4x16, reported peak bandwidth is 63GB/s. A degradation at 25% is 15.75GB/s, which corresponds to PCIe Gen4x4.
+    - The measured bandwidth is expected to be at least 80% of the expected peak PCIe generation bandwidth.
 2. **GPU Memory Check (remapped)**
     - Description: Information from nvidia-smi regarding GPU memory remapped rows.
     - Outputs: Reports the state of GPU memory (normal/faulty).
@@ -35,14 +37,40 @@ These checks are configured to run periodically (e.g., hourly), and results are
 
 ## Deep Diagnostics and Node Labeling
 
-Autopilot runs health checks periodically on GPU nodes, and if any of the health checks returns an error, the node is labeled with `autopilot.ibm.com/gpuhealth: ERR`. Otherwise, the label is set as `PASS`.
+Autopilot's periodic health checks, will label the worker nodes according to the result obtained.
+Lightweight and invasive health checks, may use different labeling system.
 
-Also, more extensive tests, namely DCGM diagnostics level 3, are also executed automatically only on nodes that have free GPUs. This deeper analysis is needed to reveal problems in the GPUs that can be found only after running level 3 DCGM diagnostic.
+If the health checks, lightweight or invasive, report success, the node is marked with
+
+```yaml
+autopilot.ibm.com/gpuhealth: PASS
+```
+
+When the lightweight health checks report an issue, the node is labelled with
+
+```yaml
+autopilot.ibm.com/gpuhealth: WARN
+```
+
+### Invasive health checks
+
+The invasive DCGM diagnostics level 3 health check, executed automatically only on nodes that have free GPUs. This deeper analysis is needed to reveal problems in the GPUs that can be found only after running level 3 DCGM diagnostic.
 This type of diagnostics can help deciding if the worker node should be used for running workloads or not. To facilitate this task, Autopilot will label nodes with key `autopilot.ibm.com/dcgm.level.3`.
 
-If errors are found during the level 3 diagnostics, the label `autopilot.ibm.com/dcgm.level.3` will contain detailed information about the error in the following format: 
+If a fatal error is found, the `gpuhealth` label is updated to evict.
 
-`ERR_Year-Month-Date_Hour.Minute.UTC_Diagnostic_Test.gpuID,Diagnostic_Test.gpuID,...`
+
+```yaml
+autopilot.ibm.com/gpuhealth: EVICT
+```
+
+Only fatal errors should produce an `EVICT` label. We follow [NVIDIA recommendations](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#id3), although it is possible to customize the list of tests through the Helm chart. The default values are `[PCIe,NVLink,ECC,GPU Memory]`.
+
+If errors are found during the level 3 diagnostics, the label `autopilot.ibm.com/dcgm.level.3` will contain detailed information about the error in the following format:
+
+```yaml
+autopilot.ibm.com/dcgm.level.3: ERR_Year-Month-Date_Hour.Minute.UTC_Diagnostic_Test.gpuID,Diagnostic_Test.gpuID,...`
+```
 
 - `ERR`: An indicator that an error has occurred
 - `Year-Month-Date_Hour.Minute.UTC`: Timestamp of completed diagnostics
@@ -63,3 +91,5 @@ All metrics are accessible through Prometheus and Grafana dashboards. The gauge
 - `node`, filter by node name
 - `cpumodel` and `gpumodel`, for heterogeneous clusters
 - `deviceid` to select specific GPUs, when available
+
+For more information on how to set up alerts based on metrics, please refer to the [alert manager folder](alertmanager/README.md).

autopilot-daemon/gpu-dcgm/entrypoint.py

@@ -15,11 +15,12 @@
 parser.add_argument('-r', '--run', type=str, default='1')
 parser.add_argument('-l', '--label_node', action='store_true')
 args = parser.parse_args()
+
 def main():
     output = os.popen('bash ./utils/briefings.sh')
     result = output.read()
     print(result)
-
+    
     if "ABORT" not in result:
         print("[[ DCGM ]] Briefings completed. Continue with dcgm evaluation.")
         command = ['dcgmi', 'diag', '-j', '-r', args.run]
@@ -217,11 +218,29 @@ def patch_node(success, output):
     timestamp = now.strftime("%Y-%m-%d_%H.%M.%SUTC")
     result = ""
     general_health = "PASS"
-    if success:
-        result = "PASS_"+timestamp
+    try:
+        k8s_node = v1.read_node(nodename)
+    except ApiException as e:
+        print("Exception when calling corev1api->read_node: %s\n" % e)
+        exit()
+
+    node_labels = k8s_node.metadata.labels
+    if os.getenv("DCGM_FATAL_ERRORS") == "":
+        # Only fatal errors should produce an EVICT label. Based on https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#id3
+        dcgm_fatal_errors = ['PCIe','NVLink','ECC','GPU Memory']
     else:
+        dcgm_fatal_errors = os.getenv("DCGM_FATAL_ERRORS")
+
+    if success and node_labels["autopilot.ibm.com/gpuhealth"] in ["PASS", "TESTING"]:
+        # If there is some other warning coming from other tests, i.e., ping or storage, we would overwrite this information. Let's play it safe at this point.
+        result = "PASS_"+timestamp
+    elif not success:
         result = "ERR_"+timestamp+"_"+output
-        general_health = "EVICT"
+        general_health = "WARN"
+        for error in dcgm_fatal_errors:
+            unified = unify_string_format(error)
+            if unified in result:
+                general_health = "EVICT"
 
     label = {
         "metadata": {
@@ -230,7 +249,6 @@ def patch_node(success, output):
                 "autopilot.ibm.com/gpuhealth": general_health}
         }
     }
-    print("label: ", result)
     try:
         api_response = v1.patch_node(nodename, label)
     except ApiException as e:

autopilot-daemon/gpu-power/power-throttle.sh

@@ -1,4 +1,12 @@
 #!/bin/bash
+OUT="$(bash /home/autopilot/utils/briefings.sh | grep ABORT)"
+echo ${OUT}
+if [[ ! -z $OUT ]]; then
+    echo "[[GPU POWER]] ABORT"
+    exit 0
+fi
+echo "[[GPU POWER]] Briefings completed. Continue with power cap evaluation."
+
 RES=$(ls -d /dev/nvidia* 2>1)
 numre='^[0-9]+$'
 D=-1

autopilot-daemon/pkg/cmd/main.go

@@ -8,7 +8,8 @@ import (
 	"os"
 	"time"
 
-	"github.com/IBM/autopilot/pkg/handlers"
+	"github.com/IBM/autopilot/pkg/handler"
+	"github.com/IBM/autopilot/pkg/healthcheck"
 	"github.com/IBM/autopilot/pkg/utils"
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -17,7 +18,7 @@ import (
 
 func main() {
 	port := flag.String("port", "3333", "Port for the webhook to listen to. Defaulted to 3333")
-	bwThreshold := flag.String("bw", "4", "Sets bandwidth threshold for the init container")
+	bwThreshold := flag.Int("bw", 4, "Sets bandwidth threshold for the init container")
 	logFile := flag.String("logfile", "", "File where to save all the events")
 	v := flag.String("loglevel", "2", "Log level")
 	repeat := flag.Int("w", 24, "Run all tests periodically on each node. Time set in hours. Defaults to 24h")
@@ -45,6 +46,9 @@ func main() {
 
 	utils.InitHardwareMetrics()
 
+	// Init the node status map
+	healthcheck.InitNodeStatusMap()
+
 	pMux := http.NewServeMux()
 	promHandler := promhttp.HandlerFor(reg, promhttp.HandlerOpts{})
 	pMux.Handle("/metrics", promHandler)
@@ -59,7 +63,7 @@ func main() {
 	}()
 
 	readinessMux := http.NewServeMux()
-	readinessMux.Handle("/readinessprobe", handlers.ReadinessProbeHandler())
+	readinessMux.Handle("/readinessprobe", handler.ReadinessProbeHandler())
 
 	go func() {
 		klog.Info("Serving Readiness Probe on :8080")
@@ -72,19 +76,19 @@ func main() {
 
 	hcMux := http.NewServeMux()
 
-	hcMux.Handle("/dcgm", handlers.DCGMHandler())
-	hcMux.Handle("/gpumem", handlers.GpuMemHandler())
-	hcMux.Handle("/gpupower", handlers.GpuPowerHandler())
-	hcMux.Handle("/iperf", handlers.IperfHandler())
-	hcMux.Handle("/iperfservers", handlers.StartIperfServersHandler())
-	hcMux.Handle("/iperfstopservers", handlers.StopAllIperfServersHandler())
-	hcMux.Handle("/iperfclients", handlers.StartIperfClientsHandler())
-	hcMux.Handle("/invasive", handlers.InvasiveCheckHandler())
-	hcMux.Handle("/pciebw", handlers.PCIeBWHandler(utils.UserConfig.BWThreshold))
-	hcMux.Handle("/ping", handlers.PingHandler())
-	hcMux.Handle("/pvc", handlers.PVCHandler())
-	hcMux.Handle("/remapped", handlers.RemappedRowsHandler())
-	hcMux.Handle("/status", handlers.SystemStatusHandler())
+	hcMux.Handle("/dcgm", handler.DCGMHandler())
+	hcMux.Handle("/gpumem", handler.GpuMemHandler())
+	hcMux.Handle("/gpupower", handler.GpuPowerHandler())
+	hcMux.Handle("/iperf", handler.IperfHandler())
+	hcMux.Handle("/iperfservers", handler.StartIperfServersHandler())
+	hcMux.Handle("/iperfstopservers", handler.StopAllIperfServersHandler())
+	hcMux.Handle("/iperfclients", handler.StartIperfClientsHandler())
+	hcMux.Handle("/invasive", handler.InvasiveCheckHandler())
+	hcMux.Handle("/pciebw", handler.PCIeBWHandler())
+	hcMux.Handle("/ping", handler.PingHandler())
+	hcMux.Handle("/pvc", handler.PVCHandler())
+	hcMux.Handle("/remapped", handler.RemappedRowsHandler())
+	hcMux.Handle("/status", handler.SystemStatusHandler())
 
 	s := &http.Server{
 		Addr:         ":" + *port,
@@ -119,7 +123,7 @@ func main() {
 	go utils.WatchNode()
 
 	// Run the health checks at startup, then start the timer
-	handlers.PeriodicCheck()
+	healthcheck.PeriodicCheck()
 
 	periodicChecksTicker := time.NewTicker(time.Duration(*repeat) * time.Hour)
 	defer periodicChecksTicker.Stop()
@@ -128,10 +132,10 @@ func main() {
 	for {
 		select {
 		case <-periodicChecksTicker.C:
-			handlers.PeriodicCheck()
+			healthcheck.PeriodicCheck()
 		case <-invasiveChecksTicker.C:
 			if *invasive > 0 {
-				handlers.InvasiveCheck()
+				healthcheck.InvasiveCheck()
 			}
 		}
 	}